{"html_url": "https://github.com/simonw/datasette/issues/262#issuecomment-1399145981", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/262", "id": 1399145981, "node_id": "IC_kwDOBm6k_c5TZUX9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-01-21T01:56:52Z", "updated_at": "2023-01-21T01:56:52Z", "author_association": "OWNER", "body": "Got first prototype working using `asyncinject` and it's pretty nice:\r\n```diff\r\ndiff --git a/datasette/views/table.py b/datasette/views/table.py\r\nindex ad45ecd3..c8690b22 100644\r\n--- a/datasette/views/table.py\r\n+++ b/datasette/views/table.py\r\n@@ -2,6 +2,7 @@ import asyncio\r\n import itertools\r\n import json\r\n \r\n+from asyncinject import Registry\r\n import markupsafe\r\n \r\n from datasette.plugins import pm\r\n@@ -538,57 +539,60 @@ class TableView(DataView):\r\n         # Execute the main query!\r\n         results = await db.execute(sql, params, truncate=True, **extra_args)\r\n \r\n-        # Calculate the total count for this query\r\n-        count = None\r\n-        if (\r\n-            not db.is_mutable\r\n-            and self.ds.inspect_data\r\n-            and count_sql == f\"select count(*) from {table_name} \"\r\n-        ):\r\n-            # We can use a previously cached table row count\r\n-            try:\r\n-                count = self.ds.inspect_data[database_name][\"tables\"][table_name][\r\n-                    \"count\"\r\n-                ]\r\n-            except KeyError:\r\n-                pass\r\n-\r\n-        # Otherwise run a select count(*) ...\r\n-        if count_sql and count is None and not nocount:\r\n-            try:\r\n-                count_rows = list(await db.execute(count_sql, from_sql_params))\r\n-                count = count_rows[0][0]\r\n-            except QueryInterrupted:\r\n-                pass\r\n-\r\n-        # Faceting\r\n-        if not self.ds.setting(\"allow_facet\") and any(\r\n-            arg.startswith(\"_facet\") for arg in request.args\r\n-        ):\r\n-            raise BadRequest(\"_facet= is not allowed\")\r\n+        # Resolve extras\r\n+        extras = _get_extras(request)\r\n+        if request.args.getlist(\"_facet\"):\r\n+            extras.add(\"facet_results\")\r\n \r\n-        # pylint: disable=no-member\r\n-        facet_classes = list(\r\n-            itertools.chain.from_iterable(pm.hook.register_facet_classes())\r\n-        )\r\n-        facet_results = {}\r\n-        facets_timed_out = []\r\n-        facet_instances = []\r\n-        for klass in facet_classes:\r\n-            facet_instances.append(\r\n-                klass(\r\n-                    self.ds,\r\n-                    request,\r\n-                    database_name,\r\n-                    sql=sql_no_order_no_limit,\r\n-                    params=params,\r\n-                    table=table_name,\r\n-                    metadata=table_metadata,\r\n-                    row_count=count,\r\n-                )\r\n+        async def extra_count():\r\n+            # Calculate the total count for this query\r\n+            count = None\r\n+            if (\r\n+                not db.is_mutable\r\n+                and self.ds.inspect_data\r\n+                and count_sql == f\"select count(*) from {table_name} \"\r\n+            ):\r\n+                # We can use a previously cached table row count\r\n+                try:\r\n+                    count = self.ds.inspect_data[database_name][\"tables\"][table_name][\r\n+                        \"count\"\r\n+                    ]\r\n+                except KeyError:\r\n+                    pass\r\n+\r\n+            # Otherwise run a select count(*) ...\r\n+            if count_sql and count is None and not nocount:\r\n+                try:\r\n+                    count_rows = list(await db.execute(count_sql, from_sql_params))\r\n+                    count = count_rows[0][0]\r\n+                except QueryInterrupted:\r\n+                    pass\r\n+            return count\r\n+\r\n+        async def facet_instances(extra_count):\r\n+            facet_instances = []\r\n+            facet_classes = list(\r\n+                itertools.chain.from_iterable(pm.hook.register_facet_classes())\r\n             )\r\n+            for facet_class in facet_classes:\r\n+                facet_instances.append(\r\n+                    facet_class(\r\n+                        self.ds,\r\n+                        request,\r\n+                        database_name,\r\n+                        sql=sql_no_order_no_limit,\r\n+                        params=params,\r\n+                        table=table_name,\r\n+                        metadata=table_metadata,\r\n+                        row_count=extra_count,\r\n+                    )\r\n+                )\r\n+            return facet_instances\r\n+\r\n+        async def extra_facet_results(facet_instances):\r\n+            facet_results = {}\r\n+            facets_timed_out = []\r\n \r\n-        async def execute_facets():\r\n             if not nofacet:\r\n                 # Run them in parallel\r\n                 facet_awaitables = [facet.facet_results() for facet in facet_instances]\r\n@@ -607,9 +611,13 @@ class TableView(DataView):\r\n                         facet_results[key] = facet_info\r\n                     facets_timed_out.extend(instance_facets_timed_out)\r\n \r\n-        suggested_facets = []\r\n+            return {\r\n+                \"results\": facet_results,\r\n+                \"timed_out\": facets_timed_out,\r\n+            }\r\n \r\n-        async def execute_suggested_facets():\r\n+        async def extra_suggested_facets(facet_instances):\r\n+            suggested_facets = []\r\n             # Calculate suggested facets\r\n             if (\r\n                 self.ds.setting(\"suggest_facets\")\r\n@@ -624,8 +632,15 @@ class TableView(DataView):\r\n                 ]\r\n                 for suggest_result in await gather(*facet_suggest_awaitables):\r\n                     suggested_facets.extend(suggest_result)\r\n+            return suggested_facets\r\n+\r\n+        # Faceting\r\n+        if not self.ds.setting(\"allow_facet\") and any(\r\n+            arg.startswith(\"_facet\") for arg in request.args\r\n+        ):\r\n+            raise BadRequest(\"_facet= is not allowed\")\r\n \r\n-        await gather(execute_facets(), execute_suggested_facets())\r\n+        # pylint: disable=no-member\r\n \r\n         # Figure out columns and rows for the query\r\n         columns = [r[0] for r in results.description]\r\n@@ -732,17 +747,56 @@ class TableView(DataView):\r\n             rows = rows[:page_size]\r\n \r\n         # human_description_en combines filters AND search, if provided\r\n-        human_description_en = filters.human_description_en(\r\n-            extra=extra_human_descriptions\r\n-        )\r\n+        async def extra_human_description_en():\r\n+            human_description_en = filters.human_description_en(\r\n+                extra=extra_human_descriptions\r\n+            )\r\n+            if sort or sort_desc:\r\n+                human_description_en = \" \".join(\r\n+                    [b for b in [human_description_en, sorted_by] if b]\r\n+                )\r\n+            return human_description_en\r\n \r\n         if sort or sort_desc:\r\n             sorted_by = \"sorted by {}{}\".format(\r\n                 (sort or sort_desc), \" descending\" if sort_desc else \"\"\r\n             )\r\n-            human_description_en = \" \".join(\r\n-                [b for b in [human_description_en, sorted_by] if b]\r\n-            )\r\n+\r\n+        async def extra_next_url():\r\n+            return next_url\r\n+\r\n+        async def extra_columns():\r\n+            return columns\r\n+\r\n+        async def extra_primary_keys():\r\n+            return pks\r\n+\r\n+        registry = Registry(\r\n+            extra_count,\r\n+            extra_facet_results,\r\n+            extra_suggested_facets,\r\n+            facet_instances,\r\n+            extra_human_description_en,\r\n+            extra_next_url,\r\n+            extra_columns,\r\n+            extra_primary_keys,\r\n+        )\r\n+\r\n+        results = await registry.resolve_multi(\r\n+            [\"extra_{}\".format(extra) for extra in extras]\r\n+        )\r\n+        data = {\r\n+            \"ok\": True,\r\n+            \"rows\": rows[:page_size],\r\n+            \"next\": next_value and str(next_value) or None,\r\n+        }\r\n+        data.update({\r\n+            key.replace(\"extra_\", \"\"): value\r\n+            for key, value in results.items()\r\n+            if key.startswith(\"extra_\")\r\n+            and key.replace(\"extra_\", \"\") in extras\r\n+        })\r\n+        return Response.json(data, default=repr)\r\n \r\n         async def extra_template():\r\n             nonlocal sort\r\n@@ -1334,3 +1388,11 @@ class TableDropView(BaseView):\r\n \r\n         await db.execute_write_fn(drop_table)\r\n         return Response.json({\"ok\": True}, status=200)\r\n+\r\n+\r\n+def _get_extras(request):\r\n+    extra_bits = request.args.getlist(\"_extra\")\r\n+    extras = set()\r\n+    for bit in extra_bits:\r\n+        extras.update(bit.split(\",\"))\r\n+    return extras\r\n```\r\nWith that in place, `http://127.0.0.1:8001/content/releases?author=25778&_size=1&_extra=count,primary_keys,columns&_facet=author` returns:\r\n```json\r\n{\r\n  \"ok\": true,\r\n  \"rows\": [\r\n    {\r\n      \"html_url\": \"https://github.com/eyeseast/geocode-sqlite/releases/tag/0.1.2\",\r\n      \"id\": 30926270,\r\n      \"author\": {\r\n        \"value\": 25778,\r\n        \"label\": \"eyeseast\"\r\n      },\r\n      \"node_id\": \"MDc6UmVsZWFzZTMwOTI2Mjcw\",\r\n      \"tag_name\": \"0.1.2\",\r\n      \"target_commitish\": \"master\",\r\n      \"name\": \"v0.1.2\",\r\n      \"draft\": 0,\r\n      \"prerelease\": 1,\r\n      \"created_at\": \"2020-09-08T17:48:24Z\",\r\n      \"published_at\": \"2020-09-08T17:50:15Z\",\r\n      \"body\": \"Basic API is in place, with CLI support for Google, Bing, MapQuest and Nominatum (OSM) geocoders.\",\r\n      \"repo\": {\r\n        \"value\": 293361514,\r\n        \"label\": \"geocode-sqlite\"\r\n      },\r\n      \"reactions\": null,\r\n      \"mentions_count\": null\r\n    }\r\n  ],\r\n  \"next\": \"30926270\",\r\n  \"primary_keys\": [\r\n    \"id\"\r\n  ],\r\n  \"columns\": [\r\n    \"html_url\",\r\n    \"id\",\r\n    \"author\",\r\n    \"node_id\",\r\n    \"tag_name\",\r\n    \"target_commitish\",\r\n    \"name\",\r\n    \"draft\",\r\n    \"prerelease\",\r\n    \"created_at\",\r\n    \"published_at\",\r\n    \"body\",\r\n    \"repo\",\r\n    \"reactions\",\r\n    \"mentions_count\"\r\n  ],\r\n  \"count\": 25,\r\n  \"facet_results\": {\r\n    \"results\": {\r\n      \"author\": {\r\n        \"name\": \"author\",\r\n        \"type\": \"column\",\r\n        \"hideable\": true,\r\n        \"toggle_url\": \"/content/releases?author=25778&_size=1&_extra=count%2Cprimary_keys%2Ccolumns\",\r\n        \"results\": [\r\n          {\r\n            \"value\": 25778,\r\n            \"label\": \"eyeseast\",\r\n            \"count\": 25,\r\n            \"toggle_url\": \"http://127.0.0.1:8001/content/releases?_size=1&_extra=count%2Cprimary_keys%2Ccolumns&_facet=author\",\r\n            \"selected\": true\r\n          }\r\n        ],\r\n        \"truncated\": false\r\n      }\r\n    },\r\n    \"timed_out\": []\r\n  }\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323658641, "label": "Add ?_extra= mechanism for requesting extra properties in JSON"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/262#issuecomment-1399178591", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/262", "id": 1399178591, "node_id": "IC_kwDOBm6k_c5TZcVf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-01-21T04:53:15Z", "updated_at": "2023-01-21T04:53:15Z", "author_association": "OWNER", "body": "Implementing this to work with the `.json` extension is going to be a lot harder.\r\n\r\nThe challenge here is that we're working with the whole `BaseView()` v.s. `TableView()` abstraction, which I've been wanting to get rid of for a long time.\r\n\r\n`BaseView()` calls `.data()` and expects to get back a `(data, extra_template_data, templates)` tuple - then if a format is in play (`.json` or `.geojson` or similar from a plugin) it hands off `data` to that. If `.csv` is involved it does something special, in order to support streaming responses. And if it's regular HTML it calls `await extra_template_data()` and combines that with `data` and passes it to the template.\r\n\r\nI want this to work completely differently: I want the formats (including HTML) to have the option of adding some extra `?_extra=` extras, then I want HTML to be able to render the page entirely from the JSON if necessary.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323658641, "label": "Add ?_extra= mechanism for requesting extra properties in JSON"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/262#issuecomment-1399178823", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/262", "id": 1399178823, "node_id": "IC_kwDOBm6k_c5TZcZH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-01-21T04:54:49Z", "updated_at": "2023-01-21T04:54:49Z", "author_association": "OWNER", "body": "I pushed my prototype so far, going to start a draft PR for it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323658641, "label": "Add ?_extra= mechanism for requesting extra properties in JSON"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/262#issuecomment-1399184540", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/262", "id": 1399184540, "node_id": "IC_kwDOBm6k_c5TZdyc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-01-21T05:35:32Z", "updated_at": "2023-01-21T05:35:32Z", "author_association": "OWNER", "body": "It's annoying that the https://docs.datasette.io/en/0.64.1/plugin_hooks.html#register-output-renderer-datasette plugin hook passes `rows` as \"list of sqlite3.Row objects\" - I'd prefer it if that plugin hook worked with JSON data, not `sqlite3.Row`.\r\n\r\nhttps://docs.datasette.io/en/0.64.1/plugin_hooks.html#render-cell-row-value-column-table-database-datasette is documented as accepting `Row` but actually gets `CustomRow`, see:\r\n\r\n- #1973", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323658641, "label": "Add ?_extra= mechanism for requesting extra properties in JSON"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/262#issuecomment-1399184642", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/262", "id": 1399184642, "node_id": "IC_kwDOBm6k_c5TZd0C", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-01-21T05:36:22Z", "updated_at": "2023-01-21T05:41:06Z", "author_association": "OWNER", "body": "Maybe `\"rows\"` should be a default `?_extra=`... but it should be possible to request `\"arrays\"` instead which would be a list of arrays, more suitable perhaps for custom renderers such as the CSV one.\r\n\r\nThis could be quite neat, in that EVERY key in the JSON representation would be defined as an extra - just some would be on by default. There could even be a mechanism for turning them back off again, maybe using `?_extra=-rows`.\r\n\r\nIn which case maybe `?_extra=` isn't actually the right name for this feature. It could be `?_key=` perhaps, or `?_field=`.\r\n\r\nBeing able to pass `?_field=count,-rows` to get back just the count (and skip executing the count entirely) would be pretty neat.\r\n\r\nAlthough `?_only=count` would be tidier. So maybe the pair of `?_only=` and `?_extra=` would make sense.\r\n\r\nWould `?_only=rows` still return the `\"ok\"` field so you can always look at that to confirm an error didn't occur?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323658641, "label": "Add ?_extra= mechanism for requesting extra properties in JSON"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1999#issuecomment-1399341151", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1999", "id": 1399341151, "node_id": "IC_kwDOBm6k_c5TaEBf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-01-21T22:03:20Z", "updated_at": "2023-01-21T22:03:20Z", "author_association": "OWNER", "body": "I think I'm going to have to write a new view function from scratch which completely ignores the existing BaseView/DataView/TableView hierarchy.\r\n\r\nHere's what I get on the incoming request:\r\n```\r\n(Pdb) request.url, request.full_path, request.host, request.url_vars\r\n('http://127.0.0.1:8001/content/repos.json', '/content/repos.json', '127.0.0.1:8001',\r\n  {'database': 'content', 'table': 'repos', 'format': 'json'})\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1551694938, "label": "?_extra= support (draft)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1999#issuecomment-1399341658", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1999", "id": 1399341658, "node_id": "IC_kwDOBm6k_c5TaEJa", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-01-21T22:06:29Z", "updated_at": "2023-01-21T22:07:30Z", "author_association": "OWNER", "body": "Relevant:\r\n- #1101\r\n- #1672\r\n- #1062", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1551694938, "label": "?_extra= support (draft)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1101#issuecomment-1399341761", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1101", "id": 1399341761, "node_id": "IC_kwDOBm6k_c5TaELB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-01-21T22:07:19Z", "updated_at": "2023-01-21T22:07:19Z", "author_association": "OWNER", "body": "Idea for supporting streaming with the `register_output_renderer` hook:\r\n\r\n```python\r\n@hookimpl\r\ndef register_output_renderer(datasette):\r\n    return {\r\n        \"extension\": \"test\",\r\n        \"render\": render_demo,\r\n        \"can_render\": can_render_demo,\r\n        \"render_stream\": render_demo_stream, # This is new\r\n    }\r\n```\r\nSo there's a new `\"render_stream\"` key which can be returned, which if present means that the output renderer supports streaming.\r\n\r\nI'll play around with the design of that function signature in:\r\n\r\n- #1999\r\n- #1062 ", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 749283032, "label": "register_output_renderer() should support streaming data"}, "performed_via_github_app": null}