{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112713581", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112713581, "node_id": "IC_kwDOBm6k_c5CUqlt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T22:31:11Z", "updated_at": "2022-04-28T22:31:11Z", "author_association": "OWNER", "body": "I'm going to change the default API response to look like this: \r\n```json\r\n{\r\n  \"rows\": [\r\n    {\r\n      \"pk\": 1,\r\n      \"created\": \"2019-01-14 08:00:00\",\r\n      \"planet_int\": 1,\r\n      \"on_earth\": 1,\r\n      \"state\": \"CA\",\r\n      \"_city_id\": 1,\r\n      \"_neighborhood\": \"Mission\",\r\n      \"tags\": \"[\\\"tag1\\\", \\\"tag2\\\"]\",\r\n      \"complex_array\": \"[{\\\"foo\\\": \\\"bar\\\"}]\",\r\n      \"distinct_some_null\": \"one\",\r\n      \"n\": \"n1\"\r\n    },\r\n    {\r\n      \"pk\": 2,\r\n      \"created\": \"2019-01-14 08:00:00\",\r\n      \"planet_int\": 1,\r\n      \"on_earth\": 1,\r\n      \"state\": \"CA\",\r\n      \"_city_id\": 1,\r\n      \"_neighborhood\": \"Dogpatch\",\r\n      \"tags\": \"[\\\"tag1\\\", \\\"tag3\\\"]\",\r\n      \"complex_array\": \"[]\",\r\n      \"distinct_some_null\": \"two\",\r\n      \"n\": \"n2\"\r\n    }\r\n  ],\r\n  \"next\": null,\r\n  \"next_url\": null\r\n}\r\n```\r\nBasically https://latest.datasette.io/fixtures/facetable.json?_shape=objects but with just the `rows`, `next` and `next_url` fields returned by default.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112716611", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112716611, "node_id": "IC_kwDOBm6k_c5CUrVD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T22:36:24Z", "updated_at": "2022-04-28T22:36:24Z", "author_association": "OWNER", "body": "Then I'm going to implement the following `?_extra=` options:\r\n\r\n- `?_extra=facet_results` - to see facet results\r\n- `?_extra=suggested_facets` - for suggested facets\r\n- `?_extra=count` - for the count of total rows\r\n- `?_extra=columns` - for a list of column names\r\n- `?_extra=primary_keys` - for a list of primary keys\r\n- `?_extra=query` - a `{\"sql\" \"select ...\", \"params\": {}}` object\r\n\r\nI thought about having `?_extra=facet_results` returned automatically if the user specifies at least one `?_facet` - but that doesn't work for default facets configured in `metadata.json` - how can the user opt out of those being returned? So I'm going to say you don't see facets at all if you don't include `?_extra=facet_results`.\r\n\r\nI'm tempted to add `?_extra=_all` to return everything, but I can decide if that's a good idea later.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112717210", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112717210, "node_id": "IC_kwDOBm6k_c5CUrea", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T22:37:37Z", "updated_at": "2022-04-28T22:37:37Z", "author_association": "OWNER", "body": "This means `filtered_table_rows_count` is going to become `count`. I had originally picked that terrible name to avoid confusion between the count of all rows in the table and the count of rows that were filtered.\r\n\r\nI'll add `?_extra=table_count` for getting back the full table count instead. I think `count` is clear enough!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112717745", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112717745, "node_id": "IC_kwDOBm6k_c5CUrmx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T22:38:39Z", "updated_at": "2022-04-28T22:39:05Z", "author_association": "OWNER", "body": "(I remain keen on the idea of shipping a plugin that restores the old default API shape to people who have written pre-Datasette-1.0 code against it, but I'll tackle that much later. I really like how jQuery has a culture of doing this.)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112721321", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112721321, "node_id": "IC_kwDOBm6k_c5CUsep", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T22:44:05Z", "updated_at": "2022-04-28T22:44:14Z", "author_association": "OWNER", "body": "I may be able to implement this mostly in the `json_renderer()` function: https://github.com/simonw/datasette/blob/94a3171b01fde5c52697aeeff052e3ad4bab5391/datasette/renderer.py#L29-L34", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112730416", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112730416, "node_id": "IC_kwDOBm6k_c5CUusw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T23:01:21Z", "updated_at": "2022-04-28T23:01:21Z", "author_association": "OWNER", "body": "I'm not sure what to do about the `\"truncated\": true/false` key.\r\n\r\nIt's not really relevant to table results, since they are paginated whether or not you ask for them to be.\r\n\r\nIt plays a role in query results, where you might run `select * from table` and get back 1000 results because Datasette truncates at that point rather than returning everything.\r\n\r\nAdding it to every table result and always setting it to `\"truncated\": false` feels confusing.\r\n\r\nI think I'm going to keep it exclusively in the default representation for the `/db?sql=...` query endpoint, and not return it at all for tables.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112732563", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112732563, "node_id": "IC_kwDOBm6k_c5CUvOT", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T23:05:03Z", "updated_at": "2022-04-28T23:05:03Z", "author_association": "OWNER", "body": "OK, the prototype of this is looking really good - it's very pleasant to use.\r\n\r\n`http://127.0.0.1:8001/github_memory/issue_comments.json?_search=simon&_sort=id&_size=5&_extra=query_ms&_extra=count&_col=body` returns this:\r\n\r\n```json\r\n{\r\n  \"rows\": [\r\n    {\r\n      \"id\": 338854988,\r\n      \"body\": \"    /database-name/table-name?name__contains=simon&sort=id+desc\\r\\n\\r\\nNote that if there's a column called \\\"sort\\\" you can still do sort__exact=blah\\r\\n\\r\\n\"\r\n    },\r\n    {\r\n      \"id\": 346427794,\r\n      \"body\": \"Thanks. There is a way to use pip to grab apsw, which also let's you configure it (flags to build extensions, use an internal sqlite, etc). Don't know how that works as a dependency for another package, though.\\n\\nOn November 22, 2017 11:38:06 AM EST, Simon Willison <notifications@github.com> wrote:\\n>I have a solution for FTS already, but I'm interested in apsw as a\\n>mechanism for allowing custom virtual tables to be written in Python\\n>(pysqlite only lets you write custom functions)\\n>\\n>Not having PyPI support is pretty tough though. I'm planning a\\n>plugin/extension system which would be ideal for things like an\\n>optional apsw mode, but that's a lot harder if apsw isn't in PyPI.\\n>\\n>-- \\n>You are receiving this because you authored the thread.\\n>Reply to this email directly or view it on GitHub:\\n>https://github.com/simonw/datasette/issues/144#issuecomment-346405660\\n\"\r\n    },\r\n    {\r\n      \"id\": 348252037,\r\n      \"body\": \"WOW!\\n\\n--\\nPaul Ford // (646) 369-7128 // @ftrain\\n\\nOn Thu, Nov 30, 2017 at 11:47 AM, Simon Willison <notifications@github.com>\\nwrote:\\n\\n> Remaining work on this now lives in a milestone:\\n> https://github.com/simonw/datasette/milestone/6\\n>\\n> \u2014\\n> You are receiving this because you were mentioned.\\n> Reply to this email directly, view it on GitHub\\n> <https://github.com/simonw/datasette/issues/153#issuecomment-348248406>,\\n> or mute the thread\\n> <https://github.com/notifications/unsubscribe-auth/AABPKHzaVPKwTOoHouK2aMUnM-mPnPk6ks5s7twzgaJpZM4Qq2zW>\\n> .\\n>\\n\"\r\n    },\r\n    {\r\n      \"id\": 391141391,\r\n      \"body\": \"I'm going to clean this up for consistency tomorrow morning so hold off\\nmerging until then please\\n\\nOn Tue, May 22, 2018 at 6:34 PM, Simon Willison <notifications@github.com>\\nwrote:\\n\\n> Yeah let's try this without pysqlite3 and see if we still get the correct\\n> version.\\n>\\n> \u2014\\n> You are receiving this because you authored the thread.\\n> Reply to this email directly, view it on GitHub\\n> <https://github.com/simonw/datasette/pull/280#issuecomment-391076458>, or mute\\n> the thread\\n> <https://github.com/notifications/unsubscribe-auth/AAihfMI-H6CBt-Py0xdBbH2xDK0KsjT2ks5t1EwYgaJpZM4UI_2m>\\n> .\\n>\\n\"\r\n    },\r\n    {\r\n      \"id\": 391355030,\r\n      \"body\": \"No objections;\\r\\nIt's good to go @simonw\\r\\n\\r\\nOn Wed, 23 May 2018, 14:51 Simon Willison, <notifications@github.com> wrote:\\r\\n\\r\\n> @r4vi <https://github.com/r4vi> any objections to me merging this?\\r\\n>\\r\\n> \u2014\\r\\n> You are receiving this because you were mentioned.\\r\\n> Reply to this email directly, view it on GitHub\\r\\n> <https://github.com/simonw/datasette/pull/280#issuecomment-391354237>, or mute\\r\\n> the thread\\r\\n> <https://github.com/notifications/unsubscribe-auth/AAihfM_2DN5WR2mkO-VK6ozDmkUQ4IMjks5t1WlcgaJpZM4UI_2m>\\r\\n> .\\r\\n>\\r\\n\"\r\n    }\r\n  ],\r\n  \"next\": \"391355030,391355030\",\r\n  \"next_url\": \"http://127.0.0.1:8001/github_memory/issue_comments.json?_search=simon&_size=5&_extra=query_ms&_extra=count&_col=body&_next=391355030%2C391355030&_sort=id\",\r\n  \"count\": 57,\r\n  \"query_ms\": 21.780223003588617\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112734577", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112734577, "node_id": "IC_kwDOBm6k_c5CUvtx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T23:08:42Z", "updated_at": "2022-04-28T23:08:42Z", "author_association": "OWNER", "body": "That prototype is a very small amount of code so far:\r\n```diff\r\ndiff --git a/datasette/renderer.py b/datasette/renderer.py\r\nindex 4508949..b600e1b 100644\r\n--- a/datasette/renderer.py\r\n+++ b/datasette/renderer.py\r\n@@ -28,6 +28,10 @@ def convert_specific_columns_to_json(rows, columns, json_cols):\r\n \r\n def json_renderer(args, data, view_name):\r\n     \"\"\"Render a response as JSON\"\"\"\r\n+    from pprint import pprint\r\n+\r\n+    pprint(data)\r\n+\r\n     status_code = 200\r\n \r\n     # Handle the _json= parameter which may modify data[\"rows\"]\r\n@@ -43,6 +47,41 @@ def json_renderer(args, data, view_name):\r\n     if \"rows\" in data and not value_as_boolean(args.get(\"_json_infinity\", \"0\")):\r\n         data[\"rows\"] = [remove_infinites(row) for row in data[\"rows\"]]\r\n \r\n+    # Start building the default JSON here\r\n+    columns = data[\"columns\"]\r\n+    next_url = data.get(\"next_url\")\r\n+    output = {\r\n+        \"rows\": [dict(zip(columns, row)) for row in data[\"rows\"]],\r\n+        \"next\": data[\"next\"],\r\n+        \"next_url\": next_url,\r\n+    }\r\n+\r\n+    extras = set(args.getlist(\"_extra\"))\r\n+\r\n+    extras_map = {\r\n+        # _extra=   :    data[field]\r\n+        \"count\": \"filtered_table_rows_count\",\r\n+        \"facet_results\": \"facet_results\",\r\n+        \"suggested_facets\": \"suggested_facets\",\r\n+        \"columns\": \"columns\",\r\n+        \"primary_keys\": \"primary_keys\",\r\n+        \"query_ms\": \"query_ms\",\r\n+        \"query\": \"query\",\r\n+    }\r\n+    for extra_key, data_key in extras_map.items():\r\n+        if extra_key in extras:\r\n+            output[extra_key] = data[data_key]\r\n+\r\n+    body = json.dumps(output, cls=CustomJSONEncoder)\r\n+    content_type = \"application/json; charset=utf-8\"\r\n+    headers = {}\r\n+    if next_url:\r\n+        headers[\"link\"] = f'<{next_url}>; rel=\"next\"'\r\n+    return Response(\r\n+        body, status=status_code, headers=headers, content_type=content_type\r\n+    )\r\n+\r\n+\r\n     # Deal with the _shape option\r\n     shape = args.get(\"_shape\", \"arrays\")\r\n     # if there's an error, ignore the shape entirely\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1114013757", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1114013757, "node_id": "IC_kwDOBm6k_c5CZoA9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-30T16:15:51Z", "updated_at": "2022-04-30T18:54:39Z", "author_association": "OWNER", "body": "Deployed a preview of this here: https://latest-1-0-alpha.datasette.io/\r\n\r\nExamples:\r\n\r\n- https://latest-1-0-alpha.datasette.io/fixtures/facetable.json\r\n- https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count\r\n\r\nSecond example produces:\r\n\r\n```json\r\n{\r\n  \"rows\": [],\r\n  \"next\": null,\r\n  \"next_url\": null,\r\n  \"count\": 15,\r\n  \"facet_results\": {\r\n    \"state\": {\r\n      \"name\": \"state\",\r\n      \"type\": \"column\",\r\n      \"hideable\": true,\r\n      \"toggle_url\": \"/fixtures/facetable.json?_size=0&_extra=facet_results&_extra=count\",\r\n      \"results\": [\r\n        {\r\n          \"value\": \"CA\",\r\n          \"label\": \"CA\",\r\n          \"count\": 10,\r\n          \"toggle_url\": \"https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=CA\",\r\n          \"selected\": false\r\n        },\r\n        {\r\n          \"value\": \"MI\",\r\n          \"label\": \"MI\",\r\n          \"count\": 4,\r\n          \"toggle_url\": \"https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=MI\",\r\n          \"selected\": false\r\n        },\r\n        {\r\n          \"value\": \"MC\",\r\n          \"label\": \"MC\",\r\n          \"count\": 1,\r\n          \"toggle_url\": \"https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=MC\",\r\n          \"selected\": false\r\n        }\r\n      ],\r\n      \"truncated\": false\r\n    }\r\n  }\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1114036946", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1114036946, "node_id": "IC_kwDOBm6k_c5CZtrS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-30T18:56:25Z", "updated_at": "2022-04-30T19:04:03Z", "author_association": "OWNER", "body": "Related:\r\n- #1558\r\n\r\nWhich talks about how there was confusion in this example: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1&_size=0\r\n\r\nWhich I fixed in #625 by introducing `tags` and `tags_2` keys, but actually the facet results would be better if they were a list rather than a dictionary.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1114037521", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1114037521, "node_id": "IC_kwDOBm6k_c5CZt0R", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-30T19:01:07Z", "updated_at": "2022-04-30T19:01:07Z", "author_association": "OWNER", "body": "I had to look up what `hideable` means - it means that you can't hide the current facet because it was defined in metadata, not as a `?_facet=` parameter:\r\n\r\nhttps://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/datasette/facets.py#L228\r\n\r\nThat's a bit of a weird thing to expose in the API. Maybe change that to `source` so it can be `metadata` or `request`? That's very slightly less coupled to how the UI works.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1114038259", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1114038259, "node_id": "IC_kwDOBm6k_c5CZt_z", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-30T19:06:03Z", "updated_at": "2022-04-30T19:06:03Z", "author_association": "OWNER", "body": "> but actually the facet results would be better if they were a list rather than a dictionary\r\n\r\nI think `facet_results` in the JSON should match this (used by the HTML) instead:\r\n\r\nhttps://github.com/simonw/datasette/blob/942411ef946e9a34a2094944d3423cddad27efd3/datasette/views/table.py#L737-L741\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}