html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/datasette/issues/1439#issuecomment-1053973425,https://api.github.com/repos/simonw/datasette/issues/1439,1053973425,IC_kwDOBm6k_c4-0lux,9599,simonw,2022-02-28T07:40:12Z,2022-02-28T07:40:12Z,OWNER,"If I make this change it will break existing links to one of the oldest Datasette demos: http://fivethirtyeight.datasettes.com/fivethirtyeight/avengers%2Favengers A plugin that fixes those by redirecting them on 404 would be neat.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045131086,https://api.github.com/repos/simonw/datasette/issues/1439,1045131086,IC_kwDOBm6k_c4-S29O,9599,simonw,2022-02-18T20:22:13Z,2022-02-18T20:22:47Z,OWNER,"Should it encode `%` symbols too, since they have a special meaning in URLs and we can't guarantee that every single web server / proxy out there will round-trip them safely using percentage encoding? If so, would need to pick a different encoding character for them. Maybe `%` becomes `-p` - and in that case `/` could become `-s` too. Is it worth expanding dash-encoding outside of just `/` and `-` and `.` though? Not sure.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045134050,https://api.github.com/repos/simonw/datasette/issues/1439,1045134050,IC_kwDOBm6k_c4-S3ri,9599,simonw,2022-02-18T20:25:04Z,2022-02-18T20:25:04Z,OWNER,Here's a useful modern spec for how existing URL percentage encoding is supposed to work: https://url.spec.whatwg.org/#percent-encoded-bytes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045024276,https://api.github.com/repos/simonw/datasette/issues/1439,1045024276,IC_kwDOBm6k_c4-Sc4U,9599,simonw,2022-02-18T19:01:42Z,2022-02-18T19:55:24Z,OWNER,"> Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly. ```python def dash_encode(s): return s.replace(""-"", ""--"").replace(""."", ""-."").replace(""/"", ""-/"") def dash_decode(s): return s.replace(""-/"", ""/"").replace(""-."", ""."").replace(""--"", ""-"") ``` ```pycon >>> dash_encode(""foo/bar/baz.csv"") 'foo-/bar-/baz-.csv' >>> dash_decode('foo-/bar-/baz-.csv') 'foo/bar/baz.csv' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045027067,https://api.github.com/repos/simonw/datasette/issues/1439,1045027067,IC_kwDOBm6k_c4-Sdj7,9599,simonw,2022-02-18T19:03:26Z,2022-02-18T19:03:26Z,OWNER,"(If I make this change it may break some existing Datasette installations when they upgrade - I could try and build a plugin for them which triggers on 404s and checks to see if the old format would return a 200 response, then returns that.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045032377,https://api.github.com/repos/simonw/datasette/issues/1439,1045032377,IC_kwDOBm6k_c4-Se25,9599,simonw,2022-02-18T19:06:50Z,2022-02-18T19:06:50Z,OWNER,"How does URL routing for https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv work? Right now it's https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/app.py#L1098-L1101 That's not going to capture the dot-dash encoding version of that table name: ```pycon >>> dot_dash_encode(""table/with/slashes.csv"") 'table-/with-/slashes-.csv' ``` Probably needs a fancy regex trick like a negative lookbehind assertion or similar.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045055772,https://api.github.com/repos/simonw/datasette/issues/1439,1045055772,IC_kwDOBm6k_c4-Skkc,9599,simonw,2022-02-18T19:23:33Z,2022-02-18T19:25:42Z,OWNER,"I want a match for this URL: /db/table-/with-/slashes-.csv Maybe this: ^/(?P[^/]+)/(?P([^/]*|(\-/)*|(\-\.)*|(\.\.)*)*$) Here we are matching a sequence of: ([^/]*|(\-/)*|(\-\.)*|(\-\-)*)* So a combination of not-slashes OR -/ or -. Or -- sequences ^/(?P[^/]+)/(?P([^/]*|(\-/)*|(\-\.)*|(\-\-)*)*$) Try that with non-capturing bits: ^/(?P[^/]+)/(?P(?:[^/]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*$) `(?:[^/]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*` visualized is: Here's the explanation on regex101.com https://regex101.com/r/CPnsIO/1 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045059427,https://api.github.com/repos/simonw/datasette/issues/1439,1045059427,IC_kwDOBm6k_c4-Sldj,9599,simonw,2022-02-18T19:26:25Z,2022-02-18T19:26:25Z,OWNER,"With this new pattern I could probably extract out the optional `.json` format string as part of the initial route capturing regex too, rather than the current `table_and_format` hack.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481,https://api.github.com/repos/simonw/datasette/issues/1439,1045069481,IC_kwDOBm6k_c4-Sn6p,9599,simonw,2022-02-18T19:34:41Z,2022-03-05T21:32:22Z,OWNER,"I think I got format extraction working! https://regex101.com/r/A0bW1D/1 ^/(?P[^/]+)/(?P(?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*?)(?:(?\w+))?$ I had to make that crazy inner one even more complicated to stop it from capturing `.` that was not part of `-.`. (?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)* Visualized: So now I have a regex which can extract out the dot-encoded table name AND spot if there is an optional `.format` at the end: If I end up using this in Datasette it's going to need VERY comprehensive unit tests and inline documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045077590,https://api.github.com/repos/simonw/datasette/issues/1439,1045077590,IC_kwDOBm6k_c4-Sp5W,9599,simonw,2022-02-18T19:41:37Z,2022-02-18T19:42:41Z,OWNER,"Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where ""system"" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence. And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too. Maybe change this system to use `.` as the escaping character instead of `-`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045075207,https://api.github.com/repos/simonw/datasette/issues/1439,1045075207,IC_kwDOBm6k_c4-SpUH,9599,simonw,2022-02-18T19:39:35Z,2022-02-18T19:40:13Z,OWNER,"> And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this: > > * `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version > * `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version > * `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version Here's what those look like with the updated version of `dot_dash_encode()` that also encodes `/` as `-/`: - `/db/-/db-/table---.csv-.csv` - HTML - `/db/-/db-/table---.csv-.csv.csv` - CSV - `/db/-/db-/table---.csv-.csv.json` - JSON ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045081042,https://api.github.com/repos/simonw/datasette/issues/1439,1045081042,IC_kwDOBm6k_c4-SqvS,9599,simonw,2022-02-18T19:44:12Z,2022-02-18T19:51:34Z,OWNER,"```python def dot_encode(s): return s.replace(""."", "".."").replace(""/"", ""./"") def dot_decode(s): return s.replace(""./"", ""/"").replace("".."", ""."") ``` No need for hyphen encoding in this variant at all, which simplifies things a bit. (Update: this is flawed, see https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033,https://api.github.com/repos/simonw/datasette/issues/1439,1045086033,IC_kwDOBm6k_c4-Sr9R,9599,simonw,2022-02-18T19:47:43Z,2022-02-18T19:51:11Z,OWNER,"- https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv - https://til.simonwillison.net/-/asgi-scope/db/./db./table-..csv..csv Do both of those survive the round-trip to populate `raw_path` correctly? No! In both cases the `/./` bit goes missing. It looks like this might even be a client issue - `curl` shows me this: ``` ~ % curl -vv -i 'https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv' * Trying 216.239.32.21:443... * Connected to datasette.io (216.239.32.21) port 443 (#0) * ALPN, offering http/1.1 * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: datasette.io * Server certificate: R3 * Server certificate: ISRG Root X1 > GET /-/asgi-scope/db/db./table-..csv..csv HTTP/1.1 ``` So `curl` decided to turn `/-/asgi-scope/db/./db./table` into `/-/asgi-scope/db/db./table` before even sending the request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045082891,https://api.github.com/repos/simonw/datasette/issues/1439,1045082891,IC_kwDOBm6k_c4-SrML,9599,simonw,2022-02-18T19:45:32Z,2022-02-18T19:45:32Z,OWNER,"```pycon >>> dot_encode(""/db/table-.csv.csv"") './db./table-..csv..csv' >>> dot_decode('./db./table-..csv..csv') '/db/table-.csv.csv' ``` I worry that web servers might treat `./` in a special way though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045095348,https://api.github.com/repos/simonw/datasette/issues/1439,1045095348,IC_kwDOBm6k_c4-SuO0,9599,simonw,2022-02-18T19:53:48Z,2022-02-18T19:53:48Z,OWNER,"> Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where ""system"" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence. > > And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too. I don't think this matters. The new regex does indeed capture that kind of page: But Datasette goes through configured route regular expressions in order - so I can have the regex that captures `/db/-/special` routes listed before the one that captures tables and formats.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045099290,https://api.github.com/repos/simonw/datasette/issues/1439,1045099290,IC_kwDOBm6k_c4-SvMa,9599,simonw,2022-02-18T19:56:18Z,2022-02-18T19:56:30Z,OWNER,"> ```python > def dash_encode(s): > return s.replace(""-"", ""--"").replace(""."", ""-."").replace(""/"", ""-/"") > > def dash_decode(s): > return s.replace(""-/"", ""/"").replace(""-."", ""."").replace(""--"", ""-"") > ``` I think **dash-encoding** (new name for this) is the right way forward here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045108611,https://api.github.com/repos/simonw/datasette/issues/1439,1045108611,IC_kwDOBm6k_c4-SxeD,9599,simonw,2022-02-18T20:02:19Z,2022-02-18T20:08:34Z,OWNER,"One other potential variant: ```python def dash_encode(s): return s.replace(""-"", ""-dash-"").replace(""."", ""-dot-"").replace(""/"", ""-slash-"") def dash_decode(s): return s.replace(""-slash-"", ""/"").replace(""-dot-"", ""."").replace(""-dash-"", ""-"") ``` Except this has bugs - it doesn't round-trip safely, because it can get confused about things like `-dash-slash-` in terms of is that a `-dash-` or a `-slash-`? ```pycon >>> dash_encode(""/db/table-.csv.csv"") '-slash-db-slash-table-dash--dot-csv-dot-csv' >>> dash_decode('-slash-db-slash-table-dash--dot-csv-dot-csv') '/db/table-.csv.csv' >>> dash_encode('-slash-db-slash-table-dash--dot-csv-dot-csv') '-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv' >>> dash_decode('-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv') '-dash/dash-db-dash/dash-table-dash--dash.dash-csv-dash.dash-csv' ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045111309,https://api.github.com/repos/simonw/datasette/issues/1439,1045111309,IC_kwDOBm6k_c4-SyIN,9599,simonw,2022-02-18T20:04:24Z,2022-02-18T20:05:40Z,OWNER,"This made me worry that my current `dash_decode()` implementation had unknown round-trip bugs, but thankfully this works OK: ```pycon >>> dash_encode(""/db/table-.csv.csv"") '-/db-/table---.csv-.csv' >>> dash_encode('-/db-/table---.csv-.csv') '---/db---/table-------.csv---.csv' >>> dash_decode('---/db---/table-------.csv---.csv') '-/db-/table---.csv-.csv' >>> dash_decode('-/db-/table---.csv-.csv') '/db/table-.csv.csv' ``` The regex still works against that double-encoded example too: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045117304,https://api.github.com/repos/simonw/datasette/issues/1439,1045117304,IC_kwDOBm6k_c4-Szl4,9599,simonw,2022-02-18T20:09:22Z,2022-02-18T20:09:22Z,OWNER,Adopting this could result in supporting database files with surprising characters in their filename too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1045269544,https://api.github.com/repos/simonw/datasette/issues/1439,1045269544,IC_kwDOBm6k_c4-TYwo,9599,simonw,2022-02-18T22:19:29Z,2022-02-18T22:19:29Z,OWNER,"Note that I've ruled out using `Accept: application/json` to return JSON because it turns out Cloudflare and potentially other CDNs ignore the `Vary: Accept` header entirely: - https://github.com/simonw/datasette/issues/1534","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1049114724,https://api.github.com/repos/simonw/datasette/issues/1439,1049114724,IC_kwDOBm6k_c4-iDhk,9599,simonw,2022-02-23T19:04:40Z,2022-02-23T19:04:40Z,OWNER,I'm going to try dash encoding for table names (and row IDs) in a branch and see how I like it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1049124390,https://api.github.com/repos/simonw/datasette/issues/1439,1049124390,IC_kwDOBm6k_c4-iF4m,9599,simonw,2022-02-23T19:15:00Z,2022-02-23T19:15:00Z,OWNER,"I'll start by modifying this function: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/utils/__init__.py#L732-L749 Later I want to move this to the routing layer to split out `format` automatically, as seen in the regexes here: https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1049126151,https://api.github.com/repos/simonw/datasette/issues/1439,1049126151,IC_kwDOBm6k_c4-iGUH,9599,simonw,2022-02-23T19:17:01Z,2022-02-23T19:17:01Z,OWNER,Actually the relevant code looks to be: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/views/base.py#L481-L498,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1405#issuecomment-888694144,https://api.github.com/repos/simonw/datasette/issues/1405,888694144,IC_kwDOBm6k_c40-GWA,9599,simonw,2021-07-28T23:51:59Z,2021-07-28T23:51:59Z,OWNER,https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/datasette/utils/__init__.py#L918-L926,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",955316250,utils.parse_metadata() should be a documented internal function, https://github.com/simonw/datasette/issues/1405#issuecomment-888694261,https://api.github.com/repos/simonw/datasette/issues/1405,888694261,IC_kwDOBm6k_c40-GX1,9599,simonw,2021-07-28T23:52:21Z,2021-07-28T23:52:21Z,OWNER,Document that it can raise a `BadMetadataError` exception.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",955316250,utils.parse_metadata() should be a documented internal function, https://github.com/simonw/datasette/issues/1402#issuecomment-886968648,https://api.github.com/repos/simonw/datasette/issues/1402,886968648,IC_kwDOBm6k_c403hFI,9599,simonw,2021-07-26T19:30:14Z,2021-07-26T19:30:14Z,OWNER,"I really like this idea. I was thinking it might make a good plugin, but there's not a great mechanism for plugins to inject extra `` content at the moment - plus this actually feels like a reasonable feature for Datasette core itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",951185411,feature request: social meta tags, https://github.com/simonw/datasette/issues/1402#issuecomment-886969541,https://api.github.com/repos/simonw/datasette/issues/1402,886969541,IC_kwDOBm6k_c403hTF,9599,simonw,2021-07-26T19:31:40Z,2021-07-26T19:31:40Z,OWNER,"Datasette could do a pretty good job of this by default, using `twitter:card` and `og:url` tags - like on https://til.simonwillison.net/jq/extracting-objects-recursively I could also provide a mechanism to customize these - in particular to add images of some sort. It feels like something that should tie in to the metadata mechanism.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",951185411,feature request: social meta tags, https://github.com/simonw/datasette/issues/1404#issuecomment-887095569,https://api.github.com/repos/simonw/datasette/issues/1404,887095569,IC_kwDOBm6k_c404AER,9599,simonw,2021-07-26T23:27:07Z,2021-07-26T23:27:07Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/docs/plugin_hooks.rst#register-routes-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",953352015,`register_routes()` hook should take `datasette` argument, https://github.com/simonw/datasette/issues/759#issuecomment-881125124,https://api.github.com/repos/simonw/datasette/issues/759,881125124,IC_kwDOBm6k_c40hOcE,9599,simonw,2021-07-16T02:11:48Z,2021-07-16T02:11:54Z,OWNER,"I added `""searchmode"": ""raw""` as a supported option for table metadata in #1389 and released that in Datasette 0.58.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612673948,fts search on a column doesn't work anymore due to escape_fts, https://github.com/simonw/datasette/issues/1394#issuecomment-881129149,https://api.github.com/repos/simonw/datasette/issues/1394,881129149,IC_kwDOBm6k_c40hPa9,9599,simonw,2021-07-16T02:23:32Z,2021-07-16T02:23:32Z,OWNER,Wrote about this in the annotated release notes for 0.58: https://simonwillison.net/2021/Jul/16/datasette-058/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by, https://github.com/simonw/datasette/issues/1231#issuecomment-881204782,https://api.github.com/repos/simonw/datasette/issues/1231,881204782,IC_kwDOBm6k_c40hh4u,9599,simonw,2021-07-16T06:14:12Z,2021-07-16T06:14:12Z,OWNER,"Here's the traceback I got from `datasette-graphql` (annoyingly only running the tests in GitHub Actions CI - I've not been able to replicate on my laptop yet): ``` tests/test_utils.py . [100%] =================================== FAILURES =================================== _________________________ test_graphql_examples[path0] _________________________ ds = path = PosixPath('/home/runner/work/datasette-graphql/datasette-graphql/examples/filters.md') @pytest.mark.asyncio @pytest.mark.parametrize( ""path"", (pathlib.Path(__file__).parent.parent / ""examples"").glob(""*.md"") ) async def test_graphql_examples(ds, path): content = path.read_text() query = graphql_re.search(content)[1] try: variables = variables_re.search(content)[1] except TypeError: variables = ""{}"" expected = json.loads(json_re.search(content)[1]) response = await ds.client.post( ""/graphql"", json={ ""query"": query, ""variables"": json.loads(variables), }, ) > assert response.status_code == 200, response.json() E AssertionError: {'data': {'repos_arraycontains': None, 'users_contains': None, 'users_date': None, 'users_endswith': None, ...}, 'erro..."", 'path': ['users_gt']}, {'locations': [{'column': 5, 'line': 34}], 'message': ""'rows'"", 'path': ['users_gte']}, ...]} E assert 500 == 200 E + where 500 = .status_code tests/test_graphql.py:142: AssertionError ----------------------------- Captured stderr call ----------------------------- table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists Traceback (most recent call last): File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py"", line 1171, in route_path response = await view(request, send) File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py"", line 151, in view request, **request.scope[""url_route""][""kwargs""] File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py"", line 123, in dispatch_request await self.ds.refresh_schemas() File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py"", line 338, in refresh_schemas await init_internal_db(internal_db) File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/utils/internal_db.py"", line 16, in init_internal_db block=True, File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 102, in execute_write return await self.execute_write_fn(_inner, block=block) File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 118, in execute_write_fn raise result File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 139, in _execute_writes result = task.fn(conn) File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 100, in _inner return conn.execute(sql, params or []) sqlite3.OperationalError: table databases already exists ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881204343,https://api.github.com/repos/simonw/datasette/issues/1231,881204343,IC_kwDOBm6k_c40hhx3,9599,simonw,2021-07-16T06:13:11Z,2021-07-16T06:13:11Z,OWNER,This just broke the `datasette-graphql` test suite: https://github.com/simonw/datasette-graphql/issues/77 - I need to figure out a solution here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881663968,https://api.github.com/repos/simonw/datasette/issues/1231,881663968,IC_kwDOBm6k_c40jR_g,9599,simonw,2021-07-16T19:18:42Z,2021-07-16T19:18:42Z,OWNER,The race condition happens inside this method - initially with the call to `await init_internal_db()`: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/app.py#L334-L359,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881664408,https://api.github.com/repos/simonw/datasette/issues/1231,881664408,IC_kwDOBm6k_c40jSGY,9599,simonw,2021-07-16T19:19:35Z,2021-07-16T19:19:35Z,OWNER,"The only place that calls `refresh_schemas()` is here: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/views/base.py#L120-L124 Ideally only one call to `refresh_schemas()` would be running at any one time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881665383,https://api.github.com/repos/simonw/datasette/issues/1231,881665383,IC_kwDOBm6k_c40jSVn,9599,simonw,2021-07-16T19:21:35Z,2021-07-16T19:21:35Z,OWNER,"https://stackoverflow.com/a/25799871/6083 has a good example of using `asyncio.Lock()`: ```python stuff_lock = asyncio.Lock() async def get_stuff(url): async with stuff_lock: if url in cache: return cache[url] stuff = await aiohttp.request('GET', url) cache[url] = stuff return stuff ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881671706,https://api.github.com/repos/simonw/datasette/issues/1231,881671706,IC_kwDOBm6k_c40jT4a,9599,simonw,2021-07-16T19:32:05Z,2021-07-16T19:32:05Z,OWNER,The test suite passes with that change.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881668759,https://api.github.com/repos/simonw/datasette/issues/1231,881668759,IC_kwDOBm6k_c40jTKX,9599,simonw,2021-07-16T19:27:46Z,2021-07-16T19:27:46Z,OWNER,"Second attempt at this: ```diff diff --git a/datasette/app.py b/datasette/app.py index 5976d8b..5f348cb 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -224,6 +224,7 @@ class Datasette: self.inspect_data = inspect_data self.immutables = set(immutables or []) self.databases = collections.OrderedDict() + self._refresh_schemas_lock = asyncio.Lock() self.crossdb = crossdb if memory or crossdb or not self.files: self.add_database(Database(self, is_memory=True), name=""_memory"") @@ -332,6 +333,12 @@ class Datasette: self.client = DatasetteClient(self) async def refresh_schemas(self): + if self._refresh_schemas_lock.locked(): + return + async with self._refresh_schemas_lock: + await self._refresh_schemas() + + async def _refresh_schemas(self): internal_db = self.databases[""_internal""] if not self.internal_db_created: await init_internal_db(internal_db) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881674857,https://api.github.com/repos/simonw/datasette/issues/1231,881674857,IC_kwDOBm6k_c40jUpp,9599,simonw,2021-07-16T19:38:39Z,2021-07-16T19:38:39Z,OWNER,I can't replicate the race condition locally with or without this patch. I'm going to push the commit and then test the CI run from `datasette-graphql` that was failing against it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1231#issuecomment-881677620,https://api.github.com/repos/simonw/datasette/issues/1231,881677620,IC_kwDOBm6k_c40jVU0,9599,simonw,2021-07-16T19:44:12Z,2021-07-16T19:44:12Z,OWNER,"That fixed the race condition in the `datasette-graphql` tests, which is the only place that I've been able to successfully replicate this. I'm going to land this change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism, https://github.com/simonw/datasette/issues/1396#issuecomment-881686662,https://api.github.com/repos/simonw/datasette/issues/1396,881686662,IC_kwDOBm6k_c40jXiG,9599,simonw,2021-07-16T20:02:44Z,2021-07-16T20:02:44Z,OWNER,Confirmed fixed: 0.58.1 was successfully published to Docker Hub in https://github.com/simonw/datasette/runs/3089447346 and the `latest` tag on https://hub.docker.com/r/datasetteproject/datasette/tags was updated.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image", https://github.com/simonw/datasette/issues/1199#issuecomment-881932880,https://api.github.com/repos/simonw/datasette/issues/1199,881932880,IC_kwDOBm6k_c40kTpQ,9599,simonw,2021-07-17T17:39:17Z,2021-07-17T17:39:17Z,OWNER,"I asked about optimizing performance on the SQLite forum and this came up as a suggestion: https://sqlite.org/forum/forumpost/9a6b9ae8e2048c8b?t=c I can start by trying this: PRAGMA mmap_size=268435456;","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792652391,Experiment with PRAGMA mmap_size=N, https://github.com/simonw/datasette/issues/123#issuecomment-882138084,https://api.github.com/repos/simonw/datasette/issues/123,882138084,IC_kwDOBm6k_c40lFvk,9599,simonw,2021-07-19T00:04:31Z,2021-07-19T00:04:31Z,OWNER,"I've been thinking more about this one today too. An extension of this (touched on in #417, Datasette Library) would be to support pointing Datasette at a directory and having it automatically load any CSV files it finds anywhere in that folder or its descendants - either loading them fully, or providing a UI that allows users to select a file to open it in Datasette. For larger files I think the right thing to do is import them into an on-disk SQLite database, which is limited only by available disk space. For smaller files loading them into an in-memory database should work fine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats, https://github.com/simonw/datasette/issues/1445#issuecomment-904024939,https://api.github.com/repos/simonw/datasette/issues/1445,904024939,IC_kwDOBm6k_c414lNr,9599,simonw,2021-08-23T18:52:35Z,2021-08-23T18:52:35Z,OWNER,"The downside of the current implementation of this trick is that it only works for exact LIKE partial matches in a specific table - if you search for `dog cat` and `dog` appears in `title` but `cat` appears in `description` you won't get back that result. I think that's fine though. If you want more advanced search there are other mechanisms you can use. This is meant to be a very quick and dirty starting point for exploring a brand new table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table, https://github.com/simonw/datasette/issues/1445#issuecomment-904026253,https://api.github.com/repos/simonw/datasette/issues/1445,904026253,IC_kwDOBm6k_c414liN,9599,simonw,2021-08-23T18:54:49Z,2021-08-23T18:54:49Z,OWNER,"The bigger problem here is UI design. This feels like a pretty niche requirement to me, so adding a prominent search box to the table page (which already has the filters interface, plus the full-text search box for tables that have FTS configured) feels untidy. I could tuck it away in the table cog menu, but that's a weird place for something like this to live. Maybe add it as a new type of filter? Filters apply to specific columns though, so this would be the first filter that applied to _all_ columns - which doesn't really fit the existing filter interface very well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table, https://github.com/simonw/datasette/issues/1445#issuecomment-904027166,https://api.github.com/repos/simonw/datasette/issues/1445,904027166,IC_kwDOBm6k_c414lwe,9599,simonw,2021-08-23T18:56:20Z,2021-08-23T18:56:20Z,OWNER,A related but potentially even more useful ability would be running a search across every column of every table in a whole database. For anything less than a few 100MB this could be incredibly useful.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table, https://github.com/simonw/datasette/issues/1445#issuecomment-904036200,https://api.github.com/repos/simonw/datasette/issues/1445,904036200,IC_kwDOBm6k_c414n9o,9599,simonw,2021-08-23T19:08:54Z,2021-08-23T19:08:54Z,OWNER,"Figured out a query for searching across every column in every table! https://til.simonwillison.net/datasette/search-all-columns-trick#user-content-same-trick-for-the-entire-database ```sql with tables as ( select name as table_name from sqlite_master where type = 'table' ), queries as ( select 'select ''' || tables.table_name || ''' as _table, rowid from ""' || tables.table_name || '"" where ' || group_concat( '""' || name || '"" like ''%'' || :search || ''%''', ' or ' ) as query from pragma_table_info(tables.table_name), tables group by tables.table_name ) select group_concat(query, ' union all ') from queries ``` The SQL query this generates for larger databases is _extremely_ long - but it does seem to work for smaller databases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table, https://github.com/simonw/datasette/issues/1445#issuecomment-904037087,https://api.github.com/repos/simonw/datasette/issues/1445,904037087,IC_kwDOBm6k_c414oLf,9599,simonw,2021-08-23T19:10:17Z,2021-08-23T19:10:17Z,OWNER,"Rather than trying to run that monstrosity in a single `union all` query, a better approach may be to use `fetch()` requests as seen in https://datasette.io/plugins/datasette-search-all","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table, https://github.com/simonw/datasette/issues/1446#issuecomment-904866495,https://api.github.com/repos/simonw/datasette/issues/1446,904866495,IC_kwDOBm6k_c417yq_,9599,simonw,2021-08-24T18:13:49Z,2021-08-24T18:13:49Z,OWNER,"OK, now the following optional CSS gives us a sticky footer: ```css html, body { height: 100%; } body { display: flex; flex-direction: column; } .not-footer { flex: 1 0 auto; } footer { flex-shrink: 0; } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978357984,Modify base.html template to support optional sticky footer, https://github.com/simonw/datasette/issues/1446#issuecomment-904954530,https://api.github.com/repos/simonw/datasette/issues/1446,904954530,IC_kwDOBm6k_c418IKi,9599,simonw,2021-08-24T20:32:47Z,2021-08-24T20:32:47Z,OWNER,"Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978357984,Modify base.html template to support optional sticky footer, https://github.com/simonw/datasette/pull/1447#issuecomment-905097468,https://api.github.com/repos/simonw/datasette/issues/1447,905097468,IC_kwDOBm6k_c418rD8,9599,simonw,2021-08-25T01:28:53Z,2021-08-25T01:28:53Z,OWNER,"Good catch, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978614898,Remove underscore from search mode parameter name, https://github.com/simonw/datasette/issues/1405#issuecomment-889525741,https://api.github.com/repos/simonw/datasette/issues/1405,889525741,IC_kwDOBm6k_c41BRXt,9599,simonw,2021-07-29T23:33:30Z,2021-07-29T23:33:30Z,OWNER,New documentation section for `datasette.utils` is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",955316250,utils.parse_metadata() should be a documented internal function, https://github.com/simonw/datasette/issues/1241#issuecomment-889539227,https://api.github.com/repos/simonw/datasette/issues/1241,889539227,IC_kwDOBm6k_c41BUqb,9599,simonw,2021-07-30T00:15:26Z,2021-07-30T00:15:26Z,OWNER,"One possible treatment: ```html

{% if query.sql and allow_execute_sql %} View and edit SQL {% endif %} Copy and share link

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,Share button for copying current URL, https://github.com/simonw/datasette/issues/1406#issuecomment-889548536,https://api.github.com/repos/simonw/datasette/issues/1406,889548536,IC_kwDOBm6k_c41BW74,9599,simonw,2021-07-30T00:43:47Z,2021-07-30T00:43:47Z,OWNER,"Still couldn't replicate on my laptop. On a hunch, I'm going to add `@pytest.mark.serial` to every test that uses `runner.isolated_filesystem()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1406#issuecomment-889547142,https://api.github.com/repos/simonw/datasette/issues/1406,889547142,IC_kwDOBm6k_c41BWmG,9599,simonw,2021-07-30T00:39:49Z,2021-07-30T00:39:49Z,OWNER,It happens in CI but not on my laptop. I think I need to run the tests on my laptop like this: https://github.com/simonw/datasette/blob/121e10c29c5b412fddf0326939f1fe46c3ad9d4a/.github/workflows/test.yml#L27-L30,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1406#issuecomment-889550391,https://api.github.com/repos/simonw/datasette/issues/1406,889550391,IC_kwDOBm6k_c41BXY3,9599,simonw,2021-07-30T00:49:31Z,2021-07-30T00:49:31Z,OWNER,That fixed it. My hunch is that Click's `runner.isolated_filesystem()` mechanism doesn't play well with `pytest-xdist`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1406#issuecomment-889553052,https://api.github.com/repos/simonw/datasette/issues/1406,889553052,IC_kwDOBm6k_c41BYCc,9599,simonw,2021-07-30T00:58:43Z,2021-07-30T00:58:43Z,OWNER,Tests are still failing in the job that calculates coverage.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1406#issuecomment-889555977,https://api.github.com/repos/simonw/datasette/issues/1406,889555977,IC_kwDOBm6k_c41BYwJ,9599,simonw,2021-07-30T01:06:57Z,2021-07-30T01:06:57Z,OWNER,"Looking at the source code in Click for `isolated_filesystem()`: https://github.com/pallets/click/blob/9da166957f5848b641231d485467f6140bca2bc0/src/click/testing.py#L450-L468 ```python @contextlib.contextmanager def isolated_filesystem( self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None ) -> t.Iterator[str]: """"""A context manager that creates a temporary directory and changes the current working directory to it. This isolates tests that affect the contents of the CWD to prevent them from interfering with each other. :param temp_dir: Create the temporary directory under this directory. If given, the created directory is not removed when exiting. .. versionchanged:: 8.0 Added the ``temp_dir`` parameter. """""" cwd = os.getcwd() t = tempfile.mkdtemp(dir=temp_dir) os.chdir(t) ``` How about if I pass in that optional `temp_dir` as a temp directory created using the `pytest-xdist` aware pytest mechanisms: https://docs.pytest.org/en/6.2.x/tmpdir.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1406#issuecomment-890259755,https://api.github.com/repos/simonw/datasette/issues/1406,890259755,IC_kwDOBm6k_c41EEkr,9599,simonw,2021-07-31T00:04:54Z,2021-07-31T00:04:54Z,OWNER,"STILL failing. I'm going to try removing all instances of `isolated_filesystem()` in favour of a different pattern using pytest temporary files, then see if I can get that to work without the serial hack. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1407#issuecomment-890388200,https://api.github.com/repos/simonw/datasette/issues/1407,890388200,IC_kwDOBm6k_c41Ej7o,9599,simonw,2021-07-31T18:38:41Z,2021-07-31T18:38:41Z,OWNER,"The `path` variable there looked like this: `/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-696/popen-gw0/uds0/datasette.sock` I think what's happening here is that `pytest-xdist` causes `tmp_path_factory.mktemp(""uds"")` to create significantly longer paths, which in this case is breaking some limit. So for this code to work with `pytest-xdist` I need to make sure the random path to `datasette.sock` is shorter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957298475,OSError: AF_UNIX path too long in ds_unix_domain_socket_server, https://github.com/simonw/datasette/issues/1407#issuecomment-890388656,https://api.github.com/repos/simonw/datasette/issues/1407,890388656,IC_kwDOBm6k_c41EkCw,9599,simonw,2021-07-31T18:42:41Z,2021-07-31T18:42:41Z,OWNER,I'll try `tempfile.gettempdir()` - on macOS it returns something like `'/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T'` which is still long but hopefully not too long.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957298475,OSError: AF_UNIX path too long in ds_unix_domain_socket_server, https://github.com/simonw/datasette/issues/1406#issuecomment-890390198,https://api.github.com/repos/simonw/datasette/issues/1406,890390198,IC_kwDOBm6k_c41Eka2,9599,simonw,2021-07-31T18:55:33Z,2021-07-31T18:55:33Z,OWNER,"To clarify: the core problem here is that an error is thrown any time you call `os.getcwd()` but the directory you are currently in has been deleted. `runner.isolated_filesystem()` assumes that the current directory in has not been deleted. But the various temporary directory utilities in `pytest` work by creating directories and then deleting them. Maybe there's a larger problem here that I play a bit fast and loose with `os.chdir()` in both the test suite and in various lines of code in Datasette itself (in particular in the publish commands)?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1406#issuecomment-890390342,https://api.github.com/repos/simonw/datasette/issues/1406,890390342,IC_kwDOBm6k_c41EkdG,9599,simonw,2021-07-31T18:56:35Z,2021-07-31T18:56:35Z,OWNER,"But... I've lost enough time to this already, and removing `runner.isolated_filesystem()` has the tests passing again. So I'm not going to work on this any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem, https://github.com/simonw/datasette/issues/1408#issuecomment-890390495,https://api.github.com/repos/simonw/datasette/issues/1408,890390495,IC_kwDOBm6k_c41Ekff,9599,simonw,2021-07-31T18:57:39Z,2021-07-31T18:57:39Z,OWNER,Opening this issue as an optional follow-up to the work I did in #1406.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957302085,"Review places in codebase that use os.chdir(), in particularly relating to tests", https://github.com/simonw/datasette/issues/1408#issuecomment-890390845,https://api.github.com/repos/simonw/datasette/issues/1408,890390845,IC_kwDOBm6k_c41Ekk9,9599,simonw,2021-07-31T19:00:32Z,2021-07-31T19:00:32Z,OWNER,"When I revisit this I can also look at dropping the `@pytest.mark.serial` hack, and maybe the `restore_working_directory()` fixture hack too: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/pytest.ini#L9-L10 https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/tests/conftest.py#L62-L75","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957302085,"Review places in codebase that use os.chdir(), in particularly relating to tests", https://github.com/simonw/datasette/issues/1409#issuecomment-890400059,https://api.github.com/repos/simonw/datasette/issues/1409,890400059,IC_kwDOBm6k_c41Em07,9599,simonw,2021-07-31T20:21:51Z,2021-07-31T20:21:51Z,OWNER,"One of these two options: - `--setting default_allow_sql off` - `--setting allow_sql_default off` Existing settings from https://docs.datasette.io/en/0.58.1/settings.html with similar names that I need to be consistent with: - `default_page_size` - `allow_facet` - `default_facet_size` - `allow_download` - `default_cache_ttl` - `default_cache_ttl_hashed` - `allow_csv_stream` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890400121,https://api.github.com/repos/simonw/datasette/issues/1409,890400121,IC_kwDOBm6k_c41Em15,9599,simonw,2021-07-31T20:22:21Z,2021-07-31T20:23:34Z,OWNER,"I think `default_allow_sql` is more consistent with the current naming conventions, because both `allow` and `default` are used as prefixes at the moment but neither of them are ever used as a suffix. Plus `default_allow_sql off` makes sense to me but `allow_default_sql off` does not - what is ""default SQL""?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890400425,https://api.github.com/repos/simonw/datasette/issues/1409,890400425,IC_kwDOBm6k_c41Em6p,9599,simonw,2021-07-31T20:25:16Z,2021-07-31T20:26:25Z,OWNER,"If I was prone to over-thinking (which I am) I'd note that `allow_facet` and `allow_download` and `allow_csv_stream` are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent. So maybe there's a future task where I introduce those as both permissions and metadata `""allow_x""` blocks, then rename the settings themselves to be called `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`. If I was going to do that I should get it in before Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890397124,https://api.github.com/repos/simonw/datasette/issues/1409,890397124,IC_kwDOBm6k_c41EmHE,9599,simonw,2021-07-31T19:51:10Z,2021-07-31T19:51:10Z,OWNER,"I think I may like `disable_sql` better. Some options: - `--setting allow_sql off` (consistent with `allow_facet` and `allow_download` and `allow_csv_stream` - all which default to `on` already) - `--setting disable_sql on` - `--setting disable_custom_sql on` The existence of three `allow_*` settings does make a strong argument for staying consistent with that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890397169,https://api.github.com/repos/simonw/datasette/issues/1409,890397169,IC_kwDOBm6k_c41EmHx,9599,simonw,2021-07-31T19:51:35Z,2021-07-31T19:51:35Z,OWNER,I'm going to stick with `--setting allow_sql off`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890397261,https://api.github.com/repos/simonw/datasette/issues/1409,890397261,IC_kwDOBm6k_c41EmJN,9599,simonw,2021-07-31T19:52:25Z,2021-07-31T19:52:25Z,OWNER,I think I can make this modification by teaching the default permissions code here to take the `allow_sql` setting into account: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/datasette/default_permissions.py#L38-L45,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890397652,https://api.github.com/repos/simonw/datasette/issues/1409,890397652,IC_kwDOBm6k_c41EmPU,9599,simonw,2021-07-31T19:56:48Z,2021-07-31T19:56:48Z,OWNER,"The other option would be to use the setting to pick the `default=` argument when calling `self.ds.permission_allowed( request.actor, ""execute-sql"", resource=database, default=True)`. The problem with that is that there are actually a few different places which perform that check, so changing all of them raises the risk of missing one in the future: https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L436-L444 https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L964-L966 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L220-L221 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L343-L345 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L134-L136 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890397753,https://api.github.com/repos/simonw/datasette/issues/1409,890397753,IC_kwDOBm6k_c41EmQ5,9599,simonw,2021-07-31T19:57:56Z,2021-07-31T19:57:56Z,OWNER,"I think the correct solution is for the default permissions logic to take the `allow_sql` setting into account, and to return `False` if that setting is set to `off` AND the current actor fails the `actor_matches_allow` checks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1409#issuecomment-890399806,https://api.github.com/repos/simonw/datasette/issues/1409,890399806,IC_kwDOBm6k_c41Emw-,9599,simonw,2021-07-31T20:18:46Z,2021-07-31T20:18:46Z,OWNER,"My rationale for removing it: https://github.com/simonw/datasette/issues/813#issuecomment-640916290 > Naming problem: Datasette already has a config option with this name: > > $ datasette serve data.db --config allow_sql:1 > > https://datasette.readthedocs.io/en/stable/config.html#allow-sql > > It's confusing to have two things called `allow_sql` that do slightly different things. > > I could retire the `--config allow_sql:0` option entirely, since the new `metadata.json` mechanism can be used to achieve the exact same thing. > > I'm going to do that. This is true. The `""allow_sql""` permissions block in `metadata.json` does indeed have a name that is easily confused with `--setting allow_sql off`. So I definitely need to pick a different name from the setting. `--setting default_allow_sql off` is a good option here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting), https://github.com/simonw/datasette/issues/1411#issuecomment-890441844,https://api.github.com/repos/simonw/datasette/issues/1411,890441844,IC_kwDOBm6k_c41ExB0,9599,simonw,2021-08-01T03:27:30Z,2021-08-01T03:27:30Z,OWNER,Confirmed: https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1 no longer exhibits the bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957345476,Canned query ?sql= is pointlessly echoed in query string starting from hidden mode, https://github.com/simonw/datasette/issues/1227#issuecomment-891352132,https://api.github.com/repos/simonw/datasette/issues/1227,891352132,IC_kwDOBm6k_c41IPRE,9599,simonw,2021-08-02T21:38:39Z,2021-08-02T21:38:39Z,OWNER,Relevant TIL: https://til.simonwillison.net/vscode/vs-code-regular-expressions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810394616,Configure sphinx.ext.extlinks for issues, https://github.com/simonw/datasette/issues/1417#issuecomment-891979858,https://api.github.com/repos/simonw/datasette/issues/1417,891979858,IC_kwDOBm6k_c41KohS,9599,simonw,2021-08-03T16:15:30Z,2021-08-03T16:15:30Z,OWNER,"Docs: https://pypi.org/project/codespell/ There's a `codespell --ignore-words=FILE` option for ignoring words. I don't have any that need ignoring yet but I'm going to add that file anyway, that way I can have codespell be a failing test but still provide a way to work around it if it incorrectly flags a correct word.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959278472,Use codespell in CI to spot spelling errors, https://github.com/simonw/datasette/pull/1418#issuecomment-891984359,https://api.github.com/repos/simonw/datasette/issues/1418,891984359,IC_kwDOBm6k_c41Kpnn,9599,simonw,2021-08-03T16:21:56Z,2021-08-03T16:21:56Z,OWNER,"Failed with: ``` docs/authentication.rst:63: perfom ==> perform docs/authentication.rst:76: perfom ==> perform docs/changelog.rst:429: repsonse ==> response docs/changelog.rst:503: permissons ==> permissions docs/changelog.rst:717: compatibilty ==> compatibility docs/changelog.rst:1172: browseable ==> browsable docs/deploying.rst:191: similiar ==> similar docs/internals.rst:434: Respons ==> Response, respond docs/internals.rst:440: Respons ==> Response, respond docs/internals.rst:717: tha ==> than, that, the docs/performance.rst:42: databse ==> database docs/plugin_hooks.rst:667: utilites ==> utilities docs/publish.rst:168: countainer ==> container docs/settings.rst:352: inalid ==> invalid docs/sql_queries.rst:406: preceeded ==> preceded, proceeded ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959284434,Spelling corrections plus CI job for codespell, https://github.com/simonw/datasette/issues/1420#issuecomment-892365639,https://api.github.com/repos/simonw/datasette/issues/1420,892365639,IC_kwDOBm6k_c41MGtH,9599,simonw,2021-08-04T05:05:07Z,2021-08-04T05:05:07Z,OWNER,https://github.com/simonw/datasette/blob/cd8b7bee8fb5c1cdce7c8dbfeb0166011abc72c6/datasette/publish/cloudrun.py#L153-L158,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option, https://github.com/simonw/datasette/issues/1420#issuecomment-892372509,https://api.github.com/repos/simonw/datasette/issues/1420,892372509,IC_kwDOBm6k_c41MIYd,9599,simonw,2021-08-04T05:22:29Z,2021-08-04T05:22:29Z,OWNER,"Testing this manually with: datasette publish cloudrun fixtures.db --memory 8G --cpu 4 \ --service fixtures-over-provisioned-issue-1420 --install datasette-psutil And for comparison: datasette publish cloudrun fixtures.db --service fixtures-default-issue-1420 \ --install datasette-psutil ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option, https://github.com/simonw/datasette/issues/1420#issuecomment-892374253,https://api.github.com/repos/simonw/datasette/issues/1420,892374253,IC_kwDOBm6k_c41MIzt,9599,simonw,2021-08-04T05:27:21Z,2021-08-04T05:29:59Z,OWNER,"I'll delete these services shortly afterwards. Right now: https://fixtures-over-provisioned-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil (deployed with `--memory 8G --cpu 4`) returns: ``` process.memory_info() pmem(rss=60456960, vms=518930432, shared=0, text=0, lib=0, data=0, dirty=0) ... psutil.cpu_times(True) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) psutil.virtual_memory() svmem(total=2147483648, available=2092531712, percent=2.6, used=33103872, free=2092531712, active=44130304, inactive=10792960, buffers=0, cached=21848064, shared=262144, slab=0) ``` https://fixtures-default-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil returns: ``` process.memory_info() pmem(rss=49324032, vms=140595200, shared=0, text=0, lib=0, data=0, dirty=0) ... psutil.cpu_times(True) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) psutil.virtual_memory() svmem(total=2147483648, available=2091188224, percent=2.6, used=40071168, free=2091188224, active=41586688, inactive=7983104, buffers=0, cached=16224256, shared=262144, slab=0) ``` These numbers are different enough that I assume this works as advertised. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option, https://github.com/simonw/datasette/issues/1420#issuecomment-892376353,https://api.github.com/repos/simonw/datasette/issues/1420,892376353,IC_kwDOBm6k_c41MJUh,9599,simonw,2021-08-04T05:33:12Z,2021-08-04T05:33:12Z,OWNER,"In the Cloud Run console (before I deleted these services) when I click ""Edit and deploy new revision"" I see this for the default one: And this for the big one: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option, https://github.com/simonw/datasette/issues/1420#issuecomment-893079520,https://api.github.com/repos/simonw/datasette/issues/1420,893079520,IC_kwDOBm6k_c41O0_g,9599,simonw,2021-08-05T00:54:59Z,2021-08-05T00:54:59Z,OWNER,"Just saw this error: `ERROR: (gcloud.run.deploy) The `--cpu` flag is not supported on the fully managed version of Cloud Run. Specify `--platform gke` or run `gcloud config set run/platform gke` to work with Cloud Run for Anthos deployed on Google Cloud.` Which is weird because I managed to run this successfully the other day. Maybe a region difference thing?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option, https://github.com/simonw/datasette/issues/1422#issuecomment-893122356,https://api.github.com/repos/simonw/datasette/issues/1422,893122356,IC_kwDOBm6k_c41O_c0,9599,simonw,2021-08-05T02:52:31Z,2021-08-05T02:52:44Z,OWNER,"If you do this it should still be possible to view the SQL - which means we need a new parameter. I propose `?_show_sql=1` to over-ride the hidden default. I think the configuration should use `hide_sql: true` - looking like this: ```yaml databases: fixtures: queries: neighborhood_search: hide_sql: true sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood title: Search neighborhoods ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query, https://github.com/simonw/datasette/issues/1422#issuecomment-893131703,https://api.github.com/repos/simonw/datasette/issues/1422,893131703,IC_kwDOBm6k_c41PBu3,9599,simonw,2021-08-05T03:16:46Z,2021-08-05T03:16:46Z,OWNER,"The logic for this is a little bit fiddly, due to the need to switch to using `?_show_sql=1` on the link depending on the context. - If metadata says hide and there's no query string, hide and link to `?_show_sql=1` - If metadata says hide but query string says `?_show_sql=1`, show and have hide link linking to URL without `?_show_sql=1` - Otherwise, show and link to `?_hide_sql=1` - ... or if that query string is there then hide and link to URL without `?_hide_sql=1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query, https://github.com/simonw/datasette/issues/1419#issuecomment-893133496,https://api.github.com/repos/simonw/datasette/issues/1419,893133496,IC_kwDOBm6k_c41PCK4,9599,simonw,2021-08-05T03:22:44Z,2021-08-05T03:22:44Z,OWNER,"I ran into this exact same problem today! I only just learned how to use filter on aggregates: https://til.simonwillison.net/sqlite/sqlite-aggregate-filter-clauses A workaround I used is to add this to the deploy command: datasette publish cloudrun ... --install=pysqlite3-binary This will install the https://pypi.org/project/pysqlite3-binary for package which bundles a more recent SQLite version.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959710008,`publish cloudrun` should deploy a more recent SQLite version, https://github.com/simonw/datasette/issues/1423#issuecomment-893996604,https://api.github.com/repos/simonw/datasette/issues/1423,893996604,IC_kwDOBm6k_c41SU48,9599,simonw,2021-08-06T04:43:07Z,2021-08-06T04:43:37Z,OWNER,"Problem: on a page which doesn't have quite enough facet values to trigger the display of the ""..."" link that links to `?_facet_size=max` the user would still have to manually count the values - up to 30 by default. So maybe the count should always be shown, perhaps as a non-bold light colored number? I could even hide it in a non-discoverable tooltip.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1423#issuecomment-894452990,https://api.github.com/repos/simonw/datasette/issues/1423,894452990,IC_kwDOBm6k_c41UET-,9599,simonw,2021-08-06T18:49:37Z,2021-08-06T18:49:37Z,OWNER,"Could display them always, like this: That's with `23`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1423#issuecomment-894453520,https://api.github.com/repos/simonw/datasette/issues/1423,894453520,IC_kwDOBm6k_c41UEcQ,9599,simonw,2021-08-06T18:50:40Z,2021-08-06T18:50:40Z,OWNER,"Point of confusion: if only 30 options are shown, but there's a `...` at the end, what would the number be? It can't be the total number of facets because we haven't counted them all - but if it's just the number of displayed facets that's like to be confusing. So the original idea of showing the counts only if `_facet_size=max` is a good one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1423#issuecomment-894454087,https://api.github.com/repos/simonw/datasette/issues/1423,894454087,IC_kwDOBm6k_c41UElH,9599,simonw,2021-08-06T18:51:42Z,2021-08-06T18:51:42Z,OWNER,"The invisible tooltip could say ""Showing 30 items, more available"" (helping save you from counting up to 20 if you know about the secret feature). The numbers could then be fully displayed on the ""..."" page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1423#issuecomment-894454644,https://api.github.com/repos/simonw/datasette/issues/1423,894454644,IC_kwDOBm6k_c41UEt0,9599,simonw,2021-08-06T18:52:49Z,2021-08-06T18:52:49Z,OWNER,"This means that the counts would be unavailable to users who cannot see tooltips (e.g. mobile users) on pages that did not have any facets that broke the 30 limit and hence displayed that ""..."" link. I think I'm OK with that, for the moment. May revisit in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1422#issuecomment-894589140,https://api.github.com/repos/simonw/datasette/issues/1422,894589140,IC_kwDOBm6k_c41UljU,9599,simonw,2021-08-07T01:58:16Z,2021-08-07T01:58:24Z,OWNER,Also need to consider this hidden field - it should pass the `_hide_sql` or `_show_sql` parameters depending on the same logic: https://github.com/simonw/datasette/blob/acc22436622ff8476c30acf45ed60f54b4aaa5d9/datasette/templates/query.html#L47-L49,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query, https://github.com/simonw/datasette/issues/1421#issuecomment-894606843,https://api.github.com/repos/simonw/datasette/issues/1421,894606843,IC_kwDOBm6k_c41Up37,9599,simonw,2021-08-07T05:17:12Z,2021-08-07T05:17:12Z,OWNER,Marking this blocked because I don't have a way around the needing-a-SQLite-SQL-parser problem at the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894606796,https://api.github.com/repos/simonw/datasette/issues/1421,894606796,IC_kwDOBm6k_c41Up3M,9599,simonw,2021-08-07T05:16:39Z,2021-08-07T05:16:39Z,OWNER,"Urgh, yeah I've seen this one before. Fixing it pretty much requires writing a full SQLite SQL syntax parser in Python, which is frustratingly complicated for solving this issue! You can work around this for a canned query by using the optional `params:` argument documented here: https://docs.datasette.io/en/stable/sql_queries.html#canned-query-parameters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1422#issuecomment-894607989,https://api.github.com/repos/simonw/datasette/issues/1422,894607989,IC_kwDOBm6k_c41UqJ1,9599,simonw,2021-08-07T05:31:57Z,2021-08-07T05:31:57Z,OWNER,"Demo: https://latest.datasette.io/fixtures/neighborhood_search Documentation: https://docs.datasette.io/en/latest/sql_queries.html#additional-canned-query-options","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query, https://github.com/simonw/datasette/issues/1421#issuecomment-894922145,https://api.github.com/repos/simonw/datasette/issues/1421,894922145,IC_kwDOBm6k_c41V22h,9599,simonw,2021-08-09T03:07:38Z,2021-08-09T03:07:38Z,OWNER,"I hoped this would work: ```sql with foo as ( explain select * from facetable where state = :state and on_earth = :on_earth and neighborhood not like '00:04' ) select p4 from foo where opcode = 'Variable' ``` But sadly [it returns an error](https://latest.datasette.io/fixtures?sql=with+foo+as+%28%0D%0A++explain+select+*+from+facetable%0D%0A++where+state+%3D+%3Astate%0D%0A++and+on_earth+%3D+%3Aon_earth%0D%0A++and+neighborhood+not+like+%2700%3A04%27%0D%0A%29%0D%0Aselect+p4+from+foo+where+opcode+%3D+%27Variable%27&state=&on_earth=&04=): > near ""explain"": syntax error","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894922703,https://api.github.com/repos/simonw/datasette/issues/1421,894922703,IC_kwDOBm6k_c41V2_P,9599,simonw,2021-08-09T03:09:29Z,2021-08-09T03:09:29Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/ad90a72afa21b737b162e2bbdddc301a97d575cd/datasette/views/database.py#L225-L231,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894921512,https://api.github.com/repos/simonw/datasette/issues/1421,894921512,IC_kwDOBm6k_c41V2so,9599,simonw,2021-08-09T03:05:26Z,2021-08-09T03:05:26Z,OWNER,"I may have a way to work around this, using `explain`. Consider this query: ```sql select * from facetable where state = :state and on_earth = :on_earth and neighborhood not like '00:04' ``` Datasette currently gets confused and shows three form fields: https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04= But... if I run `explain` [against that](https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04=) I get this (truncated): addr | opcode | p1 | p2 | p3 | p4 | p5 | comment -- | -- | -- | -- | -- | -- | -- | -- 20 | ResultRow | 6 | 10 | 0 |   | 0 |   21 | Next | 0 | 3 | 0 |   | 1 |   22 | Halt | 0 | 0 | 0 |   | 0 |   23 | Transaction | 0 | 0 | 35 | 0 | 1 |   24 | Variable | 1 | 2 | 0 | :state | 0 |   25 | Variable | 2 | 3 | 0 | :on_earth | 0 |   26 | String8 | 0 | 4 | 0 | 00:04 | 0 |   27 | Goto | 0 | 1 | 0 |   | 0 |   Could it be as simple as pulling out those `Variable` rows to figure out the names of the variables in the query?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894925437,https://api.github.com/repos/simonw/datasette/issues/1421,894925437,IC_kwDOBm6k_c41V3p9,9599,simonw,2021-08-09T03:19:00Z,2021-08-09T03:19:00Z,OWNER,"This may not work: > `ERROR: sql = 'explain select 1 + :one + :two', params = None: You did not supply a value for binding 1.` The `explain` queries themselves want me to pass them parameters. I could try using the regex to pull out candidates and passing `None` for each of those, including incorrect ones like `:31`. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894925914,https://api.github.com/repos/simonw/datasette/issues/1421,894925914,IC_kwDOBm6k_c41V3xa,9599,simonw,2021-08-09T03:20:42Z,2021-08-09T03:20:42Z,OWNER,"I think this works! ```python _re_named_parameter = re.compile("":([a-zA-Z0-9_]+)"") async def derive_named_parameters(db, sql): explain = 'explain {}'.format(sql.strip().rstrip("";"")) possible_params = _re_named_parameter.findall(sql) try: results = await db.execute(explain, {p: None for p in possible_params}) return [row[""p4""].lstrip("":"") for row in results if row[""opcode""] == ""Variable""] except sqlite3.DatabaseError: return [] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894927185,https://api.github.com/repos/simonw/datasette/issues/1421,894927185,IC_kwDOBm6k_c41V4FR,9599,simonw,2021-08-09T03:25:01Z,2021-08-09T03:25:01Z,OWNER,"One catch with this approach: if the SQL query is invalid, the parameters will not be extracted and shown as form fields. Maybe that's completely fine? Why display a form if it's going to break when the user actually runs the query? But it does bother me. I worry that someone who is still iterating on and editing their query before actually starting to use it might find the behaviour confusing. So maybe if the query raises an exception it could fall back on the regular expression results?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894929080,https://api.github.com/repos/simonw/datasette/issues/1421,894929080,IC_kwDOBm6k_c41V4i4,9599,simonw,2021-08-09T03:33:02Z,2021-08-09T03:33:02Z,OWNER,"Fixed! Fantastic, this one has been bothering me for *years*. https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894929769,https://api.github.com/repos/simonw/datasette/issues/1421,894929769,IC_kwDOBm6k_c41V4tp,9599,simonw,2021-08-09T03:36:49Z,2021-08-09T03:36:49Z,OWNER,"SQLite carries a warning about using `EXPLAIN` like this: https://www.sqlite.org/lang_explain.html > The output from EXPLAIN and EXPLAIN QUERY PLAN is intended for interactive analysis and troubleshooting only. The details of the output format are subject to change from one release of SQLite to the next. Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented. I think that's OK here, because of the regular expression fallback. If the format changes in the future in a way that breaks the query the error should be caught and the regex-captured parameters should be returned instead. Hmmm... actually that's not entirely true: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L1084-L1091 If the format changes such that the same columns are returned but the `[row[""p4""].lstrip("":"") for row in results if row[""opcode""] == ""Variable""]` list comprehension returns an empty array it will break Datasette! I'm going to take that risk for the moment, but I'll actively watch out for problems in the future. If this does turn out to be bad I can always go back to the pure regular expression mechanism. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894930013,https://api.github.com/repos/simonw/datasette/issues/1421,894930013,IC_kwDOBm6k_c41V4xd,9599,simonw,2021-08-09T03:38:06Z,2021-08-09T03:38:06Z,OWNER,"Amusing edge-case: if you run this against a `explain ...` query it falls back to using regular expressions, because `explain explain select ...` is invalid SQL. https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1425#issuecomment-894865323,https://api.github.com/repos/simonw/datasette/issues/1425,894865323,IC_kwDOBm6k_c41Vo-r,9599,simonw,2021-08-08T22:33:19Z,2021-08-08T22:33:19Z,OWNER,"I can do this with the `await_me_maybe()` function, as seen here: https://github.com/simonw/datasette/blob/a21853c9dade240734abc6b4f750fae09a3e840a/datasette/app.py#L864-L873","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1424#issuecomment-894864682,https://api.github.com/repos/simonw/datasette/issues/1424,894864682,IC_kwDOBm6k_c41Vo0q,9599,simonw,2021-08-08T22:26:46Z,2021-08-08T22:26:46Z,OWNER,"Note that the `sqlite3` exceptions are in `sqlite3` if using the Python standard library but are in `pysqlite3` if that module is being used instead. So maybe encourage people to use them from `datasette.sqlite.sqlite3` instead, which will point to the correct package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends, https://github.com/simonw/datasette/issues/1424#issuecomment-894864744,https://api.github.com/repos/simonw/datasette/issues/1424,894864744,IC_kwDOBm6k_c41Vo1o,9599,simonw,2021-08-08T22:27:31Z,2021-08-08T22:27:31Z,OWNER,https://docs.python.org/3/library/sqlite3.html#exceptions is useful - it looks like `sqlite3.DatabaseError` is the super-class of all of the other exceptions that we might see.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends, https://github.com/simonw/datasette/issues/1424#issuecomment-894864404,https://api.github.com/repos/simonw/datasette/issues/1424,894864404,IC_kwDOBm6k_c41VowU,9599,simonw,2021-08-08T22:24:06Z,2021-08-08T22:24:06Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/de5ce2e56339ad8966f417a4758f7c210c017dec/datasette/database.py#L176-L200,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends, https://github.com/simonw/datasette/issues/1424#issuecomment-894864616,https://api.github.com/repos/simonw/datasette/issues/1424,894864616,IC_kwDOBm6k_c41Vozo,9599,simonw,2021-08-08T22:26:08Z,2021-08-08T22:26:08Z,OWNER,"- `datasette.database.QueryInterrupted` for queries that were interrupted - `sqlite3.OperationalError` - `sqlite3.DatabaseError` and more","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends, https://github.com/simonw/datasette/issues/1425#issuecomment-894869692,https://api.github.com/repos/simonw/datasette/issues/1425,894869692,IC_kwDOBm6k_c41VqC8,9599,simonw,2021-08-08T23:08:29Z,2021-08-08T23:08:29Z,OWNER,Updated documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-value-column-table-database-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894881448,https://api.github.com/repos/simonw/datasette/issues/1425,894881448,IC_kwDOBm6k_c41Vs6o,9599,simonw,2021-08-09T00:24:25Z,2021-08-09T00:24:39Z,OWNER,"My hunch is that the ""skip this `render_cell()` result if it returns `None`"" logic isn't working correctly, ever since I added the `await_me_maybe` line. Could that be because Pluggy handles the ""do the next if `None` is returned"" logic itself, but I'm no-longer returning `None`, I'm returning an awaitable which when awaited returns `None`. This would suggest that all of the `await_me_maybe()` plugin hooks have the same bug. That's definitely possible - it may well be that no-one has yet stumbled across a bug caused by a plugin returning an awaitable and hence not being skipped, because plugin hooks that return awaitable are rare enough that no-one has tried two plugins which both use that trick. Still don't see why it would pass on my laptop but fail in CI though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894881016,https://api.github.com/repos/simonw/datasette/issues/1425,894881016,IC_kwDOBm6k_c41Vsz4,9599,simonw,2021-08-09T00:21:53Z,2021-08-09T00:21:53Z,OWNER,"Still one test failure: ``` def test_hook_render_cell_link_from_json(app_client): sql = """""" select '{""href"": ""http://example.com/"", ""label"":""Example""}' """""".strip() path = ""/fixtures?"" + urllib.parse.urlencode({""sql"": sql}) response = app_client.get(path) td = Soup(response.body, ""html.parser"").find(""table"").find(""tbody"").find(""td"") a = td.find(""a"") > assert a is not None, str(a) E AssertionError: None E assert None is not None ``` The weird thing about this one is that I can't replicate it on my laptop - but it happens in CI every time, including when I shell in and try to run that single test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894882123,https://api.github.com/repos/simonw/datasette/issues/1425,894882123,IC_kwDOBm6k_c41VtFL,9599,simonw,2021-08-09T00:27:43Z,2021-08-09T00:27:43Z,OWNER,"Good news: `render_cell()` is the only hook to use `firstresult=True`: https://github.com/simonw/datasette/blob/f3c9edb376a13c09b5ecf97c7390f4e49efaadf2/datasette/hookspecs.py#L62-L64 https://pluggy.readthedocs.io/en/latest/#first-result-only","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894882642,https://api.github.com/repos/simonw/datasette/issues/1425,894882642,IC_kwDOBm6k_c41VtNS,9599,simonw,2021-08-09T00:29:57Z,2021-08-09T00:29:57Z,OWNER,"Here's the code in `pluggy` that implements this: https://github.com/pytest-dev/pluggy/blob/0a064fe275060dbdb1fe6e10c888e72bc400fb33/src/pluggy/callers.py#L31-L43 ```python if hook_impl.hookwrapper: try: gen = hook_impl.function(*args) next(gen) # first yield teardowns.append(gen) except StopIteration: _raise_wrapfail(gen, ""did not yield"") else: res = hook_impl.function(*args) if res is not None: results.append(res) if firstresult: # halt further impl calls break ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894883664,https://api.github.com/repos/simonw/datasette/issues/1425,894883664,IC_kwDOBm6k_c41VtdQ,9599,simonw,2021-08-09T00:33:56Z,2021-08-09T00:33:56Z,OWNER,"I could extract that code out and write my own function which implements the equivalent of calling `pm.hook.render_cell(...)` but runs `await_me_maybe()` before checking if `res is not None`. That's pretty nasty. Could I instead call the plugin hook normally, but then have additional logic which says ""if I await it and it returns `None` then try calling the hook again but skip this one"" - not sure if there's a way to do that either. I could remove the `firstresult=True` from the hookspec - which would cause it to call and return ALL hooks - but then in my own code use only the first one. This is slightly less efficient (since it calls all the hooks and then discards all-but-one value) but it's the least unpleasant in terms of the code I would have to write - plus I don't think it's going to be THAT common for someone to have multiple expensive `render_cell()` hooks installed at once (they are usually pretty cheap).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894884874,https://api.github.com/repos/simonw/datasette/issues/1425,894884874,IC_kwDOBm6k_c41VtwK,9599,simonw,2021-08-09T00:38:20Z,2021-08-09T00:38:20Z,OWNER,I'm trying the version where I remove `firstresult=True`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894893319,https://api.github.com/repos/simonw/datasette/issues/1425,894893319,IC_kwDOBm6k_c41Vv0H,9599,simonw,2021-08-09T01:08:56Z,2021-08-09T01:09:12Z,OWNER,Demo: https://latest.datasette.io/fixtures/simple_primary_key shows `RENDER_CELL_ASYNC_RESULT` where the CSV version shows `RENDER_CELL_ASYNC`: https://latest.datasette.io/fixtures/simple_primary_key.csv - because of this test plugin code: https://github.com/simonw/datasette/blob/a390bdf9cef01d8723d025fc3348e81345ff4856/tests/plugins/my_plugin.py#L98-L122,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894900267,https://api.github.com/repos/simonw/datasette/issues/1425,894900267,IC_kwDOBm6k_c41Vxgr,9599,simonw,2021-08-09T01:31:22Z,2021-08-09T01:31:22Z,OWNER,"I used this to build a new plugin: https://github.com/simonw/datasette-query-links Demo here: https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++%27select+*+from+[facetable]%27+as+query%0D%0Aunion%0D%0Aselect%0D%0A++%27select+sqlite_version()%27%0D%0Aunion%0D%0Aselect%0D%0A++%27select+this+is+invalid+SQL+so+will+not+be+linked%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1426#issuecomment-895500565,https://api.github.com/repos/simonw/datasette/issues/1426,895500565,IC_kwDOBm6k_c41YEEV,9599,simonw,2021-08-09T20:00:04Z,2021-08-09T20:00:04Z,OWNER,"A few options for how this would work: - `datasette ... --robots allow` - `datasette ... --setting robots allow` Options could be: - `allow` - allow all crawling - `deny` - deny all crawling - `limited` - allow access to the homepage and the index pages for each database and each table, but disallow crawling any further than that The ""limited"" mode is particularly interesting. Could even make it the default, but I think that may be a bit too confusing. Idea would be to get the key pages indexed but use `nofollow` to discourage crawlers from indexing individual row pages or deep pages like `https://datasette.io/content/repos?_facet=owner&_facet=language&_facet_array=topics&topics__arraycontains=sqlite#facet-owner`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-895509536,https://api.github.com/repos/simonw/datasette/issues/1426,895509536,IC_kwDOBm6k_c41YGQg,9599,simonw,2021-08-09T20:12:57Z,2021-08-09T20:12:57Z,OWNER,I could try out the `X-Robots` HTTP header too: https://developers.google.com/search/docs/advanced/robots/robots_meta_tag#xrobotstag,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-895510773,https://api.github.com/repos/simonw/datasette/issues/1426,895510773,IC_kwDOBm6k_c41YGj1,9599,simonw,2021-08-09T20:14:50Z,2021-08-09T20:19:22Z,OWNER,"https://twitter.com/mal/status/1424825895139876870 > True pinging google should be part of the build process on a static site :) That's another aspect of this: if you DO want your site crawled, teaching the `datasette publish` command how to ping Google when a deploy has gone out could be a nice improvement. Annoyingly it looks like you need to configure an auth token of some sort in order to use their API though, which is likely too much hassle to be worth building into Datasette itself: https://developers.google.com/search/apis/indexing-api/v3/using-api ``` curl -X POST https://indexing.googleapis.com/v3/urlNotifications:publish -d '{ ""url"": ""https://careers.google.com/jobs/google/technical-writer"", ""type"": ""URL_UPDATED"" }' -H ""Content-Type: application/json"" { ""error"": { ""code"": 401, ""message"": ""Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project."", ""status"": ""UNAUTHENTICATED"" } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-895522818,https://api.github.com/repos/simonw/datasette/issues/1426,895522818,IC_kwDOBm6k_c41YJgC,9599,simonw,2021-08-09T20:34:10Z,2021-08-09T20:34:10Z,OWNER,At the very least Datasette should serve a blank `/robots.txt` by default - I'm seeing a ton of 404s for it in the logs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/859#issuecomment-905900807,https://api.github.com/repos/simonw/datasette/issues/859,905900807,IC_kwDOBm6k_c41_vMH,9599,simonw,2021-08-25T21:51:10Z,2021-08-25T21:51:10Z,OWNER,"10-20 minutes to populate `_internal`! How many databases and tables is that for? I may have to rethink the `_internal` mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed. I need to get my own stress testing rig setup for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts), https://github.com/simonw/datasette/issues/1293#issuecomment-898063815,https://api.github.com/repos/simonw/datasette/issues/1293,898063815,IC_kwDOBm6k_c41h13H,9599,simonw,2021-08-13T00:33:17Z,2021-08-13T00:33:17Z,OWNER,"Improved version of that function: ```python def columns_for_query(conn, sql): """""" Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of ``(table_name, column_name)`` pairs, one per returned column. ``(None, None)`` if no table and column could be derived. """""" rows = conn.execute('explain ' + sql).fetchall() table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'} names_by_rootpage = dict( conn.execute( 'select rootpage, name from sqlite_master where rootpage in ({})'.format( ', '.join(map(str, table_rootpage_by_register.values())) ) ) ) columns_by_column_register = {} for row in rows: if row['opcode'] in ('Rowid', 'Column'): addr, opcode, table_id, cid, column_register, p4, p5, comment = row table = names_by_rootpage[table_rootpage_by_register[table_id]] columns_by_column_register[column_register] = (table, cid) result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0] registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""])) all_column_names = {} for table in names_by_rootpage.values(): table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall() for row in table_xinfo: all_column_names[(table, row[""cid""])] = row[""name""] final_output = [] for r in registers: try: table, cid = columns_by_column_register[r] final_output.append((table, all_column_names[table, cid])) except KeyError: final_output.append((None, None)) return final_output ``` It works! ```diff diff --git a/datasette/templates/query.html b/datasette/templates/query.html index 75f7f1b..9fe1d4f 100644 --- a/datasette/templates/query.html +++ b/datasette/templates/query.html @@ -67,6 +67,8 @@

+extra_column_info: {{ extra_column_info }} + {% if display_rows %}

This data as {% for name, url in renderers.items() %}{{ name }}{{ "", "" if not loop.last }}{% endfor %}, CSV

diff --git a/datasette/views/database.py b/datasette/views/database.py index 7c36034..02f8039 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -10,6 +10,7 @@ import markupsafe from datasette.utils import ( await_me_maybe, check_visibility, + columns_for_query, derive_named_parameters, to_css_class, validate_sql_select, @@ -248,6 +249,8 @@ class QueryView(DataView): query_error = None + extra_column_info = None + # Execute query - as write or as read if write: if request.method == ""POST"": @@ -334,6 +337,10 @@ class QueryView(DataView): database, sql, params_for_query, truncate=True, **extra_args ) columns = [r[0] for r in results.description] + + # Try to figure out extra column information + db = self.ds.get_database(database) + extra_column_info = await db.execute_fn(lambda conn: columns_for_query(conn, sql)) except sqlite3.DatabaseError as e: query_error = e results = None @@ -462,6 +469,7 @@ class QueryView(DataView): ""show_hide_text"": show_hide_text, ""show_hide_hidden"": markupsafe.Markup(show_hide_hidden), ""hide_sql"": hide_sql, + ""extra_column_info"": extra_column_info, } return ( ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898065011,https://api.github.com/repos/simonw/datasette/issues/1293,898065011,IC_kwDOBm6k_c41h2Jz,9599,simonw,2021-08-13T00:36:30Z,2021-08-13T00:36:30Z,OWNER,"> https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because `paginated_view` is defined like this: > > ```sql > CREATE VIEW paginated_view AS > SELECT > content, > '- ' || content || ' -' AS content_extra > FROM no_primary_key; > ``` > > So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression. Here's what it does for that: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898065948,https://api.github.com/repos/simonw/datasette/issues/1293,898065948,IC_kwDOBm6k_c41h2Yc,9599,simonw,2021-08-13T00:38:58Z,2021-08-13T00:38:58Z,OWNER,"Trying to run `explain select * from facetable` fails with an error in my prototype, because it tries to execute `explain explain select * from facetable` - so I need to spot that error and ignore it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898066466,https://api.github.com/repos/simonw/datasette/issues/1293,898066466,IC_kwDOBm6k_c41h2gi,9599,simonw,2021-08-13T00:40:24Z,2021-08-13T00:40:24Z,OWNER,"It figures out renamed columns too: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1431#issuecomment-898072940,https://api.github.com/repos/simonw/datasette/issues/1431,898072940,IC_kwDOBm6k_c41h4Fs,9599,simonw,2021-08-13T00:58:40Z,2021-08-13T00:58:40Z,OWNER,"While I'm doing this I should rename this internal variable to avoid confusion in the future: https://github.com/simonw/datasette/blob/e837095ef35ae155b4c78cc9a8b7133a48c94f03/datasette/app.py#L203","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969840302,`--help-config` should be called `--help-settings`, https://github.com/simonw/datasette/issues/1432#issuecomment-898074849,https://api.github.com/repos/simonw/datasette/issues/1432,898074849,IC_kwDOBm6k_c41h4jh,9599,simonw,2021-08-13T01:03:40Z,2021-08-13T01:03:40Z,OWNER,"Also this method: https://github.com/simonw/datasette/blob/77f46297a88ac7e49dad2139410b01ee56d5f99c/datasette/app.py#L422-L424 And the places that use it: https://github.com/simonw/datasette/blob/fc4846850fffd54561bc125332dfe97bb41ff42e/datasette/views/base.py#L617 https://github.com/simonw/datasette/blob/fc4846850fffd54561bc125332dfe97bb41ff42e/datasette/views/database.py#L459 Which is used in this template: https://github.com/simonw/datasette/blob/77f46297a88ac7e49dad2139410b01ee56d5f99c/datasette/templates/table.html#L204 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/issues/1432#issuecomment-898079507,https://api.github.com/repos/simonw/datasette/issues/1432,898079507,IC_kwDOBm6k_c41h5sT,9599,simonw,2021-08-13T01:08:42Z,2021-08-13T01:09:41Z,OWNER,"This is going to break some plugins: https://ripgrep.datasette.io/-/ripgrep?pattern=config%3D&literal=on&glob=%21datasette%2F** > ### datasette-cluster-map/tests/test_cluster_map.py > > @pytest.mark.asyncio > > async def test_respects_base_url(): > ds = Datasette([], memory=True, config={""base_url"": ""/foo/""}) > response = await ds.client.get(""/:memory:?sql=select+1+as+latitude,+2+as+longitude"") > assert ( > > ### datasette-export-notebook/tests/test_export_notebook.py > > @pytest.mark.asyncio > > async def test_notebook_no_csv(db_path): > datasette = Datasette([db_path], config={""allow_csv_stream"": False}) > response = await datasette.client.get(""/db/big.Notebook"") > assert "".csv"" not in response.text > > ### datasette-publish-vercel/tests/test_publish_vercel.py > metadata=metadata, > cors=True, > config={""default_page_size"": 10, ""sql_time_limit_ms"": 2000} > ).app() > """""" > > ### datasette-publish-vercel/datasette_publish_vercel/__init__.py > metadata=metadata{extras}, > cors=True, > config={settings} > > ).app() > > """""".strip() > > ### datasette-search-all/tests/test_search_all.py > > async def test_base_url(db_path, path): > sqlite_utils.Database(db_path)[""creatures""].enable_fts([""name"", ""description""]) > datasette = Datasette([db_path], config={""base_url"": ""/foo/""}) > response = await datasette.client.get(path) > assert response.status_code == 200 I should fix those as soon as this goes out in a release. I won't close this issue until then.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/issues/1432#issuecomment-898084675,https://api.github.com/repos/simonw/datasette/issues/1432,898084675,IC_kwDOBm6k_c41h69D,9599,simonw,2021-08-13T01:11:30Z,2021-08-13T01:11:30Z,OWNER,It's only `datasette-publish-vercel` that will break the actual functionality - the others will have broken tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/issues/1429#issuecomment-897960049,https://api.github.com/repos/simonw/datasette/issues/1429,897960049,IC_kwDOBm6k_c41hchx,9599,simonw,2021-08-12T20:53:04Z,2021-08-12T20:53:04Z,OWNER,"Maybe something like this: > [Next page](#) - 100 per page ([show 1,000 per page](#))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969548935,UI for setting `?_size=max` on table page, https://github.com/simonw/datasette/issues/942#issuecomment-897996296,https://api.github.com/repos/simonw/datasette/issues/942,897996296,IC_kwDOBm6k_c41hlYI,9599,simonw,2021-08-12T22:01:36Z,2021-08-12T22:01:36Z,OWNER,"I'm going with `""columns"": {""name-of-column"": ""description-of-column""}`. If I decide to make `""col""` and `""nocol""` available in metadata I'll use those as the keys in the metadata, for consistency with the existing query string parameters. I'm OK with having both `""columns"": ...` and `""col"": ...` keys in the metadata, even though they could be a tiny bit confusing without the documentation.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898021895,https://api.github.com/repos/simonw/datasette/issues/942,898021895,IC_kwDOBm6k_c41hroH,9599,simonw,2021-08-12T22:51:36Z,2021-08-12T22:51:36Z,OWNER,"Prototype: ```diff diff --git a/datasette/static/app.css b/datasette/static/app.css index c6be1e9..5ca64cb 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -784,9 +784,14 @@ svg.dropdown-menu-icon { font-size: 0.7em; color: #666; margin: 0; - padding: 0; padding: 4px 8px 4px 8px; } +.dropdown-menu .dropdown-column-description { + margin: 0; + color: #666; + padding: 4px 8px 4px 8px; + max-width: 20em; +} .dropdown-menu li { border-bottom: 1px solid #ccc; } diff --git a/datasette/static/table.js b/datasette/static/table.js index 991346d..a903112 100644 --- a/datasette/static/table.js +++ b/datasette/static/table.js @@ -9,6 +9,7 @@ var DROPDOWN_HTML = ``; var DROPDOWN_ICON_SVG = ` @@ -166,6 +167,14 @@ var DROPDOWN_ICON_SVG = `
{% for column in display_columns %} - + {% if not column.sortable %} {{ column.name }} {% else %} diff --git a/datasette/views/table.py b/datasette/views/table.py index 456d806..486a613 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -125,6 +125,7 @@ class RowTableShared(DataView): """"""Returns columns, rows for specified table - including fancy foreign key treatment"""""" db = self.ds.databases[database] table_metadata = self.ds.table_metadata(database, table) + column_descriptions = table_metadata.get(""columns"") or {} column_details = {col.name: col for col in await db.table_column_details(table)} sortable_columns = await self.sortable_columns_for_table(database, table, True) pks = await db.primary_keys(table) @@ -147,6 +148,7 @@ class RowTableShared(DataView): ""is_pk"": r[0] in pks_for_display, ""type"": type_, ""notnull"": notnull, + ""description"": column_descriptions.get(r[0]), } ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898022235,https://api.github.com/repos/simonw/datasette/issues/942,898022235,IC_kwDOBm6k_c41hrtb,9599,simonw,2021-08-12T22:52:23Z,2021-08-12T22:52:23Z,OWNER,I like this. Need to solve for mobile though where the cog menu isn't visible - I think I'll do that with a definition list at the top of the page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898037456,https://api.github.com/repos/simonw/datasette/issues/942,898037456,IC_kwDOBm6k_c41hvbQ,9599,simonw,2021-08-12T23:23:34Z,2021-08-12T23:23:34Z,OWNER,"Prototype with a `
`: ```diff diff --git a/datasette/static/app.css b/datasette/static/app.css index c6be1e9..bf068fd 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -836,6 +841,16 @@ svg.dropdown-menu-icon { background-repeat: no-repeat; } +dl.column-descriptions dt { + font-weight: bold; +} +dl.column-descriptions dd { + padding-left: 1.5em; + white-space: pre-wrap; + line-height: 1.1em; + color: #666; +} + .anim-scale-in { animation-name: scale-in; animation-duration: 0.15s; diff --git a/datasette/templates/table.html b/datasette/templates/table.html index 211352b..466e8a4 100644 --- a/datasette/templates/table.html +++ b/datasette/templates/table.html @@ -51,6 +51,14 @@ {% block description_source_license %}{% include ""_description_source_license.html"" %}{% endblock %} +{% if metadata.columns %} +
+ {% for column_name, column_description in metadata.columns.items() %} +
{{ column_name }}
{{ column_description }}
+ {% endfor %} +
+{% endif %} + {% if filtered_table_rows_count or human_description_en %}

{% if filtered_table_rows_count or filtered_table_rows_count == 0 %}{{ ""{:,}"".format(filtered_table_rows_count) }} row{% if filtered_table_rows_count == 1 %}{% else %}s{% endif %}{% endif %} {% if human_description_en %}{{ human_description_en }}{% endif %} ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898037650,https://api.github.com/repos/simonw/datasette/issues/942,898037650,IC_kwDOBm6k_c41hveS,9599,simonw,2021-08-12T23:23:54Z,2021-08-12T23:23:54Z,OWNER,I like this enough that I'm going to ship it as an alpha and try it out on a couple of live projects.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898051645,https://api.github.com/repos/simonw/datasette/issues/942,898051645,IC_kwDOBm6k_c41hy49,9599,simonw,2021-08-13T00:02:25Z,2021-08-13T00:02:25Z,OWNER,"And on mobile: ![5FAF8D73-7199-4BB7-A5B8-9E46DCB4A985](https://user-images.githubusercontent.com/9599/129284817-dc13cbf4-144e-4f4c-8fb7-470602e2eea0.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898050457,https://api.github.com/repos/simonw/datasette/issues/942,898050457,IC_kwDOBm6k_c41hymZ,9599,simonw,2021-08-12T23:59:53Z,2021-08-12T23:59:53Z,OWNER,"Documentation: https://docs.datasette.io/en/latest/metadata.html#column-descriptions Live demo: https://latest.datasette.io/fixtures/roadside_attractions ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/1293#issuecomment-898056013,https://api.github.com/repos/simonw/datasette/issues/1293,898056013,IC_kwDOBm6k_c41hz9N,9599,simonw,2021-08-13T00:12:09Z,2021-08-13T00:12:09Z,OWNER,"Having added column metadata in #1430 (ref #942) I could also include a definition list at the top of the query results page exposing the column descriptions for any columns, using the same EXPLAIN mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1429#issuecomment-898185944,https://api.github.com/repos/simonw/datasette/issues/1429,898185944,IC_kwDOBm6k_c41iTrY,9599,simonw,2021-08-13T04:37:41Z,2021-08-13T04:37:41Z,OWNER,"If a count is available and the count is less than 1,000 it could say ""Show all"" instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969548935,UI for setting `?_size=max` on table page, https://github.com/simonw/datasette/issues/1293#issuecomment-898506647,https://api.github.com/repos/simonw/datasette/issues/1293,898506647,IC_kwDOBm6k_c41jh-X,9599,simonw,2021-08-13T14:43:19Z,2021-08-13T14:43:19Z,OWNER,Work will continue in PR #1434.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898517872,https://api.github.com/repos/simonw/datasette/issues/1293,898517872,IC_kwDOBm6k_c41jktw,9599,simonw,2021-08-13T15:00:50Z,2021-08-13T15:00:50Z,OWNER,"The primary key column (or `rowid`) often resolves to an `index` record in the `sqlite_master` table, e.g. the second row in this: type | name | tbl_name | rootpage | sql -- | -- | -- | -- | -- table | simple_primary_key | simple_primary_key | 2 | CREATE TABLE simple_primary_key ( id varchar(30) primary key, content text ) index | sqlite_autoindex_simple_primary_key_1 | simple_primary_key | 3 |   ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898519924,https://api.github.com/repos/simonw/datasette/issues/1293,898519924,IC_kwDOBm6k_c41jlN0,9599,simonw,2021-08-13T15:03:36Z,2021-08-13T15:03:36Z,OWNER,"Weird edge-case: adding an `order by` changes the order of the columns with respect to the information I am deriving about them. Without order by this gets it right: With order by: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898524057,https://api.github.com/repos/simonw/datasette/issues/1293,898524057,IC_kwDOBm6k_c41jmOZ,9599,simonw,2021-08-13T15:06:37Z,2021-08-13T15:06:37Z,OWNER,"Comparing the `explain` for the two versions of that query - one with the order by and one without: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898527525,https://api.github.com/repos/simonw/datasette/issues/1293,898527525,IC_kwDOBm6k_c41jnEl,9599,simonw,2021-08-13T15:08:03Z,2021-08-13T15:08:03Z,OWNER,Am I going to need to look at the `ResultRow` and its columns but then wind back to that earlier `MakeRecord` and its columns?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898536181,https://api.github.com/repos/simonw/datasette/issues/1293,898536181,IC_kwDOBm6k_c41jpL1,9599,simonw,2021-08-13T15:17:20Z,2021-08-13T15:20:33Z,OWNER,"Documentation for `MakeRecord`: https://www.sqlite.org/opcode.html#MakeRecord Running `explain` inside `sqlite3` provides extra comments and indentation which make it easier to understand: ``` sqlite> explain select neighborhood, facet_cities.name, state ...> from facetable ...> join facet_cities ...> on facetable.city_id = facet_cities.id ...> where neighborhood like '%bob%'; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 15 0 00 Start at 15 1 OpenRead 0 43 0 7 00 root=43 iDb=0; facetable 2 OpenRead 1 42 0 2 00 root=42 iDb=0; facet_cities 3 Rewind 0 14 0 00 4 Column 0 6 3 00 r[3]=facetable.neighborhood 5 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 6 IfNot 1 13 1 00 7 Column 0 5 4 00 r[4]=facetable.city_id 8 SeekRowid 1 13 4 00 intkey=r[4] 9 Column 0 6 5 00 r[5]=facetable.neighborhood 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 ResultRow 5 3 0 00 output=r[5..7] 13 Next 0 4 0 01 14 Halt 0 0 0 00 15 Transaction 0 0 35 0 01 usesStmtJournal=0 16 String8 0 2 0 %bob% 00 r[2]='%bob%' 17 Goto 0 1 0 00 ``` Compared with: ``` sqlite> explain select neighborhood, facet_cities.name, state ...> from facetable ...> join facet_cities ...> on facetable.city_id = facet_cities.id ...> where neighborhood like '%bob%' order by neighborhood ...> ; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 25 0 00 Start at 25 1 SorterOpen 2 5 0 k(1,B) 00 2 OpenRead 0 43 0 7 00 root=43 iDb=0; facetable 3 OpenRead 1 42 0 2 00 root=42 iDb=0; facet_cities 4 Rewind 0 16 0 00 5 Column 0 6 3 00 r[3]=facetable.neighborhood 6 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 7 IfNot 1 15 1 00 8 Column 0 5 4 00 r[4]=facetable.city_id 9 SeekRowid 1 15 4 00 intkey=r[4] 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 Column 0 6 5 00 r[5]=facetable.neighborhood 13 MakeRecord 5 3 9 00 r[9]=mkrec(r[5..7]) 14 SorterInsert 2 9 5 3 00 key=r[9] 15 Next 0 5 0 01 16 OpenPseudo 3 10 5 00 5 columns in r[10] 17 SorterSort 2 24 0 00 18 SorterData 2 10 3 00 r[10]=data 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] 23 SorterNext 2 18 0 00 24 Halt 0 0 0 00 25 Transaction 0 0 35 0 01 usesStmtJournal=0 26 String8 0 2 0 %bob% 00 r[2]='%bob%' 27 Goto 0 1 0 00 ``` So actually it looks like the `SorterSort` may be key to understanding this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898540260,https://api.github.com/repos/simonw/datasette/issues/1293,898540260,IC_kwDOBm6k_c41jqLk,9599,simonw,2021-08-13T15:23:28Z,2021-08-13T15:23:28Z,OWNER,"SorterInsert: > Register P2 holds an SQL index key made using the MakeRecord instructions. This opcode writes that key into the sorter P1. Data for the entry is nil. SorterData: > Write into register P2 the current sorter data for sorter cursor P1. Then clear the column header cache on cursor P3. > > This opcode is normally use to move a record out of the sorter and into a register that is the source for a pseudo-table cursor created using OpenPseudo. That pseudo-table cursor is the one that is identified by parameter P3. Clearing the P3 column cache as part of this opcode saves us from having to issue a separate NullRow instruction to clear that cache. OpenPseudo: > Open a new cursor that points to a fake table that contains a single row of data. The content of that one row is the content of memory register P2. In other words, cursor P1 becomes an alias for the MEM_Blob content contained in register P2. > > A pseudo-table created by this opcode is used to hold a single row output from the sorter so that the row can be decomposed into individual columns using the Column opcode. The Column opcode is the only cursor opcode that works with a pseudo-table. > > P3 is the number of fields in the records that will be stored by the pseudo-table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898541543,https://api.github.com/repos/simonw/datasette/issues/1293,898541543,IC_kwDOBm6k_c41jqfn,9599,simonw,2021-08-13T15:25:26Z,2021-08-13T15:25:26Z,OWNER,"But the debug output here seems to be saying what we want it to say: ``` 17 SorterSort 2 24 0 00 18 SorterData 2 10 3 00 r[10]=data 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] ``` We want to get back `neighborhood`, `facet_cities.name`, `state`. Why then are we seeing `[('facet_cities', 'name'), ('facetable', 'state'), (None, None)]`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898541972,https://api.github.com/repos/simonw/datasette/issues/1293,898541972,IC_kwDOBm6k_c41jqmU,9599,simonw,2021-08-13T15:26:06Z,2021-08-13T15:29:06Z,OWNER,"ResultRow: > The registers P1 through P1+P2-1 contain a single row of results. This opcode causes the sqlite3_step() call to terminate with an SQLITE_ROW return code and it sets up the sqlite3_stmt structure to provide access to the r(P1)..r(P1+P2-1) values as the result row. Column: > Interpret the data that cursor P1 points to as a structure built using the MakeRecord instruction. (See the MakeRecord opcode for additional information about the format of the data.) Extract the P2-th column from this record. If there are less that (P2+1) values in the record, extract a NULL. > > The value extracted is stored in register P3.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898545815,https://api.github.com/repos/simonw/datasette/issues/1293,898545815,IC_kwDOBm6k_c41jriX,9599,simonw,2021-08-13T15:31:53Z,2021-08-13T15:31:53Z,OWNER,"My hunch here is that registers or columns are being reused in a way that makes my code break - my code is pretty dumb, there are places in it where maybe the first mention of a register wins instead of the last one?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898554427,https://api.github.com/repos/simonw/datasette/issues/1293,898554427,IC_kwDOBm6k_c41jto7,9599,simonw,2021-08-13T15:45:32Z,2021-08-13T15:45:32Z,OWNER,"Some useful debug output: ``` table_rootpage_by_register={0: 43, 1: 42} names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} result_registers=[6, 7, 8] columns_by_column_register={3: ('facetable', 6), 4: ('facetable', 5), 6: ('facet_cities', 1), 7: ('facetable', 4), 5: ('facetable', 6)} all_column_names={('facet_cities', 0): 'id', ('facet_cities', 1): 'name', ('facetable', 0): 'pk', ('facetable', 1): 'created', ('facetable', 2): 'planet_int', ('facetable', 3): 'on_earth', ('facetable', 4): 'state', ('facetable', 5): 'city_id', ('facetable', 6): 'neighborhood', ('facetable', 7): 'tags', ('facetable', 8): 'complex_array', ('facetable', 9): 'distinct_some_null'} ``` The `result_registers` should each correspond to the correct entry in `columns_by_column_register` but they do not. Python code: ```python def columns_for_query(conn, sql, params=None): """""" Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of ``(table_name, column_name)`` pairs corresponding to the columns that would be returned by that SQL query. Each pair indicates the source table and column for the returned column, or ``(None, None)`` if no table and column could be derived (e.g. for ""select 1"") """""" if sql.lower().strip().startswith(""explain""): return [] opcodes = conn.execute(""explain "" + sql, params).fetchall() table_rootpage_by_register = { r[""p1""]: r[""p2""] for r in opcodes if r[""opcode""] == ""OpenRead"" } print(f""{table_rootpage_by_register=}"") names_and_types_by_rootpage = dict( [(r[0], (r[1], r[2])) for r in conn.execute( ""select rootpage, name, type from sqlite_master where rootpage in ({})"".format( "", "".join(map(str, table_rootpage_by_register.values())) ) )] ) print(f""{names_and_types_by_rootpage=}"") columns_by_column_register = {} for opcode in opcodes: if opcode[""opcode""] in (""Rowid"", ""Column""): addr, opcode, table_id, cid, column_register, p4, p5, comment = opcode try: table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] columns_by_column_register[column_register] = (table, cid) except KeyError: pass result_row = [dict(r) for r in opcodes if r[""opcode""] == ""ResultRow""][0] result_registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""])) print(f""{result_registers=}"") print(f""{columns_by_column_register=}"") all_column_names = {} for (table, _) in names_and_types_by_rootpage.values(): table_xinfo = conn.execute(""pragma table_xinfo({})"".format(table)).fetchall() for column_info in table_xinfo: all_column_names[(table, column_info[""cid""])] = column_info[""name""] print(f""{all_column_names=}"") final_output = [] for register in result_registers: try: table, cid = columns_by_column_register[register] final_output.append((table, all_column_names[table, cid])) except KeyError: final_output.append((None, None)) return final_output ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898554859,https://api.github.com/repos/simonw/datasette/issues/1293,898554859,IC_kwDOBm6k_c41jtvr,9599,simonw,2021-08-13T15:46:18Z,2021-08-13T15:46:18Z,OWNER,So it looks like the bug is in the code that populates `columns_by_column_register`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898567974,https://api.github.com/repos/simonw/datasette/issues/1293,898567974,IC_kwDOBm6k_c41jw8m,9599,simonw,2021-08-13T16:07:00Z,2021-08-13T16:07:00Z,OWNER,"So this line: ``` 19 Column 3 2 8 00 r[8]=state ``` Means ""Take column 2 of table 3 (the pseudo-table) and store it in register 8""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898564705,https://api.github.com/repos/simonw/datasette/issues/1293,898564705,IC_kwDOBm6k_c41jwJh,9599,simonw,2021-08-13T16:02:12Z,2021-08-13T16:04:06Z,OWNER,"More debug output: ``` table_rootpage_by_register={0: 43, 1: 42} names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_id=0 cid=6 column_register=3 table_id=0 cid=5 column_register=4 table_id=1 cid=1 column_register=6 table_id=0 cid=4 column_register=7 table_id=0 cid=6 column_register=5 table_id=3 cid=2 column_register=8 table_id=3 cid=2 column_register=8 KeyError 3 table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3 columns_by_column_register[column_register] = (table, cid) column_register=8 = (table='facetable', cid=2) table_id=3 cid=1 column_register=7 KeyError 3 table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3 columns_by_column_register[column_register] = (table, cid) column_register=7 = (table='facetable', cid=1) table_id=3 cid=0 column_register=6 KeyError 3 table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3 columns_by_column_register[column_register] = (table, cid) column_register=6 = (table='facetable', cid=0) result_registers=[6, 7, 8] columns_by_column_register={3: ('facetable', 6), 4: ('facetable', 5), 6: ('facet_cities', 1), 7: ('facetable', 4), 5: ('facetable', 6)} all_column_names={('facet_cities', 0): 'id', ('facet_cities', 1): 'name', ('facetable', 0): 'pk', ('facetable', 1): 'created', ('facetable', 2): 'planet_int', ('facetable', 3): 'on_earth', ('facetable', 4): 'state', ('facetable', 5): 'city_id', ('facetable', 6): 'neighborhood', ('facetable', 7): 'tags', ('facetable', 8): 'complex_array', ('facetable', 9): 'distinct_some_null'} ``` Those `KeyError` are happening here because of a lookup in `table_rootpage_by_register` for `table_id=3` - but `table_rootpage_by_register` only has keys 0 and 1. It looks like that `3` actually corresponds to the `OpenPseudo` table from here: ``` 16 OpenPseudo 3 10 5 00 5 columns in r[10] 17 SorterSort 2 24 0 00 18 SorterData 2 10 3 00 r[10]=data 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] ``` Python code: ```python def columns_for_query(conn, sql, params=None): """""" Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of ``(table_name, column_name)`` pairs corresponding to the columns that would be returned by that SQL query. Each pair indicates the source table and column for the returned column, or ``(None, None)`` if no table and column could be derived (e.g. for ""select 1"") """""" if sql.lower().strip().startswith(""explain""): return [] opcodes = conn.execute(""explain "" + sql, params).fetchall() table_rootpage_by_register = { r[""p1""]: r[""p2""] for r in opcodes if r[""opcode""] == ""OpenRead"" } print(f""{table_rootpage_by_register=}"") names_and_types_by_rootpage = dict( [(r[0], (r[1], r[2])) for r in conn.execute( ""select rootpage, name, type from sqlite_master where rootpage in ({})"".format( "", "".join(map(str, table_rootpage_by_register.values())) ) )] ) print(f""{names_and_types_by_rootpage=}"") columns_by_column_register = {} for opcode_row in opcodes: if opcode_row[""opcode""] in (""Rowid"", ""Column""): addr, opcode, table_id, cid, column_register, p4, p5, comment = opcode_row print(f""{table_id=} {cid=} {column_register=}"") try: table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] columns_by_column_register[column_register] = (table, cid) except KeyError as e: print("" KeyError"") print("" "", e) print("" table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]"") print(f"" {names_and_types_by_rootpage=} {table_rootpage_by_register=} {table_id=}"") print("" columns_by_column_register[column_register] = (table, cid)"") print(f"" {column_register=} = ({table=}, {cid=})"") pass result_row = [dict(r) for r in opcodes if r[""opcode""] == ""ResultRow""][0] result_registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""])) print(f""{result_registers=}"") print(f""{columns_by_column_register=}"") all_column_names = {} for (table, _) in names_and_types_by_rootpage.values(): table_xinfo = conn.execute(""pragma table_xinfo({})"".format(table)).fetchall() for column_info in table_xinfo: all_column_names[(table, column_info[""cid""])] = column_info[""name""] print(f""{all_column_names=}"") final_output = [] for register in result_registers: try: table, cid = columns_by_column_register[register] final_output.append((table, all_column_names[table, cid])) except KeyError: final_output.append((None, None)) return final_output ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898572065,https://api.github.com/repos/simonw/datasette/issues/1293,898572065,IC_kwDOBm6k_c41jx8h,9599,simonw,2021-08-13T16:13:16Z,2021-08-13T16:13:16Z,OWNER,"Aha! That `MakeRecord` line says `r[5..7]` - and r5 = neighborhood, r6 = facet_cities.name, r7 = facetable.state So if the `MakeRecord` defines what goes into that pseudo-table column 2 of that pseudo-table would be `state` - which is what we want. This is really convoluted. I'm no longer confident I can get this to work in a sensible way, especially since I've not started exploring what complex nested tables with CTEs and sub-selects do yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898569319,https://api.github.com/repos/simonw/datasette/issues/1293,898569319,IC_kwDOBm6k_c41jxRn,9599,simonw,2021-08-13T16:09:01Z,2021-08-13T16:10:48Z,OWNER,"Need to figure out what column 2 of that pseudo-table is. I think the answer is here: ``` 4 Rewind 0 16 0 00 5 Column 0 6 3 00 r[3]=facetable.neighborhood 6 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 7 IfNot 1 15 1 00 8 Column 0 5 4 00 r[4]=facetable.city_id 9 SeekRowid 1 15 4 00 intkey=r[4] 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 Column 0 6 5 00 r[5]=facetable.neighborhood 13 MakeRecord 5 3 9 00 r[9]=mkrec(r[5..7]) 14 SorterInsert 2 9 5 3 00 key=r[9] 15 Next 0 5 0 01 16 OpenPseudo 3 10 5 00 5 columns in r[10] ``` I think the `OpenPseduo` line puts five columns in `r[10]` - and those five columns are the five from the previous block - maybe the five leading up to the `MakeRecord` call on line 13. In which case column 2 would be `facet_cities.name` - assuming we start counting from 0. But the debug code said ""r[8]=state"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898576097,https://api.github.com/repos/simonw/datasette/issues/1293,898576097,IC_kwDOBm6k_c41jy7h,9599,simonw,2021-08-13T16:19:57Z,2021-08-13T16:19:57Z,OWNER,"I think I need to look out for `OpenPseudo` and, when that occurs, take a look at the most recent `SorterInsert` and use that to find the `MakeRecord` and then use the `MakeRecord` to figure out the columns that went into it. After all of that I'll be able to resolve that ""table 3"" reference.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898760020,https://api.github.com/repos/simonw/datasette/issues/1293,898760020,IC_kwDOBm6k_c41kf1U,9599,simonw,2021-08-13T23:00:28Z,2021-08-13T23:01:27Z,OWNER,"New theory: this is all about `SorterOpen` and `SorterInsert`. Consider the following with extra annotations at the end of the lines after the `--`: ``` addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 25 0 00 Start at 25 1 SorterOpen 2 5 0 k(1,B) 00 -- New SORTER in r2 with 5 slots 2 OpenRead 0 43 0 7 00 root=43 iDb=0; facetable 3 OpenRead 1 42 0 2 00 root=42 iDb=0; facet_cities 4 Rewind 0 16 0 00 5 Column 0 6 3 00 r[3]=facetable.neighborhood 6 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 7 IfNot 1 15 1 00 8 Column 0 5 4 00 r[4]=facetable.city_id 9 SeekRowid 1 15 4 00 intkey=r[4] 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 Column 0 6 5 00 r[5]=facetable.neighborhood 13 MakeRecord 5 3 9 00 r[9]=mkrec(r[5..7]) 14 SorterInsert 2 9 5 3 00 key=r[9]-- WRITES record from r9 (line above) into sorter in r2 15 Next 0 5 0 01 16 OpenPseudo 3 10 5 00 5 columns in r[10] 17 SorterSort 2 24 0 00 -- runs the sort, not relevant to my goal 18 SorterData 2 10 3 00 r[10]=data -- ""Write into register P2 (r10) the current sorter data for sorter cursor P1 (sorter 2)"" 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] 23 SorterNext 2 18 0 00 24 Halt 0 0 0 00 25 Transaction 0 0 35 0 01 usesStmtJournal=0 26 String8 0 2 0 %bob% 00 r[2]='%bob%' 27 Goto 0 1 0 00 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898760808,https://api.github.com/repos/simonw/datasette/issues/1293,898760808,IC_kwDOBm6k_c41kgBo,9599,simonw,2021-08-13T23:03:01Z,2021-08-13T23:03:01Z,OWNER,Another idea: strip out any `order by` clause to try and keep this simpler. I doubt that's going to cope with complex nested queries though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898788262,https://api.github.com/repos/simonw/datasette/issues/1293,898788262,IC_kwDOBm6k_c41kmum,9599,simonw,2021-08-14T01:22:26Z,2021-08-14T01:51:08Z,OWNER,"Tried a more complicated query: ```sql explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101 ``` Here's the explain: ``` sqlite> explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101 ...> ; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 41 0 00 Start at 41 1 OpenEphemeral 2 6 0 k(1,-B) 00 nColumn=6 2 Integer 101 1 0 00 r[1]=101; LIMIT counter 3 OpenRead 0 32 0 4 00 root=32 iDb=0; searchable 4 Integer 16 3 0 00 r[3]=16; return address 5 Once 0 16 0 00 6 OpenEphemeral 3 1 0 k(1,) 00 nColumn=1; Result of SELECT 1 7 VOpen 1 0 0 vtab:7FCBCA72BE80 00 8 Function0 1 7 6 unknown(-1) 01 r[6]=func(r[7]) 9 Integer 5 4 0 00 r[4]=5 10 Integer 1 5 0 00 r[5]=1 11 VFilter 1 16 4 00 iplan=r[4] zplan='' 12 Rowid 1 8 0 00 r[8]=rowid 13 MakeRecord 8 1 9 C 00 r[9]=mkrec(r[8]) 14 IdxInsert 3 9 8 1 00 key=r[9] 15 VNext 1 12 0 00 16 Return 3 0 0 00 17 Rewind 3 33 0 00 18 Column 3 0 2 00 r[2]= 19 IsNull 2 32 0 00 if r[2]==NULL goto 32 20 SeekRowid 0 32 2 00 intkey=r[2] 21 Column 0 1 10 00 r[10]=searchable.text1 22 Sequence 2 11 0 00 r[11]=cursor[2].ctr++ 23 IfNotZero 1 27 0 00 if r[1]!=0 then r[1]--, goto 27 24 Last 2 0 0 00 25 IdxLE 2 32 10 1 00 key=r[10] 26 Delete 2 0 0 00 27 Rowid 0 12 0 00 r[12]=rowid 28 Column 0 2 13 00 r[13]=searchable.text2 29 Column 0 3 14 00 r[14]=searchable.name with . and spaces 30 MakeRecord 10 5 16 00 r[16]=mkrec(r[10..14]) 31 IdxInsert 2 16 10 5 00 key=r[16] 32 Next 3 18 0 00 33 Sort 2 40 0 00 34 Column 2 4 15 00 r[15]=[name with . and spaces] 35 Column 2 3 14 00 r[14]=text2 36 Column 2 0 13 00 r[13]=text1 37 Column 2 2 12 00 r[12]=pk 38 ResultRow 12 4 0 00 output=r[12..15] 39 Next 2 34 0 00 40 Halt 0 0 0 00 41 Transaction 0 0 35 0 01 usesStmtJournal=0 42 Variable 1 7 0 :search 00 r[7]=parameter(1,:search) 43 Goto 0 1 0 00 ``` Here the `ResultRow` is for registers `12..15` - but those all refer to `Column` records in `2` - where `2` is the first `OpenEphemeral` declared right at the start. I'm having enormous trouble figuring out how that ephemeral table gets populated by the other operations in a way that would let me derive which columns end up in the `ResultRow`. Frustratingly SQLite seems to be able to figure that out just fine, see the column of comments on the right hand side - but I only get those in the `sqlite3` CLI shell, they're not available to me with SQLite when called as a library from Python. Maybe the key to that is this section: ``` 27 Rowid 0 12 0 00 r[12]=rowid 28 Column 0 2 13 00 r[13]=searchable.text2 29 Column 0 3 14 00 r[14]=searchable.name with . and spaces 30 MakeRecord 10 5 16 00 r[16]=mkrec(r[10..14]) 31 IdxInsert 2 16 10 5 00 key=r[16] ``` MakeRecord: > Convert P2 registers beginning with P1 into the record format use as a data record in a database table or as a key in an index. The Column opcode can decode the record later. > > P4 may be a string that is P2 characters long. The N-th character of the string indicates the column affinity that should be used for the N-th field of the index key. > > The mapping from character to affinity is given by the SQLITE_AFF_ macros defined in sqliteInt.h. > > If P4 is NULL then all index fields have the affinity BLOB. > > The meaning of P5 depends on whether or not the SQLITE_ENABLE_NULL_TRIM compile-time option is enabled: > > * If SQLITE_ENABLE_NULL_TRIM is enabled, then the P5 is the index of the right-most table that can be null-trimmed. > > * If SQLITE_ENABLE_NULL_TRIM is omitted, then P5 has the value OPFLAG_NOCHNG_MAGIC if the MakeRecord opcode is allowed to accept no-change records with serial_type 10. This value is only used inside an assert() and does not affect the end result. IdxInsert: > Register P2 holds an SQL index key made using the MakeRecord instructions. This opcode writes that key into the index P1. Data for the entry is nil. > > If P4 is not zero, then it is the number of values in the unpacked key of reg(P2). In that case, P3 is the index of the first register for the unpacked key. The availability of the unpacked key can sometimes be an optimization. > > If P5 has the OPFLAG_APPEND bit set, that is a hint to the b-tree layer that this insert is likely to be an append. > > If P5 has the OPFLAG_NCHANGE bit set, then the change counter is incremented by this instruction. If the OPFLAG_NCHANGE bit is clear, then the change counter is unchanged. > > If the OPFLAG_USESEEKRESULT flag of P5 is set, the implementation might run faster by avoiding an unnecessary seek on cursor P1. However, the OPFLAG_USESEEKRESULT flag must only be set if there have been no prior seeks on the cursor or if the most recent seek used a key equivalent to P2. > > This instruction only works for indices. The equivalent instruction for tables is Insert. IdxLE: > The P4 register values beginning with P3 form an unpacked index key that omits the PRIMARY KEY or ROWID. Compare this key value against the index that P1 is currently pointing to, ignoring the PRIMARY KEY or ROWID on the P1 index. > > If the P1 index entry is less than or equal to the key value then jump to P2. Otherwise fall through to the next instruction.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898913554,https://api.github.com/repos/simonw/datasette/issues/1293,898913554,IC_kwDOBm6k_c41lFUS,9599,simonw,2021-08-14T16:13:40Z,2021-08-14T16:13:40Z,OWNER,"I think I need to care about the following: - `ResultRow` and `Column` for the final result - `OpenRead` for opening tables - `OpenEphemeral` then `MakeRecord` and `IdxInsert` for writing records into ephemeral tables `Column` may reference either a table (from `OpenRead`) or an ephemeral table (from `OpenEphemeral`). That *might* be enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898913629,https://api.github.com/repos/simonw/datasette/issues/1293,898913629,IC_kwDOBm6k_c41lFVd,9599,simonw,2021-08-14T16:14:12Z,2021-08-14T16:14:12Z,OWNER,I would feel a lot more comfortable about all of this if I had a robust mechanism for running the Datasette test suite against multiple versions of SQLite itself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898936068,https://api.github.com/repos/simonw/datasette/issues/1293,898936068,IC_kwDOBm6k_c41lK0E,9599,simonw,2021-08-14T17:44:54Z,2021-08-14T17:44:54Z,OWNER,"Another interesting query to consider: https://latest.datasette.io/fixtures?sql=explain+select+*+from++pragma_table_info%28+%27123_starts_with_digits%27%29 That one shows `VColumn` instead of `Column`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898933865,https://api.github.com/repos/simonw/datasette/issues/1293,898933865,IC_kwDOBm6k_c41lKRp,9599,simonw,2021-08-14T17:27:16Z,2021-08-14T17:28:29Z,OWNER,"Maybe I split this out into a separate Python library that gets tested against *every* SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself? Datasette could depend on that library. The library could be released independently of Datasette any time a new SQLite version comes out. I could even run a separate git scraper repo that checks for new SQLite releases and submits PRs against the library when a new release comes out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898961535,https://api.github.com/repos/simonw/datasette/issues/1293,898961535,IC_kwDOBm6k_c41lRB_,9599,simonw,2021-08-14T21:37:24Z,2021-08-14T21:37:24Z,OWNER,Did some more research into building SQLite custom versions via `pysqlite3` - here's what I figured out for macOS (which should hopefully work for Linux too): https://til.simonwillison.net/sqlite/build-specific-sqlite-pysqlite-macos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-899915829,https://api.github.com/repos/simonw/datasette/issues/1293,899915829,IC_kwDOBm6k_c41o6A1,9599,simonw,2021-08-17T01:02:35Z,2021-08-17T01:02:35Z,OWNER,"New approach: this time I'm building a simplified executor for the bytecode operations themselves. ```python def execute_operations(operations, max_iterations = 100, trace=None): trace = trace or (lambda *args: None) registers: Dict[int, Any] = {} cursors: Dict[int, Tuple[str, Dict]] = {} instruction_pointer = 0 iterations = 0 result_row = None while True: iterations += 1 if iterations > max_iterations: break operation = operations[instruction_pointer] trace(instruction_pointer, dict(operation)) opcode = operation[""opcode""] if opcode == ""Init"": if operation[""p2""] != 0: instruction_pointer = operation[""p2""] continue else: instruction_pointer += 1 continue elif opcode == ""Goto"": instruction_pointer = operation[""p2""] continue elif opcode == ""Halt"": break elif opcode == ""OpenRead"": cursors[operation[""p1""]] = (""database_table"", { ""rootpage"": operation[""p2""], ""connection"": operation[""p3""], }) elif opcode == ""OpenEphemeral"": cursors[operation[""p1""]] = (""ephemeral"", { ""num_columns"": operation[""p2""], ""index_keys"": [], }) elif opcode == ""MakeRecord"": registers[operation[""p3""]] = (""MakeRecord"", { ""registers"": list(range(operation[""p1""] + operation[""p2""])) }) elif opcode == ""IdxInsert"": record = registers[operation[""p2""]] cursors[operation[""p1""]][1][""index_keys""].append(record) elif opcode == ""Rowid"": registers[operation[""p2""]] = (""rowid"", { ""table"": operation[""p1""] }) elif opcode == ""Sequence"": registers[operation[""p2""]] = (""sequence"", { ""next_from_cursor"": operation[""p1""] }) elif opcode == ""Column"": registers[operation[""p3""]] = (""column"", { ""cursor"": operation[""p1""], ""column_offset"": operation[""p2""] }) elif opcode == ""ResultRow"": p1 = operation[""p1""] p2 = operation[""p2""] trace(""ResultRow: "", list(range(p1, p1 + p2)), registers) result_row = [registers.get(i) for i in range(p1, p1 + p2)] elif opcode == ""Integer"": registers[operation[""p2""]] = (""Integer"", operation[""p1""]) elif opcode == ""String8"": registers[operation[""p2""]] = (""String"", operation[""p4""]) instruction_pointer += 1 return {""registers"": registers, ""cursors"": cursors, ""result_row"": result_row} ``` Results are promising! ``` execute_operations(db.execute(""explain select 'hello', 55, rowid, * from searchable"").fetchall()) {'registers': {1: ('String', 'hello'), 2: ('Integer', 55), 3: ('rowid', {'table': 0}), 4: ('rowid', {'table': 0}), 5: ('column', {'cursor': 0, 'column_offset': 1}), 6: ('column', {'cursor': 0, 'column_offset': 2}), 7: ('column', {'cursor': 0, 'column_offset': 3})}, 'cursors': {0: ('database_table', {'rootpage': 32, 'connection': 0})}, 'result_row': [('String', 'hello'), ('Integer', 55), ('rowid', {'table': 0}), ('rowid', {'table': 0}), ('column', {'cursor': 0, 'column_offset': 1}), ('column', {'cursor': 0, 'column_offset': 2}), ('column', {'cursor': 0, 'column_offset': 3})]} ``` Here's what happens with a union across three tables: ``` execute_operations(db.execute(f"""""" explain select data as content from binary_data union select pk as content from complex_foreign_keys union select name as content from facet_cities """"""}).fetchall()) {'registers': {1: ('column', {'cursor': 4, 'column_offset': 0}), 2: ('MakeRecord', {'registers': [0, 1, 2, 3]}), 3: ('column', {'cursor': 0, 'column_offset': 1}), 4: ('column', {'cursor': 3, 'column_offset': 0})}, 'cursors': {3: ('ephemeral', {'num_columns': 1, 'index_keys': [('MakeRecord', {'registers': [0, 1]}), ('MakeRecord', {'registers': [0, 1]}), ('MakeRecord', {'registers': [0, 1, 2, 3]})]}), 2: ('database_table', {'rootpage': 44, 'connection': 0}), 4: ('database_table', {'rootpage': 24, 'connection': 0}), 0: ('database_table', {'rootpage': 42, 'connection': 0})}, 'result_row': [('column', {'cursor': 3, 'column_offset': 0})]} ``` Note how the result_row refers to cursor 3, which is an ephemeral table which had three different sets of `MakeRecord` index keys assigned to it - indicating that the output column is NOT from the same underlying table source.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1423#issuecomment-899744109,https://api.github.com/repos/simonw/datasette/issues/1423,899744109,IC_kwDOBm6k_c41oQFt,9599,simonw,2021-08-16T18:58:29Z,2021-08-16T18:58:29Z,OWNER,"I didn't bother with the tooltip, just the visible display if `?_facet_size=max`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1423#issuecomment-899749881,https://api.github.com/repos/simonw/datasette/issues/1423,899749881,IC_kwDOBm6k_c41oRf5,9599,simonw,2021-08-16T19:07:02Z,2021-08-16T19:07:02Z,OWNER,"Demo: https://latest.datasette.io/fixtures/compound_three_primary_keys?_facet=content&_facet_size=max&_facet=pk1&_facet=pk2 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1438#issuecomment-900681413,https://api.github.com/repos/simonw/datasette/issues/1438,900681413,IC_kwDOBm6k_c41r07F,9599,simonw,2021-08-17T22:47:44Z,2021-08-17T22:47:44Z,OWNER,I deployed another copy of `fixtures.db` on Vercel at https://til.simonwillison.net/fixtures so I can compare it with `fixtures.db` on Cloud Run at https://latest.datasette.io/fixtures,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900690998,https://api.github.com/repos/simonw/datasette/issues/1438,900690998,IC_kwDOBm6k_c41r3Q2,9599,simonw,2021-08-17T23:11:16Z,2021-08-17T23:12:25Z,OWNER,"I have completely failed to replicate this initial bug - but it's still there on the `thesession.vercel.app` deployment (even though my own deployments to Vercel do not exhibit it). Here's a one-liner to replicate it against that deployment: `curl -s 'https://thesession.vercel.app/thesession?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22' | rg '.csv'` Whit outputs this: `

This data as json, CSV

` It looks like, rather than being URL-encoded, the original query string is somehow making it through to Jinja and then being auto-escaped there. The weird thing is that the equivalent query executed against my `til.simonwillison.net` Vercel instance does this: `curl -s 'https://til.simonwillison.net/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25a%25%22' | rg '.csv'` `

This data as json, CSV

`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1439#issuecomment-900699670,https://api.github.com/repos/simonw/datasette/issues/1439,900699670,IC_kwDOBm6k_c41r5YW,9599,simonw,2021-08-17T23:34:23Z,2021-08-17T23:34:23Z,OWNER,"The challenge comes down to telling the difference between the following: - `/db/table` - an HTML table page - `/db/table.csv` - the CSV version of `/db/table` - `/db/table.csv` - no this one is actually a database table called `table.csv` - `/db/table.csv.csv` - the CSV version of `/db/table.csv` - `/db/table.csv.csv.csv` and so on...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900705226,https://api.github.com/repos/simonw/datasette/issues/1439,900705226,IC_kwDOBm6k_c41r6vK,9599,simonw,2021-08-17T23:50:32Z,2021-08-17T23:50:47Z,OWNER,"An alternative solution would be to use some form of escaping for the characters that form the name of the table. The obvious way to do this would be URL-encoding - but it doesn't hold for `.` characters. The hex for that is `%2E` but watch what happens with that in a URL: ``` # Against Cloud Run: curl -s 'https://datasette.io/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path 'path': '/-/asgi-scope/foo/bar/baz.', 'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz.', 'root_path': '', # Against Vercel: curl -s 'https://til.simonwillison.net/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path 'path': '/-/asgi-scope/foo/bar%2Fbaz%2E', 'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz%2E', 'root_path': '', ``` Surprisingly in this case Vercel DOES keep it intact, but Cloud Run does not. It's still no good though: I need a solution that works on Vercel, Cloud Run and every other potential hosting provider too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900709703,https://api.github.com/repos/simonw/datasette/issues/1439,900709703,IC_kwDOBm6k_c41r71H,9599,simonw,2021-08-18T00:03:09Z,2021-08-18T00:03:09Z,OWNER,"But... what if I invent my own escaping scheme? I actually did this once before, in https://github.com/simonw/datasette/commit/9fdb47ca952b93b7b60adddb965ea6642b1ff523 - while I was working on porting Datasette to ASGI in https://github.com/simonw/datasette/issues/272#issuecomment-494192779 because ASGI didn't yet have the `raw_path` mechanism. I could bring that back - it looked like this: ``` ""table/and/slashes"" => ""tableU+002FandU+002Fslashes"" ""~table"" => ""U+007Etable"" ""+bobcats!"" => ""U+002Bbobcats!"" ""U+007Etable"" => ""UU+002B007Etable"" ``` But I didn't particularly like it - it was quite verbose.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900711967,https://api.github.com/repos/simonw/datasette/issues/1439,900711967,IC_kwDOBm6k_c41r8Yf,9599,simonw,2021-08-18T00:08:09Z,2021-08-18T00:08:09Z,OWNER,"Here's an alternative I just made up which I'm calling ""dot dash"" encoding: ```python def dot_dash_encode(s): return s.replace(""-"", ""--"").replace(""."", ""-."") def dot_dash_decode(s): return s.replace(""-."", ""."").replace(""--"", ""-"") ``` And some examples: ```python for example in ( ""hello"", ""hello.csv"", ""hello-and-so-on.csv"", ""hello-.csv"", ""hello--and--so--on-.csv"", ""hello.csv."", ""hello.csv.-"", ""hello.csv.--"", ): print(example) print(dot_dash_encode(example)) print(example == dot_dash_decode(dot_dash_encode(example))) print() ``` Outputs: ``` hello hello True hello.csv hello-.csv True hello-and-so-on.csv hello--and--so--on-.csv True hello-.csv hello---.csv True hello--and--so--on-.csv hello----and----so----on---.csv True hello.csv. hello-.csv-. True hello.csv.- hello-.csv-.-- True hello.csv.-- hello-.csv-.---- True ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900712981,https://api.github.com/repos/simonw/datasette/issues/1439,900712981,IC_kwDOBm6k_c41r8oV,9599,simonw,2021-08-18T00:09:59Z,2021-08-18T00:12:32Z,OWNER,"So given the original examples, a table called `table.csv` would have the following URLs: - `/db/table-.csv` - the HTML version - `/db/table-.csv.csv` - the CSV version - `/db/table-.csv.json` - the JSON version And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this: - `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version - `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version - `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900714630,https://api.github.com/repos/simonw/datasette/issues/1439,900714630,IC_kwDOBm6k_c41r9CG,9599,simonw,2021-08-18T00:13:33Z,2021-08-18T00:13:33Z,OWNER,"The documentation should definitely cover how table names become URLs, in case any third party code needs to be able to calculate this themselves.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900715375,https://api.github.com/repos/simonw/datasette/issues/1439,900715375,IC_kwDOBm6k_c41r9Nv,9599,simonw,2021-08-18T00:15:28Z,2021-08-18T00:15:28Z,OWNER,"Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1438#issuecomment-900500824,https://api.github.com/repos/simonw/datasette/issues/1438,900500824,IC_kwDOBm6k_c41rI1Y,9599,simonw,2021-08-17T17:38:16Z,2021-08-17T17:38:16Z,OWNER,"Relevant template code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/templates/query.html#L71 `renderers` comes from here: https://github.com/simonw/datasette/blob/2883098770fc66e50183b2b231edbde20848d4d6/datasette/views/base.py#L593-L608","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900502364,https://api.github.com/repos/simonw/datasette/issues/1438,900502364,IC_kwDOBm6k_c41rJNc,9599,simonw,2021-08-17T17:40:41Z,2021-08-17T17:40:41Z,OWNER,Bug is likely in `path_with_format` itself: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L710-L729,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900513267,https://api.github.com/repos/simonw/datasette/issues/1438,900513267,IC_kwDOBm6k_c41rL3z,9599,simonw,2021-08-17T17:57:05Z,2021-08-17T17:57:05Z,OWNER,"I'm having trouble replicating this bug outside of Vercel. Against Cloud Run: view-source:https://latest.datasette.io/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22 The HTML here is: ```html

This data as json, ... CSV

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900516826,https://api.github.com/repos/simonw/datasette/issues/1438,900516826,IC_kwDOBm6k_c41rMva,9599,simonw,2021-08-17T18:02:27Z,2021-08-17T18:02:27Z,OWNER,"The key difference I can spot between Vercel and Cloud Run is that `+` in a query string gets converted to `%20` by Vercel before it gets to my app, but does not for Cloud Run: ``` # Vercel ~ % curl -s 'https://til.simonwillison.net/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'sql=select%20*%20from%20tunes%20where%20name%20like%20%22%25' b'wise%20maid%25%22%0D%0A', 'raw_path': b'/-/asgi-scope', # Cloud Run ~ % curl -s 'https://latest-with-plugins.datasette.io/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%2' b'2%0D%0A', 'raw_path': b'/-/asgi-scope', ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900518343,https://api.github.com/repos/simonw/datasette/issues/1438,900518343,IC_kwDOBm6k_c41rNHH,9599,simonw,2021-08-17T18:04:42Z,2021-08-17T18:04:42Z,OWNER,Here's how `request.query_string` works: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/asgi.py#L86-L88,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1293#issuecomment-901475812,https://api.github.com/repos/simonw/datasette/issues/1293,901475812,IC_kwDOBm6k_c41u23k,9599,simonw,2021-08-18T22:41:19Z,2021-08-18T22:41:19Z,OWNER,"> Maybe I split this out into a separate Python library that gets tested against _every_ SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself? I'm going to do this, and call the Python library `sqlite-explain`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1415#issuecomment-902251316,https://api.github.com/repos/simonw/datasette/issues/1415,902251316,IC_kwDOBm6k_c41x0M0,9599,simonw,2021-08-19T21:14:15Z,2021-08-19T21:14:15Z,OWNER,"https://github.com/ahmetb/cloud-run-faq#how-do-i-continuously-deploy-to-cloud-run suggests the following: > - `roles/run.admin` to deploy applications > - `roles/iam.serviceAccountUser` on the service account that your app will use It also links to https://cloud.google.com/run/docs/reference/iam/roles","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959137143,feature request: document minimum permissions for service account for cloudrun, https://github.com/simonw/datasette/issues/1443#issuecomment-902258509,https://api.github.com/repos/simonw/datasette/issues/1443,902258509,IC_kwDOBm6k_c41x19N,9599,simonw,2021-08-19T21:25:07Z,2021-08-19T21:25:07Z,OWNER,https://docs.datasette.io/en/latest/internals.html#databases,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974995592,datasette.databases should be a documented property, https://github.com/simonw/datasette/pull/1434#issuecomment-902254712,https://api.github.com/repos/simonw/datasette/issues/1434,902254712,IC_kwDOBm6k_c41x1B4,9599,simonw,2021-08-19T21:18:31Z,2021-08-19T21:18:57Z,OWNER,"I deployed a demo to https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app using the mechanism from #1442. e.g. demo here: https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app/fixtures?sql=select+*+from+searchable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",970463436,Enrich arbitrary query results with foreign key links and column descriptions, https://github.com/simonw/datasette/issues/1426#issuecomment-902260338,https://api.github.com/repos/simonw/datasette/issues/1426,902260338,IC_kwDOBm6k_c41x2Zy,9599,simonw,2021-08-19T21:28:25Z,2021-08-19T21:29:40Z,OWNER,"Actually it looks like you can send a `sitemap.xml` to Google using an unauthenticated GET request to: https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP According to https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-902260799,https://api.github.com/repos/simonw/datasette/issues/1426,902260799,IC_kwDOBm6k_c41x2g_,9599,simonw,2021-08-19T21:29:13Z,2021-08-19T21:29:13Z,OWNER,"Bing's equivalent is: https://www.bing.com/webmasters/help/Sitemaps-3b5cf6ed http://www.bing.com/ping?sitemap=FULL_URL_OF_SITEMAP","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-902263367,https://api.github.com/repos/simonw/datasette/issues/1426,902263367,IC_kwDOBm6k_c41x3JH,9599,simonw,2021-08-19T21:33:51Z,2021-08-19T21:36:28Z,OWNER,"I was worried about if it's possible to allow access to `/fixtures` but deny access to `/fixtures?sql=...` From various answers on Stack Overflow it looks like this should handle that: ``` User-agent: * Disallow: /fixtures? ``` I could use this for tables too - it may well be OK to access table index pages while still avoiding pagination, facets etc. I think this should block both query strings and row pages while allowing the table page itself: ``` User-agent: * Disallow: /fixtures/searchable? Disallow: /fixtures/searchable/* ``` Could even accompany that with a `sitemap.xml` that explicitly lists all of the tables - which would mean adding sitemaps to Datasette core too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1442#issuecomment-902191150,https://api.github.com/repos/simonw/datasette/issues/1442,902191150,IC_kwDOBm6k_c41xlgu,9599,simonw,2021-08-19T19:43:05Z,2021-08-19T19:43:59Z,OWNER,"Maybe as simple as teaching https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml to run on pushes to ALL branches: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L3-L6 And then quit early if the branch is not in some allow-list. If it IS in the allow-list, use the name of the branch to dynamically construct the name of the Cloud Run service here: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L60 Need to skip the documentation build and deployment stuff for other branches though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902217726,https://api.github.com/repos/simonw/datasette/issues/1442,902217726,IC_kwDOBm6k_c41xr_-,9599,simonw,2021-08-19T20:21:47Z,2021-08-19T20:21:47Z,OWNER,I think the neatest way to implement this would be for the `on -> push -> branches` list to be the list of branches that should be deployed in this way. The rest of the code can react to that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902231018,https://api.github.com/repos/simonw/datasette/issues/1442,902231018,IC_kwDOBm6k_c41xvPq,9599,simonw,2021-08-19T20:42:08Z,2021-08-19T20:42:08Z,OWNER,If I get this working I should document it on https://docs.datasette.io/en/stable/contributing.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902235714,https://api.github.com/repos/simonw/datasette/issues/1442,902235714,IC_kwDOBm6k_c41xwZC,9599,simonw,2021-08-19T20:50:38Z,2021-08-19T20:50:38Z,OWNER,"Would this allow anyone to push a PR to this repo that would result in their code being deployed against my Cloud Run account? I'm reasonably confident that it would not, since the secrets would not be visible to their PR branch.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902239215,https://api.github.com/repos/simonw/datasette/issues/1442,902239215,IC_kwDOBm6k_c41xxPv,9599,simonw,2021-08-19T20:56:46Z,2021-08-19T20:56:46Z,OWNER,"I'm going to only run the tests if it's a push to `main` - that way I can ship demo branches really quickly, even if they don't yet have passing tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902243498,https://api.github.com/repos/simonw/datasette/issues/1442,902243498,IC_kwDOBm6k_c41xySq,9599,simonw,2021-08-19T21:04:01Z,2021-08-19T21:04:01Z,OWNER,That successfully deployed to https://datasette-latest-deploy-this-branch-j7hipcg4aq-uc.a.run.app/ even though the tests failed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1415#issuecomment-902250361,https://api.github.com/repos/simonw/datasette/issues/1415,902250361,IC_kwDOBm6k_c41xz95,9599,simonw,2021-08-19T21:12:28Z,2021-08-19T21:12:28Z,OWNER,I would love to know this too! I always find figuring out minimal permissions to be really difficult.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959137143,feature request: document minimum permissions for service account for cloudrun, https://github.com/simonw/datasette/issues/894#issuecomment-902375088,https://api.github.com/repos/simonw/datasette/issues/894,902375088,IC_kwDOBm6k_c41ySaw,9599,simonw,2021-08-20T02:07:13Z,2021-08-20T02:07:26Z,OWNER,Maybe `?_sort_numeric=col` and `?_sort_numeric_desc=col` would be better here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/894#issuecomment-902375388,https://api.github.com/repos/simonw/datasette/issues/894,902375388,IC_kwDOBm6k_c41ySfc,9599,simonw,2021-08-20T02:07:53Z,2021-08-20T02:07:53Z,OWNER,I could add these sorting links to the cog menu for any `TEXT` columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/236#issuecomment-922075480,https://api.github.com/repos/simonw/datasette/issues/236,922075480,IC_kwDOBm6k_c429cFY,9599,simonw,2021-09-17T20:54:13Z,2021-09-17T20:54:13Z,OWNER,"That's so useful @sethvincent! Really interesting reading your code there, especially clever how you're using the `base_url` config. I'd be very interested to see what your demo looks like without using serverless - completely agree that the less additional dependencies there are for this the better. I'm also very interested in figuring out a way to run Datasette in Lambda but with the SQLite database on an EFS volume. Do you have a feel for how hard that would be?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin, https://github.com/simonw/datasette/issues/1449#issuecomment-907537693,https://api.github.com/repos/simonw/datasette/issues/1449,907537693,IC_kwDOBm6k_c42F-0d,9599,simonw,2021-08-28T00:31:26Z,2021-08-28T00:31:26Z,OWNER,Terminology question: is it correct to call these subcommands or should they be commands? `publish_subcommand()` adds subcommands of the format `datasette publish X` - but are we instead adding commands with this new one?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907537366,https://api.github.com/repos/simonw/datasette/issues/1449,907537366,IC_kwDOBm6k_c42F-vW,9599,simonw,2021-08-28T00:29:16Z,2021-08-28T00:29:29Z,OWNER,"The closest plugin hook to this right now is [publish_subcommand](https://docs.datasette.io/en/stable/plugin_hooks.html#publish-subcommand-publish) - which looks like this: ```python @hookimpl def publish_subcommand(publish): @publish.command() @add_common_publish_arguments_and_options @click.option( ""-k"", ""--api_key"", help=""API key for talking to my hosting provider"", ) def my_hosting_provider(...): ``` But there are also several plugin hooks with `register_` prefixes, which may be a good naming convention to stick to here: `register_output_renderer(datasette)`, `register_routes(datasette)`, `register_facet_classes()`, `register_magic_parameters(datasette)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907537610,https://api.github.com/repos/simonw/datasette/issues/1449,907537610,IC_kwDOBm6k_c42F-zK,9599,simonw,2021-08-28T00:30:51Z,2021-08-28T00:30:51Z,OWNER,"There's also the option for plugins to muck around with existing registered commands - this could get a bit untidy if multiple plugins try to do it, but being able to replace `serve` with a fresh implementation that adds an additional command-line option before calling back to the original might open up some interesting possibilities.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907542214,https://api.github.com/repos/simonw/datasette/issues/1449,907542214,IC_kwDOBm6k_c42F_7G,9599,simonw,2021-08-28T01:02:38Z,2021-08-28T01:02:38Z,OWNER,"Partial prototype of `datasette-verify`: ```python from datasette import hookimpl import click @hookimpl def register_commands(cli): from datasette.cli import sqlite_extensions @cli.command() @click.argument(""files"", type=click.Path(exists=True), nargs=-1) @sqlite_extensions def verify(files, sqlite_extensions): ""Verify that files can be opened by Datasette"" for file in files: print(file) ``` I had to move the `from datasette.cli import sqlite_extensions` inside the hook function to avoid a circular import.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907538940,https://api.github.com/repos/simonw/datasette/issues/1449,907538940,IC_kwDOBm6k_c42F_H8,9599,simonw,2021-08-28T00:39:28Z,2021-08-28T00:39:28Z,OWNER,"Here's the answer to that: ``` ~ % datasette --help Usage: datasette [OPTIONS] COMMAND [ARGS]... Datasette! Options: --version Show the version and exit. --help Show this message and exit. Commands: serve* Serve up specified SQLite database files with a web UI inspect install Install Python packages - e.g. package Package specified SQLite files into a new datasette Docker... plugins List currently available plugins publish Publish specified SQLite database files to the internet... uninstall Uninstall Python packages (e.g. ``` Since it's adding extra things that show up in `--help` under the ""Commands:"" heading, I should call them commands.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907539065,https://api.github.com/repos/simonw/datasette/issues/1449,907539065,IC_kwDOBm6k_c42F_J5,9599,simonw,2021-08-28T00:40:24Z,2021-08-28T00:40:24Z,OWNER,I'm going to call the new hook `register_commands` - since it will allow ambitious plugins to register more than one command if they want to. That's also pleasingly similar to `register_routes`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907539251,https://api.github.com/repos/simonw/datasette/issues/1449,907539251,IC_kwDOBm6k_c42F_Mz,9599,simonw,2021-08-28T00:41:37Z,2021-08-28T00:41:50Z,OWNER,The first example plugin I'm going to build for this will be `datasette verify file.db file2.db` - it will take one or more paths to SQLite files and verify if they can be opened by Datasette or not.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907539668,https://api.github.com/repos/simonw/datasette/issues/1449,907539668,IC_kwDOBm6k_c42F_TU,9599,simonw,2021-08-28T00:44:16Z,2021-08-28T00:44:16Z,OWNER,"Considering this piece of code: https://github.com/simonw/datasette/blob/a1a33bb5822214be1cebd98cd858b2058d91a4aa/datasette/cli.py#L122-L142 I think the hook itself gets called with a single argument, `cli`, which it can then use in the standard Click way to register extra stuff. I can't pass it `datasette` (like I do with `register_routes()`) because the Datasette object itself is instantiated by the `serve` command, which will not have been called.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1450#issuecomment-907540240,https://api.github.com/repos/simonw/datasette/issues/1450,907540240,IC_kwDOBm6k_c42F_cQ,9599,simonw,2021-08-28T00:48:30Z,2021-08-28T00:48:30Z,OWNER,"I'll go with this: ``` % datasette --help Usage: datasette [OPTIONS] COMMAND [ARGS]... Datasette is an open source multi-tool for exploring and publishing data About Datasette: https://datasette.io/ Full documentation: https://docs.datasette.io/ Options: --version Show the version and exit. --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981681138,"Datasette --help should show something more useful than ""Datasette!""", https://github.com/simonw/datasette/issues/1449#issuecomment-907540790,https://api.github.com/repos/simonw/datasette/issues/1449,907540790,IC_kwDOBm6k_c42F_k2,9599,simonw,2021-08-28T00:52:32Z,2021-08-28T00:52:32Z,OWNER,I don't think I can get this new hook to support the handy [--plugins-dir= mechanism](https://docs.datasette.io/en/stable/plugins.html#one-off-plugins-using-plugins-dir) for loading plugins from Python files as opposed to registering them with setuptools - that mechanism is itself implemented inside of code called by `datasette serve` so I don't have a way of taking advantage of it from outside that command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907540928,https://api.github.com/repos/simonw/datasette/issues/1449,907540928,IC_kwDOBm6k_c42F_nA,9599,simonw,2021-08-28T00:53:37Z,2021-08-28T00:53:37Z,OWNER,I'll probably have to use this mechanism for the tests then: https://til.simonwillison.net/pytest/registering-plugins-in-tests,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907543982,https://api.github.com/repos/simonw/datasette/issues/1449,907543982,IC_kwDOBm6k_c42GAWu,9599,simonw,2021-08-28T01:14:27Z,2021-08-28T01:14:27Z,OWNER,"Writing the test for this is proving difficult, because the `cli` module has already been imported when I attempt to register a new plugin - so it doesn't pick up on the additional command registrations. Trying to work around that with `importlib.reload(cli)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907547624,https://api.github.com/repos/simonw/datasette/issues/1449,907547624,IC_kwDOBm6k_c42GBPo,9599,simonw,2021-08-28T01:44:57Z,2021-08-28T01:58:35Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#register-commands-cli,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907547736,https://api.github.com/repos/simonw/datasette/issues/1449,907547736,IC_kwDOBm6k_c42GBRY,9599,simonw,2021-08-28T01:45:36Z,2021-08-28T01:45:36Z,OWNER,I need to push this out as an alpha so I can release a demo plugin that uses it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1446#issuecomment-908832938,https://api.github.com/repos/simonw/datasette/issues/1446,908832938,IC_kwDOBm6k_c42K7Cq,9599,simonw,2021-08-31T01:54:59Z,2021-08-31T01:54:59Z,OWNER,I used the sticky footer mechanism in `datasette.app`: https://github.com/simonw/datasette.app/issues/3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978357984,Modify base.html template to support optional sticky footer, https://github.com/simonw/datasette/pull/1455#issuecomment-913001298,https://api.github.com/repos/simonw/datasette/issues/1455,913001298,IC_kwDOBm6k_c42a0tS,9599,simonw,2021-09-04T16:31:32Z,2021-09-04T16:31:32Z,OWNER,Great idea!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988325628,Add scientists to target groups, https://github.com/simonw/datasette/pull/1455#issuecomment-913001416,https://api.github.com/repos/simonw/datasette/issues/1455,913001416,IC_kwDOBm6k_c42a0vI,9599,simonw,2021-09-04T16:32:21Z,2021-09-04T16:32:21Z,OWNER,I'll add researchers too.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988325628,Add scientists to target groups, https://github.com/simonw/datasette/issues/1459#issuecomment-913218494,https://api.github.com/repos/simonw/datasette/issues/1459,913218494,IC_kwDOBm6k_c42bpu-,9599,simonw,2021-09-05T19:58:51Z,2021-09-05T19:59:15Z,OWNER,"This idea makes sense to me. There's actually an existing option that takes a path, called `--get` - it returns the HTML or JSON for that oath directly to the console, eg `datasette my.db --get /mydb/mytable.json` So... one option would be to allow combining that with `-o` to open that URL in the browser: datasette my.db -o --get /mydb So some options here are: - `datasette my.db --open-url /mydb` - `datasette my.db --open-path /mydb` - `datasette my.db --open --get /mydb` I quite like that last combination option, mainly to avoid adding even more command options. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988556488,suggestion: allow `datasette --open` to take a relative URL, https://github.com/simonw/datasette/issues/1461#issuecomment-914439356,https://api.github.com/repos/simonw/datasette/issues/1461,914439356,IC_kwDOBm6k_c42gTy8,9599,simonw,2021-09-07T16:11:37Z,2021-09-07T16:11:37Z,OWNER,"``` (datasette) datasette % blacken-docs docs/*.rst docs/authentication.rst: Rewriting... docs/internals.rst:169: code block parse error Cannot parse: 14:0: docs/plugin_hooks.rst:251: code block parse error Cannot parse: 6:4: ] docs/plugin_hooks.rst:312: code block parse error Cannot parse: 38:0: docs/spatialite.rst: Rewriting... docs/testing_plugins.rst:135: code block parse error Cannot parse: 5:0: docs/writing_plugins.rst: Rewriting... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",989986586,Try blacken-docs, https://github.com/simonw/datasette/issues/1461#issuecomment-914440282,https://api.github.com/repos/simonw/datasette/issues/1461,914440282,IC_kwDOBm6k_c42gUBa,9599,simonw,2021-09-07T16:12:57Z,2021-09-07T16:12:57Z,OWNER,"Here's the diff it produced from that first run: ```diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..8008023 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,7 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so: .. code-block:: python response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - } - }, ""actor"")) + response.set_cookie(""ds_actor"", datasette.sign({""a"": {""id"": ""cleopaws""}}, ""actor"")) Note that you need to pass ``""actor""`` as the namespace to :ref:`datasette_sign`. @@ -412,12 +408,16 @@ To include an expiry, add a ``""e""`` key to the cookie value containing a `base62 expires_at = int(time.time()) + (24 * 60 * 60) response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - }, - ""e"": baseconv.base62.encode(expires_at), - }, ""actor"")) + response.set_cookie( + ""ds_actor"", + datasette.sign( + { + ""a"": {""id"": ""cleopaws""}, + ""e"": baseconv.base62.encode(expires_at), + }, + ""actor"", + ), + ) The resulting cookie will encode data that looks something like this: diff --git a/docs/spatialite.rst b/docs/spatialite.rst index d1b300b..556bad8 100644 --- a/docs/spatialite.rst +++ b/docs/spatialite.rst @@ -58,19 +58,22 @@ Here's a recipe for taking a table with existing latitude and longitude columns, .. code-block:: python import sqlite3 - conn = sqlite3.connect('museums.db') + + conn = sqlite3.connect(""museums.db"") # Lead the spatialite extension: conn.enable_load_extension(True) - conn.load_extension('/usr/local/lib/mod_spatialite.dylib') + conn.load_extension(""/usr/local/lib/mod_spatialite.dylib"") # Initialize spatial metadata for this database: - conn.execute('select InitSpatialMetadata(1)') + conn.execute(""select InitSpatialMetadata(1)"") # Add a geometry column called point_geom to our museums table: conn.execute(""SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);"") # Now update that geometry column with the lat/lon points - conn.execute(''' + conn.execute( + """""" UPDATE museums SET point_geom = GeomFromText('POINT('||""longitude""||' '||""latitude""||')',4326); - ''') + """""" + ) # Now add a spatial index to that column conn.execute('select CreateSpatialIndex(""museums"", ""point_geom"");') # If you don't commit your changes will not be persisted: @@ -186,13 +189,14 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac .. code-block:: python import sqlite3 - conn = sqlite3.connect('places.db') + + conn = sqlite3.connect(""places.db"") # Enable SpatialLite extension conn.enable_load_extension(True) - conn.load_extension('/usr/local/lib/mod_spatialite.dylib') + conn.load_extension(""/usr/local/lib/mod_spatialite.dylib"") # Create the masic countries table - conn.execute('select InitSpatialMetadata(1)') - conn.execute('create table places (id integer primary key, name text);') + conn.execute(""select InitSpatialMetadata(1)"") + conn.execute(""create table places (id integer primary key, name text);"") # Add a MULTIPOLYGON Geometry column conn.execute(""SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);"") # Add a spatial index against the new column @@ -201,13 +205,17 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac from shapely.geometry.multipolygon import MultiPolygon from shapely.geometry import shape import requests - geojson = requests.get('https://data.whosonfirst.org/404/227/475/404227475.geojson').json() + + geojson = requests.get( + ""https://data.whosonfirst.org/404/227/475/404227475.geojson"" + ).json() # Convert to ""Well Known Text"" format - wkt = shape(geojson['geometry']).wkt + wkt = shape(geojson[""geometry""]).wkt # Insert and commit the record - conn.execute(""INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))"", ( - ""Wales"", wkt - )) + conn.execute( + ""INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))"", + (""Wales"", wkt), + ) conn.commit() Querying polygons using within() diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst index bd60a4b..5af01f6 100644 --- a/docs/writing_plugins.rst +++ b/docs/writing_plugins.rst @@ -18,9 +18,10 @@ The quickest way to start writing a plugin is to create a ``my_plugin.py`` file from datasette import hookimpl + @hookimpl def prepare_connection(conn): - conn.create_function('hello_world', 0, lambda: 'Hello world!') + conn.create_function(""hello_world"", 0, lambda: ""Hello world!"") If you save this in ``plugins/my_plugin.py`` you can then start Datasette like this:: @@ -60,22 +61,18 @@ The example consists of two files: a ``setup.py`` file that defines the plugin: from setuptools import setup - VERSION = '0.1' + VERSION = ""0.1"" setup( - name='datasette-plugin-demos', - description='Examples of plugins for Datasette', - author='Simon Willison', - url='https://github.com/simonw/datasette-plugin-demos', - license='Apache License, Version 2.0', + name=""datasette-plugin-demos"", + description=""Examples of plugins for Datasette"", + author=""Simon Willison"", + url=""https://github.com/simonw/datasette-plugin-demos"", + license=""Apache License, Version 2.0"", version=VERSION, - py_modules=['datasette_plugin_demos'], - entry_points={ - 'datasette': [ - 'plugin_demos = datasette_plugin_demos' - ] - }, - install_requires=['datasette'] + py_modules=[""datasette_plugin_demos""], + entry_points={""datasette"": [""plugin_demos = datasette_plugin_demos""]}, + install_requires=[""datasette""], ) And a Python module file, ``datasette_plugin_demos.py``, that implements the plugin: @@ -88,12 +85,12 @@ And a Python module file, ``datasette_plugin_demos.py``, that implements the plu @hookimpl def prepare_jinja2_environment(env): - env.filters['uppercase'] = lambda u: u.upper() + env.filters[""uppercase""] = lambda u: u.upper() @hookimpl def prepare_connection(conn): - conn.create_function('random_integer', 2, random.randint) + conn.create_function(""random_integer"", 2, random.randint) Having built a plugin in this way you can turn it into an installable package using the following command:: @@ -123,11 +120,13 @@ To bundle the static assets for a plugin in the package that you publish to PyPI .. code-block:: python - package_data={ - 'datasette_plugin_name': [ - 'static/plugin.js', - ], - }, + package_data = ( + { + ""datasette_plugin_name"": [ + ""static/plugin.js"", + ], + }, + ) Where ``datasette_plugin_name`` is the name of the plugin package (note that it uses underscores, not hyphens) and ``static/plugin.js`` is the path within that package to the static file. @@ -152,11 +151,13 @@ Templates should be bundled for distribution using the same ``package_data`` mec .. code-block:: python - package_data={ - 'datasette_plugin_name': [ - 'templates/my_template.html', - ], - }, + package_data = ( + { + ""datasette_plugin_name"": [ + ""templates/my_template.html"", + ], + }, + ) You can also use wildcards here such as ``templates/*.html``. See `datasette-edit-schema `__ for an example of this pattern. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",989986586,Try blacken-docs, https://github.com/simonw/datasette/issues/1461#issuecomment-914441037,https://api.github.com/repos/simonw/datasette/issues/1461,914441037,IC_kwDOBm6k_c42gUNN,9599,simonw,2021-09-07T16:13:59Z,2021-09-07T16:13:59Z,OWNER,"I don't think I'll adopt it for this project. For example, here: ```diff response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - } - }, ""actor"")) + response.set_cookie(""ds_actor"", datasette.sign({""a"": {""id"": ""cleopaws""}}, ""actor"")) ``` I chose to use the multi-line version to help emphasize the structure - the single-line replacement loses that. I think I'll continue to make my own editorial choices about how the code examples are laid out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",989986586,Try blacken-docs, https://github.com/simonw/datasette/issues/1462#issuecomment-914644260,https://api.github.com/repos/simonw/datasette/issues/1462,914644260,IC_kwDOBm6k_c42hF0k,9599,simonw,2021-09-07T21:34:32Z,2021-09-07T21:34:32Z,OWNER,"I think this is a setting. There are two relevant settings at the moment: ``` ""template_debug"": false, ""trace_debug"": false, ``` For consistence then this should be called `something_debug` - but do I want a single setting that exposes the `_internal` database and adds those debug options to the menu, or do I want those as two separate settings? - `internal_debug` to enable access to that `_internal` database - `menu_debug` for those menu options?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990367646,"Separate out ""debug"" options from ""root"" options", https://github.com/simonw/datasette/issues/1468#issuecomment-917839801,https://api.github.com/repos/simonw/datasette/issues/1468,917839801,IC_kwDOBm6k_c42tR-5,9599,simonw,2021-09-13T04:54:17Z,2021-09-13T04:54:17Z,OWNER,Here's a already open issue for this: #972,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994390593,Faceting for custom SQL queries, https://github.com/simonw/datasette/issues/1468#issuecomment-917839507,https://api.github.com/repos/simonw/datasette/issues/1468,917839507,IC_kwDOBm6k_c42tR6T,9599,simonw,2021-09-13T04:53:22Z,2021-09-13T04:53:22Z,OWNER,"At the moment this isn't possible - though there's a workaround which is to define a SQL view for the query, at which point facets will be displayed again. I did a lot of the work required to support this when I refactored how facets worked a while back - but to finally implement this I need to refactor the table view and the arbitrary query view to share much more logic than they do at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994390593,Faceting for custom SQL queries, https://github.com/simonw/datasette/issues/1469#issuecomment-917839062,https://api.github.com/repos/simonw/datasette/issues/1469,917839062,IC_kwDOBm6k_c42tRzW,9599,simonw,2021-09-13T04:52:01Z,2021-09-13T04:52:01Z,OWNER,Here's the code at fault: https://github.com/simonw/datasette/blob/b28b6cd2fe97f7e193a235877abeec2c8eb0a821/datasette/static/table.js#L137-L146,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994450961,"Column cog shows ""facet by this"" when already default faceted", https://github.com/simonw/datasette/issues/1466#issuecomment-917840012,https://api.github.com/repos/simonw/datasette/issues/1466,917840012,IC_kwDOBm6k_c42tSCM,9599,simonw,2021-09-13T04:54:59Z,2021-09-13T04:54:59Z,OWNER,Especially relevant now that 0.2.0 is out which is a much higher quality release. https://github.com/simonw/datasette-app/releases/tag/0.2.0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991467558,Add Datasette Desktop to installation documentation, https://github.com/simonw/datasette/pull/1481#issuecomment-939074818,https://api.github.com/repos/simonw/datasette/issues/1481,939074818,IC_kwDOBm6k_c43-SUC,9599,simonw,2021-10-08T19:40:23Z,2021-10-08T19:40:23Z,OWNER,"Then I created myself a temporary 3.10 environment using `pipenv` like so: cd /tmp mkdir py310 cd py310 pipenv shell --python /Users/simon/.pyenv/versions/3.10.0/bin/python And used that with my Datasette checkout like so: cd ~/.../datasette pip install -e '.[test]' pytest ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939075686,https://api.github.com/repos/simonw/datasette/issues/1481,939075686,IC_kwDOBm6k_c43-Shm,9599,simonw,2021-10-08T19:42:00Z,2021-10-08T19:42:00Z,OWNER,"Running `pytest -x --pdb` helped me see this error: ``` File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 122, in dispatch_request await self.ds.refresh_schemas() File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 344, in refresh_schemas await self._refresh_schemas() File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 349, in _refresh_schemas await init_internal_db(internal_db) File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/internal_db.py"", line 5, in init_internal_db await db.execute_write( File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 102, in execute_write return await self.execute_write_fn(_inner, block=block) File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 113, in execute_write_fn reply_queue = janus.Queue() File ""/Users/simon/.local/share/virtualenvs/py310-Z8fTATkJ/lib/python3.10/site-packages/janus/__init__.py"", line 39, in __init__ self._async_not_empty = asyncio.Condition(self._async_mutex) File ""/Users/simon/.pyenv/versions/3.10.0/lib/python3.10/asyncio/locks.py"", line 234, in __init__ raise ValueError(""loop argument must agree with lock"") ValueError: loop argument must agree with lock ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939076399,https://api.github.com/repos/simonw/datasette/issues/1481,939076399,IC_kwDOBm6k_c43-Ssv,9599,simonw,2021-10-08T19:43:33Z,2021-10-08T19:43:33Z,OWNER,"So maybe this is an issue with Janus? I'm using https://pypi.org/project/janus/ 0.6.1 which is the latest release, from October 2020.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939078095,https://api.github.com/repos/simonw/datasette/issues/1481,939078095,IC_kwDOBm6k_c43-THP,9599,simonw,2021-10-08T19:47:29Z,2021-10-08T19:47:29Z,OWNER,"Only mention I can find of that ""loop argument must agree with lock"" error is here - which doesn't have any tips for a workaround yet: https://giters.com/django/channels_redis/issues/278","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939078872,https://api.github.com/repos/simonw/datasette/issues/1481,939078872,IC_kwDOBm6k_c43-TTY,9599,simonw,2021-10-08T19:49:08Z,2021-10-08T19:49:08Z,OWNER,"Here's the code that raises that error: https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/asyncio/locks.py#L219-L234 ```python class Condition(_ContextManagerMixin, mixins._LoopBoundMixin): """"""Asynchronous equivalent to threading.Condition. This class implements condition variable objects. A condition variable allows one or more coroutines to wait until they are notified by another coroutine. A new Lock object is created and used as the underlying lock. """""" def __init__(self, lock=None, *, loop=mixins._marker): super().__init__(loop=loop) if lock is None: lock = Lock() elif lock._loop is not self._get_loop(): raise ValueError(""loop argument must agree with lock"") ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939079727,https://api.github.com/repos/simonw/datasette/issues/1481,939079727,IC_kwDOBm6k_c43-Tgv,9599,simonw,2021-10-08T19:50:52Z,2021-10-08T19:50:52Z,OWNER,"And here's the relevant Janus code: https://github.com/aio-libs/janus/blob/d7970f8b76bcac2e087067ca4575ac845e481874/janus/__init__.py#L24-L42 ```python class Queue(Generic[T]): def __init__(self, maxsize: int = 0) -> None: self._loop = current_loop() self._maxsize = maxsize self._init(maxsize) self._unfinished_tasks = 0 self._sync_mutex = threading.Lock() self._sync_not_empty = threading.Condition(self._sync_mutex) self._sync_not_full = threading.Condition(self._sync_mutex) self._all_tasks_done = threading.Condition(self._sync_mutex) self._async_mutex = asyncio.Lock() # ""loop argument must agree with lock"" exception is raised here: self._async_not_empty = asyncio.Condition(self._async_mutex) self._async_not_full = asyncio.Condition(self._async_mutex) self._finished = asyncio.Event() self._finished.set() ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939100803,https://api.github.com/repos/simonw/datasette/issues/1481,939100803,IC_kwDOBm6k_c43-YqD,9599,simonw,2021-10-08T20:33:42Z,2021-10-08T20:33:42Z,OWNER,"There's a tiny chance this could be a bug in Python 3.10 itself - I filed an issue here: https://bugs.python.org/issue45416 - in which I said: > In Python 3.10 it is not possible to instantiate an asyncio.Condition that wraps an asyncio.Lock without raising a ""loop argument must agree with lock"" exception. > > This code raises that exception: > > asyncio.Condition(asyncio.Lock()) > > This worked in previous Python versions. > > Note that the error only occurs if an event loop is running. Here's a simple script that replicates the problem: > > import asyncio > > # This runs without an exception: > print(asyncio.Condition(asyncio.Lock())) > > # This does not work: > async def example(): > print(asyncio.Condition(asyncio.Lock())) > > # This raises ""ValueError: loop argument must agree with lock"": > asyncio.run(example())","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939180313,https://api.github.com/repos/simonw/datasette/issues/1481,939180313,IC_kwDOBm6k_c43-sEZ,9599,simonw,2021-10-08T23:41:39Z,2021-10-08T23:41:39Z,OWNER,I submitted a PR to Janus with a workaround for this: https://github.com/aio-libs/janus/pull/359,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939185319,https://api.github.com/repos/simonw/datasette/issues/1481,939185319,IC_kwDOBm6k_c43-tSn,9599,simonw,2021-10-09T00:04:54Z,2021-10-09T00:04:54Z,OWNER,"I applied my PR against Janus to my local copy of Datasette like so: pip uninstall janus pip install https://github.com/aio-libs/janus/archive/9e13d3fb74e2c93d7501443b370a455d1b302b1f.zip Then I ran the Datasette tests and got a much happier pass rate. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/issues/1482#issuecomment-939191311,https://api.github.com/repos/simonw/datasette/issues/1482,939191311,IC_kwDOBm6k_c43-uwP,9599,simonw,2021-10-09T00:35:04Z,2021-10-09T00:35:04Z,OWNER,I think that SQLite error message difference was caused by https://github.com/python/cpython/commit/a50e28377bcf37121b55c2de70d95a5386c478f8 or related work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1021550542,Support Python 3.10, https://github.com/simonw/datasette/issues/1470#issuecomment-938124652,https://api.github.com/repos/simonw/datasette/issues/1470,938124652,IC_kwDOBm6k_c436qVs,9599,simonw,2021-10-07T20:17:53Z,2021-10-07T20:18:55Z,OWNER,"Here's the exception: ``` -> params[f""p{len(params)}""] = components[0] (Pdb) list 603 604 # Figure out the SQL for next-based-on-primary-key first 605 next_by_pk_clauses = [] 606 if use_rowid: 607 next_by_pk_clauses.append(f""rowid > :p{len(params)}"") 608 -> params[f""p{len(params)}""] = components[0] 609 else: 610 # Apply the tie-breaker based on primary keys 611 if len(components) == len(pks): 612 param_len = len(params) 613 next_by_pk_clauses.append( ``` Debugger shows that `components` is an empty array, so `components[0]` cannot be resolved: ``` -> params[f""p{len(params)}""] = components[0] (Pdb) params {'search': 'hello'} (Pdb) components [] ``` So the bug is in this code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/views/table.py#L604-L617 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error, https://github.com/simonw/datasette/issues/1470#issuecomment-938131806,https://api.github.com/repos/simonw/datasette/issues/1470,938131806,IC_kwDOBm6k_c436sFe,9599,simonw,2021-10-07T20:28:30Z,2021-10-07T20:28:30Z,OWNER,"On further investigation this isn't related to `_search` at all - it happens when you explicitly sort by `_sort=rowid` and apply a `_next` - https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 works without an error (currently) - https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid shows that error","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error, https://github.com/simonw/datasette/issues/1480#issuecomment-938134038,https://api.github.com/repos/simonw/datasette/issues/1480,938134038,IC_kwDOBm6k_c436soW,9599,simonw,2021-10-07T20:31:46Z,2021-10-07T20:31:46Z,OWNER,"I've had this problem too - my solution was to not use Cloud Run for databases larger than about 2GB, but the way you describe it here makes me think that maybe there is a workaround here which could get it to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database, https://github.com/simonw/datasette/pull/1481#issuecomment-938142436,https://api.github.com/repos/simonw/datasette/issues/1481,938142436,IC_kwDOBm6k_c436urk,9599,simonw,2021-10-07T20:44:43Z,2021-10-07T20:44:43Z,OWNER,"The 3.10 tests failed a lot. Trying to run this locally: ``` /tmp % pyenv install 3.10 python-build: definition not found: 3.10 The following versions contain `3.10' in the name: 3.10.0a6 3.10-dev miniconda-3.10.1 miniconda3-3.10.1 See all available versions with `pyenv install --list'. If the version you need is missing, try upgrading pyenv: brew update && brew upgrade pyenv ``` So trying: brew update && brew upgrade pyenv Then did this: ``` /tmp % brew upgrade pyenv ==> Upgrading 1 outdated package: pyenv 1.2.24.1 -> 2.1.0 ``` This decided to upgrade everything by downloaded everything on the internet. Aah, Homebrew. But it looks like I have `3.10.0` available to `pyenv` now. ``` /tmp % pyenv install 3.10.0 python-build: use openssl@1.1 from homebrew python-build: use readline from homebrew Downloading Python-3.10.0.tar.xz... -> https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tar.xz Installing Python-3.10.0... ... ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/issues/111#issuecomment-923106887,https://api.github.com/repos/simonw/datasette/issues/111,923106887,IC_kwDOBm6k_c43BX5H,9599,simonw,2021-09-20T16:58:39Z,2021-09-20T16:58:39Z,OWNER,Still a good idea today too! Would be great for https://cdc-vaccination-history.datasette.io/ for example.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924432643,https://api.github.com/repos/simonw/datasette/issues/111,924432643,IC_kwDOBm6k_c43GbkD,9599,simonw,2021-09-21T22:23:23Z,2021-09-21T22:23:23Z,OWNER,I'm going to use https://github.com/dateutil/dateutil for this - it's been maintained constantly (by an evolving team of contributors) [since 2003](https://github.com/dateutil/dateutil/commit/68ae2757ae15c84bf947d47a82a314b3b975bc9b) and is a very trustworthy dependency.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924437942,https://api.github.com/repos/simonw/datasette/issues/111,924437942,IC_kwDOBm6k_c43Gc22,9599,simonw,2021-09-21T22:32:59Z,2021-09-21T22:47:07Z,OWNER,"Side-note: Django 4.0 [will switch](https://docs.djangoproject.com/en/dev/releases/4.0/#zoneinfo-default-timezone-implementation) from using `pytz` to using the standard library `zoneinfo` module introduced in Python 3.9, which has a backport that works as far back as 3.6: https://github.com/pganssle/zoneinfo (https://pypi.org/project/backports.zoneinfo/) If I need to handle timezones I'll use that, but I think I can get away without it? Django does this: https://github.com/django/django/blob/b0ed619303d2fb723330ca9efa3acf23d49f1d19/setup.cfg#L39-L43 ``` install_requires = asgiref >= 3.3.2 backports.zoneinfo; python_version<""3.9"" sqlparse >= 0.2.2 tzdata; sys_platform == 'win32' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924435971,https://api.github.com/repos/simonw/datasette/issues/111,924435971,IC_kwDOBm6k_c43GcYD,9599,simonw,2021-09-21T22:29:15Z,2021-09-21T22:29:49Z,OWNER,"So this is a metadata key called `updated` which can be applied at the table, database or instance level. It is represented as a `.isoformat()` timestamp. Question: should I support just the date - `yyyy-mm-dd` - in addition to the datetime? I think so. I can easily imagine situations where the exact time of day that a change was made hasn't been recorded, but the overall date is known. But in that case, should the `updated` key sometimes be `yyyy-mm-dd` and sometimes be the full isoformat datetime? Or should there be an `updated_date` key that's used for just the date?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924438481,https://api.github.com/repos/simonw/datasette/issues/111,924438481,IC_kwDOBm6k_c43Gc_R,9599,simonw,2021-09-21T22:34:03Z,2021-09-21T22:34:21Z,OWNER,"The simplest possible version of this: it's always represented as a UTC ISO datetime, like this: ""updated"": ""2020-10-31T12:00:00+00:00"" Later versions of Datasette could extend this to handle other timezones or support just the date (though that's a backwards incompatible change so probably better to decide on the date thing right now).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924443089,https://api.github.com/repos/simonw/datasette/issues/111,924443089,IC_kwDOBm6k_c43GeHR,9599,simonw,2021-09-21T22:45:14Z,2021-09-21T22:45:26Z,OWNER,"The audiences I care about here are: - Producers of this timestamp - generally that will be users who are using `datasette publish` to share their data - Human consumers of this timestamp - end users who look at a Datasette site and want to know how recent the data is - Machine consumers of this timestamp - API integrations that might want to check if a Datasette instance has been updated before downloading new data For producers I think there are going to be two categories. The first is users who run ""publish"" and want the site to reflect when they did so (probably using `--updated=now` when they publish). The second are users who are willing to spend more time thinking about this - for example my various git scraping projects where I want to use a date derived from the git history. For humans... I'd like to be able to calculate a relative time (3 hours ago) in addition to showing the display time, because that helps avoid confusion over timezones. For machine consumers, it might be nice to offer the option of a calculated Unix timestamp-since-1970, since those can be easier to work with in some languages than running a full ISO date parser.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/1470#issuecomment-939386591,https://api.github.com/repos/simonw/datasette/issues/1470,939386591,IC_kwDOBm6k_c43_ebf,9599,simonw,2021-10-10T01:17:34Z,2021-10-10T01:17:34Z,OWNER,I'll open a separate issue for removing `_next=` when running a search.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error, https://github.com/simonw/datasette/issues/1479#issuecomment-932807859,https://api.github.com/repos/simonw/datasette/issues/1479,932807859,IC_kwDOBm6k_c43mYSz,9599,simonw,2021-10-02T19:22:35Z,2021-10-02T19:22:35Z,OWNER,"I'm pretty sure this is a Windows issue, not a Fly issue. I imagine it affects other forms of `datasette publish` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish", https://github.com/simonw/datasette/issues/1479#issuecomment-932808043,https://api.github.com/repos/simonw/datasette/issues/1479,932808043,IC_kwDOBm6k_c43mYVr,9599,simonw,2021-10-02T19:23:52Z,2021-10-02T19:23:52Z,OWNER,I suspect the root cause of this may be in this code: https://github.com/simonw/datasette/blob/63886178a649586b403966a27a45881709d2b868/datasette/utils/__init__.py#L673-L677,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish", https://github.com/simonw/datasette/issues/1479#issuecomment-932808216,https://api.github.com/repos/simonw/datasette/issues/1479,932808216,IC_kwDOBm6k_c43mYYY,9599,simonw,2021-10-02T19:25:09Z,2021-10-02T19:25:09Z,OWNER,"Actually no, from that stack trace you provided: ``` File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 610, in invoke return callback(*args, **kwargs) File ""c:\users\grott\anaconda3\lib\site-packages\datasette\cli.py"", line 283, in package call(args) File ""c:\users\grott\anaconda3\lib\contextlib.py"", line 119, in __exit__ next(self.gen) File ""c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py"", line 451, in temporary_docker_directory tmp.cleanup() ``` It looks like the problem occurs here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L449-L452","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish", https://github.com/simonw/datasette/issues/1497#issuecomment-953508979,https://api.github.com/repos/simonw/datasette/issues/1497,953508979,IC_kwDOBm6k_c441WRz,9599,simonw,2021-10-28T05:13:49Z,2021-10-28T05:13:49Z,OWNER,Wrote about this in my weeknotes: https://simonwillison.net/2021/Oct/28/weeknotes-kubernetes-web-components/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/pull/1487#issuecomment-942722595,https://api.github.com/repos/simonw/datasette/issues/1487,942722595,IC_kwDOBm6k_c44MM4j,9599,simonw,2021-10-13T21:08:53Z,2021-10-13T21:08:53Z,OWNER,Thanks for this!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1023245060,"Added instructions for installing plugins via pipx, #1486", https://github.com/simonw/datasette/issues/1469#issuecomment-942725632,https://api.github.com/repos/simonw/datasette/issues/1469,942725632,IC_kwDOBm6k_c44MNoA,9599,simonw,2021-10-13T21:13:30Z,2021-10-13T21:13:30Z,OWNER,"The core problem here is treating the `?_facet=` query string parameters as the point of truth for which facets are currently enabled. Instead, I could use a `data-` attribute on the displayed facets.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994450961,"Column cog shows ""facet by this"" when already default faceted", https://github.com/simonw/datasette/issues/1488#issuecomment-942779926,https://api.github.com/repos/simonw/datasette/issues/1488,942779926,IC_kwDOBm6k_c44Ma4W,9599,simonw,2021-10-13T22:59:05Z,2021-10-13T22:59:05Z,OWNER,This is weird - as far as I can tell `httpx` has included the query string in `raw_path` for well over a year: https://github.com/encode/httpx/commit/8e4a8a1c73f60fe5754f95b308beaa725cb8791d#diff-c9a78eb3b5f5c4fac4e5552165fbdd5320c7e3fadf9eedabcb5461393466c090R235,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1488#issuecomment-942777414,https://api.github.com/repos/simonw/datasette/issues/1488,942777414,IC_kwDOBm6k_c44MaRG,9599,simonw,2021-10-13T22:52:40Z,2021-10-13T22:52:40Z,OWNER,"Upgrading to 0.20.0 gives me lots of the following errors: '{""ok"": false, ""error"": ""Database not found: .json?_sort=relationships"", ""status"": 404, ""title"": null}' It looks like the full query string is now being treated as the name of the database. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1488#issuecomment-942778382,https://api.github.com/repos/simonw/datasette/issues/1488,942778382,IC_kwDOBm6k_c44MagO,9599,simonw,2021-10-13T22:55:01Z,2021-10-13T22:55:01Z,OWNER,"I think the issue is in `route_path()`: ``` > /Users/simon/Dropbox/Development/datasette/datasette/app.py(1182)route_path() -> response = await view(request, send) (Pdb) path '/_memory.json?sql=select+sqlite_version()' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1488#issuecomment-942778673,https://api.github.com/repos/simonw/datasette/issues/1488,942778673,IC_kwDOBm6k_c44Makx,9599,simonw,2021-10-13T22:55:44Z,2021-10-13T22:55:44Z,OWNER,"``` (Pdb) request.scope['path'] '/_memory.json' (Pdb) request.scope['raw_path'] b'/_memory.json?sql=select+sqlite_version()' ``` So `raw_path` now includes the query string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1488#issuecomment-942782673,https://api.github.com/repos/simonw/datasette/issues/1488,942782673,IC_kwDOBm6k_c44MbjR,9599,simonw,2021-10-13T23:04:54Z,2021-10-13T23:04:54Z,OWNER,"I think this is the change in `httpx` which is causing the bug for me: https://github.com/encode/httpx/commit/ff9813e84dab56f0f3c4ef3a159a4cce8c644a91#diff-0d0cbe9ebcd03cc8c780b0407762540a082f70cc64257f2fcd588cc30f43c15cR96 Previously it was using `path` from `path, _, query = full_path.partition(b""?"")` to populate the `raw_path` key - but it changed to instead using `request.url.raw_path` which presumably implements the logic that includes the query string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/pull/1489#issuecomment-943594712,https://api.github.com/repos/simonw/datasette/issues/1489,943594712,IC_kwDOBm6k_c44PhzY,9599,simonw,2021-10-14T18:04:11Z,2021-10-14T18:04:11Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026379132,"Update pyyaml requirement from ~=5.3 to >=5.3,<7.0", https://github.com/simonw/datasette/pull/1458#issuecomment-943620649,https://api.github.com/repos/simonw/datasette/issues/1458,943620649,IC_kwDOBm6k_c44PoIp,9599,simonw,2021-10-14T18:38:58Z,2021-10-14T18:38:58Z,OWNER,"This is a great idea, thanks.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988555009,Rework the `--static` documentation a bit, https://github.com/simonw/datasette/pull/1467#issuecomment-943623246,https://api.github.com/repos/simonw/datasette/issues/1467,943623246,IC_kwDOBm6k_c44PoxO,9599,simonw,2021-10-14T18:42:19Z,2021-10-14T18:42:19Z,OWNER,This looks like a good fix to me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991575770,Add Authorization header when CORS flag is set, https://github.com/simonw/datasette/pull/1467#issuecomment-943632697,https://api.github.com/repos/simonw/datasette/issues/1467,943632697,IC_kwDOBm6k_c44PrE5,9599,simonw,2021-10-14T18:54:18Z,2021-10-14T18:54:18Z,OWNER,The test there failed because it turns out there's a whole bunch of places that set the `Access-Control-Allow-Origin` header. I'm going to close this PR and ship a fix that refactors those places to use the same code.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",991575770,Add Authorization header when CORS flag is set, https://github.com/simonw/datasette/pull/1481#issuecomment-944986367,https://api.github.com/repos/simonw/datasette/issues/1481,944986367,IC_kwDOBm6k_c44U1j_,9599,simonw,2021-10-16T19:07:38Z,2021-10-16T19:09:02Z,OWNER,This is blocking an upgrade for the Homebrew Datasette package: https://github.com/Homebrew/homebrew-core/pull/86932,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-945020210,https://api.github.com/repos/simonw/datasette/issues/1481,945020210,IC_kwDOBm6k_c44U90y,9599,simonw,2021-10-16T23:19:51Z,2021-10-16T23:19:51Z,OWNER,"Since that Janus PR hasn't been merged yet, one temporary option for a fix would be to entirely vendor the fixed Janus - https://github.com/aio-libs/janus/blob/9e13d3fb74e2c93d7501443b370a455d1b302b1f/janus/__init__.py - since it's only a single module.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/issues/1470#issuecomment-946097058,https://api.github.com/repos/simonw/datasette/issues/1470,946097058,IC_kwDOBm6k_c44ZEui,9599,simonw,2021-10-18T19:30:15Z,2021-10-18T19:30:15Z,OWNER,https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid is fixed now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error, https://github.com/simonw/datasette/issues/1493#issuecomment-946360891,https://api.github.com/repos/simonw/datasette/issues/1493,946360891,IC_kwDOBm6k_c44aFI7,9599,simonw,2021-10-19T04:37:27Z,2021-10-19T04:37:27Z,OWNER,"I renamed `/:memory:` to `/_memory` in version 0.55 - https://docs.datasette.io/en/stable/changelog.html#v0-55 But... in 0.59 I stopped following HTTP redirects by default, which is why this used to work and no longer does! So the fix is to update the Homebrew regression test to use this instead: datasette --get '/_memory.json?sql=select+3*5' Thanks for catching this!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1028115674,`--get '/:memory:.json?sql=select+3*5'` error with datasette 0.59, https://github.com/simonw/datasette/issues/1496#issuecomment-949912718,https://api.github.com/repos/simonw/datasette/issues/1496,949912718,IC_kwDOBm6k_c44noSO,9599,simonw,2021-10-22T19:38:23Z,2021-10-22T19:38:23Z,OWNER,https://docs.datasette.io/en/latest/sql_queries.html#named-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033864602,Named parameters docs should include an example of a cast, https://github.com/simonw/datasette/issues/1482#issuecomment-950402273,https://api.github.com/repos/simonw/datasette/issues/1482,950402273,IC_kwDOBm6k_c44pfzh,9599,simonw,2021-10-24T22:00:29Z,2021-10-24T22:00:29Z,OWNER,Janus 0.6.2 is out now and should have the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1021550542,Support Python 3.10, https://github.com/simonw/datasette/pull/1495#issuecomment-950403521,https://api.github.com/repos/simonw/datasette/issues/1495,950403521,IC_kwDOBm6k_c44pgHB,9599,simonw,2021-10-24T22:09:18Z,2021-10-24T22:09:18Z,OWNER,"This is a great idea - I've wanted this myself before, but never spent any time thinking about how to achieve it. I think your design here is exactly right - an optional third item in the tuple consisting of a dictionary of options to pass to the view function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options, https://github.com/simonw/datasette/pull/1495#issuecomment-950403692,https://api.github.com/repos/simonw/datasette/issues/1495,950403692,IC_kwDOBm6k_c44pgJs,9599,simonw,2021-10-24T22:10:43Z,2021-10-24T22:10:43Z,OWNER,"To land this change we'll need a unit test that demonstrates the new capability - I suggest putting that next to this test: https://github.com/simonw/datasette/blob/15a9d4abfff0c45dee2a9f851326e1d61b1c678c/tests/test_plugins.py#L648-L659 It will also need documentation, which should be added here: https://github.com/simonw/datasette/blob/15a9d4abfff0c45dee2a9f851326e1d61b1c678c/docs/plugin_hooks.rst#register-routes-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options, https://github.com/simonw/datasette/issues/1497#issuecomment-950410554,https://api.github.com/repos/simonw/datasette/issues/1497,950410554,IC_kwDOBm6k_c44ph06,9599,simonw,2021-10-24T23:01:20Z,2021-10-24T23:01:28Z,OWNER,"I can replicate locally by running: ``` docker build -f Dockerfile \ -t datasetteproject/datasette:0.59.1 \ --build-arg VERSION=0.59.1 . ``` This gives me the same error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950410718,https://api.github.com/repos/simonw/datasette/issues/1497,950410718,IC_kwDOBm6k_c44ph3e,9599,simonw,2021-10-24T23:02:30Z,2021-10-24T23:02:30Z,OWNER,I got the same error publishing 0.59: https://github.com/simonw/datasette/actions/runs/1343251945,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950415129,https://api.github.com/repos/simonw/datasette/issues/1497,950415129,IC_kwDOBm6k_c44pi8Z,9599,simonw,2021-10-24T23:21:33Z,2021-10-24T23:21:33Z,OWNER,That fixed it! Resulting image is 249MB which is a very slight size reduction (I think previous was 259MB (uncompressed).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950411320,https://api.github.com/repos/simonw/datasette/issues/1497,950411320,IC_kwDOBm6k_c44piA4,9599,simonw,2021-10-24T23:06:05Z,2021-10-24T23:06:05Z,OWNER,"Right now the base image is: https://github.com/simonw/datasette/blob/e6e44372b34414eac2f36a4c1120af4f755aa423/Dockerfile#L1 I'm going to try `python:3.9.7-slim-buster` instead: https://hub.docker.com/layers/python/library/python/3.9.7-slim-buster/images/sha256-290b95e4b379762a9bd3d72644598e0972f4e2b5442bba60592c018fadcc744d?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950411417,https://api.github.com/repos/simonw/datasette/issues/1497,950411417,IC_kwDOBm6k_c44piCZ,9599,simonw,2021-10-24T23:06:45Z,2021-10-24T23:11:14Z,OWNER,"Same errors with `3.9.7`: ``` #5 41.46 /usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory #5 41.46 dpkg: error processing package libc6:amd64 (--configure): #5 41.46 installed libc6:amd64 package post-installation script subprocess returned error exit status 127 #5 41.47 Errors were encountered while processing: #5 41.47 libc6:amd64 #5 41.50 E: Sub-process /usr/bin/dpkg returned an error code (1) ``` I'm suspicious of this part of the `Dockerfile`: https://github.com/simonw/datasette/blob/e6e44372b34414eac2f36a4c1120af4f755aa423/Dockerfile#L1-L18 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950411808,https://api.github.com/repos/simonw/datasette/issues/1497,950411808,IC_kwDOBm6k_c44piIg,9599,simonw,2021-10-24T23:08:59Z,2021-10-24T23:08:59Z,OWNER,"Looks like it's this bug, reported on the Debian mailing list: https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1818037.html No obvious workaround there though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950411912,https://api.github.com/repos/simonw/datasette/issues/1497,950411912,IC_kwDOBm6k_c44piKI,9599,simonw,2021-10-24T23:09:41Z,2021-10-24T23:09:41Z,OWNER,Here that is in the Debian bug tracker: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993755,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950412628,https://api.github.com/repos/simonw/datasette/issues/1497,950412628,IC_kwDOBm6k_c44piVU,9599,simonw,2021-10-24T23:13:20Z,2021-10-24T23:13:27Z,OWNER,"I think the root cause here is that I'm using a Debian Buster base image and then installing SpatiaLite from Debian unstable (sid) - as described in this comment: https://github.com/simonw/datasette/issues/1249#issuecomment-804309510 That's has worked fine in the past, but Sid is unstable - and this seems to be one of those instabilities.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950413185,https://api.github.com/repos/simonw/datasette/issues/1497,950413185,IC_kwDOBm6k_c44pieB,9599,simonw,2021-10-24T23:16:25Z,2021-10-24T23:18:30Z,OWNER,"Debian stable these days is ""bullseye"" - https://www.debian.org/releases/ - which has the version of SpatiaLite that I was previously pulling in from Sid: https://packages.debian.org/bullseye/libsqlite3-mod-spatialite So upgrading to the 3.9.7-slim-bullseye base image may help. https://hub.docker.com/layers/python/library/python/3.9.7-slim-bullseye/images/sha256-67af5f544115124dc6d6da1d9d2815aa9825f6fd4aa6710adb0ec1725280fb89?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950415822,https://api.github.com/repos/simonw/datasette/issues/1497,950415822,IC_kwDOBm6k_c44pjHO,9599,simonw,2021-10-24T23:25:45Z,2021-10-24T23:25:45Z,OWNER,I'm going to attempt to publish `0.59` to Docker Hub using https://github.com/simonw/datasette/blob/2c31d1cd9cd3b63458ccbe391866499fa3f44978/.github/workflows/push_docker_tag.yml - if that works I'll push `0.59.1` as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416061,https://api.github.com/repos/simonw/datasette/issues/1497,950416061,IC_kwDOBm6k_c44pjK9,9599,simonw,2021-10-24T23:27:18Z,2021-10-24T23:27:18Z,OWNER,That worked: https://hub.docker.com/layers/datasetteproject/datasette/0.59/images/sha256-038decc28e0ea84b281ecc0058fe8eba7aa99596e5a2177ff714092ad03294ed?context=explore,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416460,https://api.github.com/repos/simonw/datasette/issues/1497,950416460,IC_kwDOBm6k_c44pjRM,9599,simonw,2021-10-24T23:30:10Z,2021-10-24T23:30:10Z,OWNER,"Testing that newly published image: ``` % docker run -p 8002:8001 -v `pwd`:/mnt \ datasetteproject/datasette:0.59 datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db Unable to find image 'datasetteproject/datasette:0.59' locally 0.59: Pulling from datasetteproject/datasette 7d63c13d9b9b: Already exists 6ad2a11ca37b: Already exists e9edbe81a001: Already exists 36629b83aba2: Already exists 7338abefe51c: Already exists 6d71b6b88b82: Pull complete 8c4da3c56bdc: Pull complete Digest: sha256:038decc28e0ea84b281ecc0058fe8eba7aa99596e5a2177ff714092ad03294ed Status: Downloaded newer image for datasetteproject/datasette:0.59 INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ``` and `http://localhost:8002/versions.json` returns: ```json { ""python"": { ""version"": ""3.9.7"", ""full"": ""3.9.7 (default, Oct 12 2021, 02:43:43) \n[GCC 10.2.1 20210110]"" }, ""datasette"": { ""version"": ""0.59"" }, ""asgi"": ""3.0"", ""uvicorn"": ""0.15.0"", ""sqlite"": { ""version"": ""3.34.1"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416659,https://api.github.com/repos/simonw/datasette/issues/1497,950416659,IC_kwDOBm6k_c44pjUT,9599,simonw,2021-10-24T23:31:41Z,2021-10-24T23:31:41Z,OWNER,"Published `0.59.1` as well: https://github.com/simonw/datasette/runs/3991214225?check_suite_focus=true Result: https://hub.docker.com/layers/datasetteproject/datasette/0.59.1/images/sha256-dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416682,https://api.github.com/repos/simonw/datasette/issues/1497,950416682,IC_kwDOBm6k_c44pjUq,9599,simonw,2021-10-24T23:31:51Z,2021-10-24T23:31:51Z,OWNER,One catch: the `latest` tag on Docker Hub is still three months old.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416802,https://api.github.com/repos/simonw/datasette/issues/1497,950416802,IC_kwDOBm6k_c44pjWi,9599,simonw,2021-10-24T23:32:39Z,2021-10-24T23:32:39Z,OWNER,"That's because the `publish.yml` workflow ends with this, which isn't in the `push_docker_tag.yml` workflow: https://github.com/simonw/datasette/blob/2c31d1cd9cd3b63458ccbe391866499fa3f44978/.github/workflows/publish.yml#L117-L119","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950417375,https://api.github.com/repos/simonw/datasette/issues/1497,950417375,IC_kwDOBm6k_c44pjff,9599,simonw,2021-10-24T23:36:54Z,2021-10-24T23:36:54Z,OWNER,"Tried fixing this by pushing a new `latest` tag from my laptop: ``` (datasette) datasette % docker pull datasetteproject/datasette:0.59.1 0.59.1: Pulling from datasetteproject/datasette 7d63c13d9b9b: Already exists 6ad2a11ca37b: Already exists e9edbe81a001: Already exists 36629b83aba2: Already exists 7338abefe51c: Already exists 6b825daddc6c: Pull complete d7508b065a21: Pull complete Digest: sha256:dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c Status: Downloaded newer image for datasetteproject/datasette:0.59.1 docker.io/datasetteproject/datasette:0.59.1 (datasette) datasette % docker tag datasetteproject/datasette:0.59.1 datasetteproject/datasette:latest (datasette) datasette % docker push datasetteproject/datasette:latest The push refers to repository [docker.io/datasetteproject/datasette] d668c99b6ff1: Layer already exists aa20c9013575: Layer already exists c97eebf2b227: Layer already exists 284a6c64b82c: Layer already exists 388eedeb736e: Layer already exists 2feece0964b8: Layer already exists e8b689711f21: Layer already exists errors: denied: requested access to the resource is denied unauthorized: authentication required ``` So I logged in with `docker login`: ``` (datasette) datasette % docker login Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username: datasetteproject Password: ``` And ran the push again and it worked: ``` (datasette) datasette % docker push datasetteproject/datasette:latest The push refers to repository [docker.io/datasetteproject/datasette] d668c99b6ff1: Layer already exists aa20c9013575: Layer already exists c97eebf2b227: Layer already exists 284a6c64b82c: Layer already exists 388eedeb736e: Layer already exists 2feece0964b8: Layer already exists e8b689711f21: Layer already exists latest: digest: sha256:dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c size: 1793 ``` https://hub.docker.com/layers/datasetteproject/datasette/latest/images/sha256-dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/878#issuecomment-970712713,https://api.github.com/repos/simonw/datasette/issues/878,970712713,IC_kwDOBm6k_c452-aJ,9599,simonw,2021-11-16T21:54:33Z,2021-11-16T21:54:33Z,OWNER,I'm going to continue working on this in a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970673085,https://api.github.com/repos/simonw/datasette/issues/878,970673085,IC_kwDOBm6k_c4520u9,9599,simonw,2021-11-16T20:58:24Z,2021-11-16T20:58:24Z,OWNER,"New test: ```python class Complex(AsyncBase): def __init__(self): self.log = [] async def d(self): await asyncio.sleep(random() * 0.1) print(""LOG: d"") self.log.append(""d"") async def c(self): await asyncio.sleep(random() * 0.1) print(""LOG: c"") self.log.append(""c"") async def b(self, c, d): print(""LOG: b"") self.log.append(""b"") async def a(self, b, c): print(""LOG: a"") self.log.append(""a"") async def go(self, a): print(""LOG: go"") self.log.append(""go"") return self.log @pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once assert tuple(result) in ( # c and d could happen in either order (""c"", ""d"", ""b"", ""a"", ""go""), (""d"", ""c"", ""b"", ""a"", ""go""), ) ``` And this code passes it: ```python import asyncio from functools import wraps import inspect try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def __new__(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {""_registry"": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == ""resolve"": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != ""self"" and not p.startswith(""_"") } for key, method in _registry.items() } new_attrs[""_graph""] = graph return super().__new__(cls, name, bases, new_attrs) def make_method(method): parameters = inspect.signature(method).parameters.keys() @wraps(method) async def inner(self, _results=None, **kwargs): print(""\n{}.{}({}) _results={}"".format(self, method.__name__, kwargs, _results)) # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != ""self""] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), ""The following DI parameters could not be found in the registry: {}"".format( missing ) results = {} results.update(kwargs) if to_resolve: resolved_parameters = await self.resolve(to_resolve, _results) results.update(resolved_parameters) return_value = await method(self, **results) if _results is not None: _results[method.__name__] = return_value return return_value return inner class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names, results=None): print(""\n resolve: "", names) if results is None: results = {} # Come up with an execution plan, just for these nodes ts = graphlib.TopologicalSorter() to_do = set(names) done = set() while to_do: item = to_do.pop() dependencies = self._graph[item] ts.add(item, *dependencies) done.add(item) # Add any not-done dependencies to the queue to_do.update({k for k in dependencies if k not in done}) ts.prepare() plan = [] while ts.is_active(): node_group = ts.get_ready() plan.append(node_group) ts.done(*node_group) print(""plan:"", plan) results = {} for node_group in plan: awaitables = [ self._registry[name]( self, _results=results, **{k: v for k, v in results.items() if k in self._graph[name]}, ) for name in node_group ] print("" results = "", results) print("" awaitables: "", awaitables) awaitable_results = await asyncio.gather(*awaitables) results.update( {p[0].__name__: p[1] for p in zip(awaitables, awaitable_results)} ) print("" End of resolve(), returning"", results) return {key: value for key, value in results.items() if key in names} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970705738,https://api.github.com/repos/simonw/datasette/issues/878,970705738,IC_kwDOBm6k_c4528tK,9599,simonw,2021-11-16T21:44:31Z,2021-11-16T21:44:31Z,OWNER,Wrote a TIL about what I learned using `TopologicalSorter`: https://til.simonwillison.net/python/graphlib-topologicalsorter,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1509#issuecomment-970544733,https://api.github.com/repos/simonw/datasette/issues/1509,970544733,IC_kwDOBm6k_c452VZd,9599,simonw,2021-11-16T18:22:32Z,2021-11-16T18:22:32Z,OWNER,"This is mainly happening here: - https://github.com/simonw/datasette/issues/782","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054243511,Datasette 1.0 JSON API (and documentation), https://github.com/simonw/datasette/issues/782#issuecomment-970554697,https://api.github.com/repos/simonw/datasette/issues/782,970554697,IC_kwDOBm6k_c452X1J,9599,simonw,2021-11-16T18:32:03Z,2021-11-16T18:32:03Z,OWNER,"I'm going to take another look at this: - https://github.com/simonw/datasette/issues/878","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/782#issuecomment-970553780,https://api.github.com/repos/simonw/datasette/issues/782,970553780,IC_kwDOBm6k_c452Xm0,9599,simonw,2021-11-16T18:30:51Z,2021-11-16T18:30:58Z,OWNER,"OK, I'm ready to start working on this today. I'm going to go with a default representation that looks like this: ```json { ""rows"": [ {""id"": 1, ""name"": ""One""}, {""id"": 2, ""name"": ""Two""} ], ""next_url"": null } ``` Note that there's no `count` - all it provides is the current selection of results and an indication as to how the next can be retrieved (`null` if there are no more results). I'll implement `?_extra=` to provide everything else.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/pull/1512#issuecomment-970718337,https://api.github.com/repos/simonw/datasette/issues/1512,970718337,IC_kwDOBm6k_c452_yB,9599,simonw,2021-11-16T22:02:30Z,2021-11-16T22:02:30Z,OWNER,"I've decided to make the clever `asyncio` dependency injection opt-in - so you can either decorate with `@inject` or you can set `inject_all = True` on the class - for example: ```python import asyncio from datasette.utils.asyncdi import AsyncBase, inject class Simple(AsyncBase): def __init__(self): self.log = [] @inject async def two(self): self.log.append(""two"") @inject async def one(self, two): self.log.append(""one"") return self.log async def not_inject(self, one, two): return one + two class Complex(AsyncBase): inject_all = True def __init__(self): self.log = [] async def b(self): self.log.append(""b"") async def a(self, b): self.log.append(""a"") async def go(self, a): self.log.append(""go"") return self.log ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/issues/878#issuecomment-970624197,https://api.github.com/repos/simonw/datasette/issues/878,970624197,IC_kwDOBm6k_c452ozF,9599,simonw,2021-11-16T19:49:05Z,2021-11-16T19:49:05Z,OWNER,"Here's the latest version of my weird dependency injection async class: ```python import inspect class AsyncMeta(type): def __new__(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {""_registry"": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == ""resolve"": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Topological sort of _registry by parameter dependencies graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != ""self"" and not p.startswith(""_"") } for key, method in _registry.items() } new_attrs[""_graph""] = graph return super().__new__(cls, name, bases, new_attrs) def make_method(method): @wraps(method) async def inner(self, **kwargs): parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != ""self""] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), ""The following DI parameters could not be found in the registry: {}"".format( missing ) results = {} results.update(kwargs) results.update(await self.resolve(to_resolve)) return await method(self, **results) return inner bad = [0] class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names): print("" resolve({})"".format(names)) results = {} # Resolve them in the correct order ts = TopologicalSorter() ts2 = TopologicalSorter() print("" names = "", names) print("" self._graph = "", self._graph) for name in names: if self._graph[name]: ts.add(name, *self._graph[name]) ts2.add(name, *self._graph[name]) print("" static_order ="", tuple(ts2.static_order())) ts.prepare() while ts.is_active(): print("" is_active, i = "", bad[0]) bad[0] += 1 if bad[0] > 20: print("" Infinite loop?"") break nodes = ts.get_ready() print("" Do nodes:"", nodes) awaitables = [self._registry[name](self, **{ k: v for k, v in results.items() if k in self._graph[name] }) for name in nodes] print("" awaitables: "", awaitables) awaitable_results = await asyncio.gather(*awaitables) results.update({ p[0].__name__: p[1] for p in zip(awaitables, awaitable_results) }) print(results) for node in nodes: ts.done(node) return results ``` Example usage: ```python class Foo(AsyncBase): async def graa(self, boff): print(""graa"") return 5 async def boff(self): print(""boff"") return 8 async def other(self, boff, graa): print(""other"") return 5 + boff + graa foo = Foo() await foo.other() ``` Output: ``` resolve(['boff', 'graa']) names = ['boff', 'graa'] self._graph = {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}} static_order = ('boff', 'graa') is_active, i = 0 Do nodes: ('boff',) awaitables: [] resolve([]) names = [] self._graph = {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}} static_order = () boff {'boff': 8} is_active, i = 1 Do nodes: ('graa',) awaitables: [] resolve([]) names = [] self._graph = {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}} static_order = () graa {'boff': 8, 'graa': 5} other 18 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970655304,https://api.github.com/repos/simonw/datasette/issues/878,970655304,IC_kwDOBm6k_c452wZI,9599,simonw,2021-11-16T20:32:16Z,2021-11-16T20:32:16Z,OWNER,"This code is really fiddly. I just got to this version: ```python import asyncio from functools import wraps import inspect try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def __new__(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {""_registry"": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == ""resolve"": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != ""self"" and not p.startswith(""_"") } for key, method in _registry.items() } new_attrs[""_graph""] = graph return super().__new__(cls, name, bases, new_attrs) def make_method(method): @wraps(method) async def inner(self, _results=None, **kwargs): print(""inner - _results="", _results) parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != ""self""] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), ""The following DI parameters could not be found in the registry: {}"".format( missing ) results = {} results.update(kwargs) if to_resolve: resolved_parameters = await self.resolve(to_resolve, _results) results.update(resolved_parameters) return_value = await method(self, **results) if _results is not None: _results[method.__name__] = return_value return return_value return inner class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names, results=None): print(""\n resolve: "", names) if results is None: results = {} # Resolve them in the correct order ts = graphlib.TopologicalSorter() for name in names: ts.add(name, *self._graph[name]) ts.prepare() async def resolve_nodes(nodes): print("" resolve_nodes"", nodes) print("" (current results = {})"".format(repr(results))) awaitables = [ self._registry[name]( self, _results=results, **{k: v for k, v in results.items() if k in self._graph[name]}, ) for name in nodes if name not in results ] print("" awaitables: "", awaitables) awaitable_results = await asyncio.gather(*awaitables) results.update( {p[0].__name__: p[1] for p in zip(awaitables, awaitable_results)} ) if not ts.is_active(): # Nothing has dependencies - just resolve directly print("" no dependencies, resolve directly"") await resolve_nodes(names) else: # Resolve in topological order while ts.is_active(): nodes = ts.get_ready() print("" ts.get_ready() returned nodes:"", nodes) await resolve_nodes(nodes) for node in nodes: ts.done(node) print("" End of resolve(), returning"", results) return {key: value for key, value in results.items() if key in names} ``` With this test: ```python class Complex(AsyncBase): def __init__(self): self.log = [] async def c(self): print(""LOG: c"") self.log.append(""c"") async def b(self, c): print(""LOG: b"") self.log.append(""b"") async def a(self, b, c): print(""LOG: a"") self.log.append(""a"") async def go(self, a): print(""LOG: go"") self.log.append(""go"") return self.log @pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once assert result == [""c"", ""b"", ""a"", ""go""] ``` This test sometimes passes, and sometimes fails! Output for a pass: ``` tests/test_asyncdi.py inner - _results= None resolve: ['a'] ts.get_ready() returned nodes: ('c', 'b') resolve_nodes ('c', 'b') (current results = {}) awaitables: [, ] inner - _results= {} LOG: c inner - _results= {'c': None} resolve: ['c'] ts.get_ready() returned nodes: ('c',) resolve_nodes ('c',) (current results = {'c': None}) awaitables: [] End of resolve(), returning {'c': None} LOG: b ts.get_ready() returned nodes: ('a',) resolve_nodes ('a',) (current results = {'c': None, 'b': None}) awaitables: [] inner - _results= {'c': None, 'b': None} LOG: a End of resolve(), returning {'c': None, 'b': None, 'a': None} LOG: go ``` Output for a fail: ``` tests/test_asyncdi.py inner - _results= None resolve: ['a'] ts.get_ready() returned nodes: ('b', 'c') resolve_nodes ('b', 'c') (current results = {}) awaitables: [, ] inner - _results= {} resolve: ['c'] ts.get_ready() returned nodes: ('c',) resolve_nodes ('c',) (current results = {}) awaitables: [] inner - _results= {} LOG: c inner - _results= {'c': None} LOG: c End of resolve(), returning {'c': None} LOG: b ts.get_ready() returned nodes: ('a',) resolve_nodes ('a',) (current results = {'c': None, 'b': None}) awaitables: [] inner - _results= {'c': None, 'b': None} LOG: a End of resolve(), returning {'c': None, 'b': None, 'a': None} LOG: go F =================================================================================================== FAILURES =================================================================================================== _________________________________________________________________________________________________ test_complex _________________________________________________________________________________________________ @pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once > assert result == [""c"", ""b"", ""a"", ""go""] E AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go'] E At index 1 diff: 'c' != 'b' E Left contains one more item: 'go' E Use -v to get the full diff tests/test_asyncdi.py:48: AssertionError ================== short test summary info ================================ FAILED tests/test_asyncdi.py::test_complex - AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go'] ``` I figured out why this is happening. `a` requires `b` and `c` `b` also requires `c` The code decides to run `b` and `c` in parallel. If `c` completes first, then when `b` runs it gets to use the already-calculated result for `c` - so it doesn't need to call `c` again. If `b` gets to that point before `c` does it also needs to call `c`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970655927,https://api.github.com/repos/simonw/datasette/issues/878,970655927,IC_kwDOBm6k_c452wi3,9599,simonw,2021-11-16T20:33:11Z,2021-11-16T20:33:11Z,OWNER,"What should be happening here instead is it should resolve the full graph and notice that `c` is depended on by both `b` and `a` - so it should run `c` first, then run the next ones in parallel. So maybe the algorithm I'm inheriting from https://docs.python.org/3/library/graphlib.html isn't the correct algorithm?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970657874,https://api.github.com/repos/simonw/datasette/issues/878,970657874,IC_kwDOBm6k_c452xBS,9599,simonw,2021-11-16T20:36:01Z,2021-11-16T20:36:01Z,OWNER,"My goal here is to calculate the most efficient way to resolve the different nodes, running them in parallel where possible. So for this class: ```python class Complex(AsyncBase): async def d(self): pass async def c(self): pass async def b(self, c, d): pass async def a(self, b, c): pass async def go(self, a): pass ``` A call to `go()` should do this: - `c` and `d` in parallel - `b` - `a` - `go`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970660299,https://api.github.com/repos/simonw/datasette/issues/878,970660299,IC_kwDOBm6k_c452xnL,9599,simonw,2021-11-16T20:39:43Z,2021-11-16T20:42:27Z,OWNER,"But that does seem to be the plan that `TopographicalSorter` provides: ```python graph = {""go"": {""a""}, ""a"": {""b"", ""c""}, ""b"": {""c"", ""d""}} ts = TopologicalSorter(graph) ts.prepare() while ts.is_active(): nodes = ts.get_ready() print(nodes) ts.done(*nodes) ``` Outputs: ``` ('c', 'd') ('b',) ('a',) ('go',) ``` Also: ```python graph = {""go"": {""d"", ""e"", ""f""}, ""d"": {""b"", ""c""}, ""b"": {""c""}} ts = TopologicalSorter(graph) ts.prepare() while ts.is_active(): nodes = ts.get_ready() print(nodes) ts.done(*nodes) ``` Gives: ``` ('e', 'f', 'c') ('b',) ('d',) ('go',) ``` I'm confident that `TopologicalSorter` is the way to do this. I think I need to rewrite my code to call it once to get that plan, then `await asyncio.gather(*nodes)` in turn to execute it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1513#issuecomment-970738130,https://api.github.com/repos/simonw/datasette/issues/1513,970738130,IC_kwDOBm6k_c453EnS,9599,simonw,2021-11-16T22:32:19Z,2021-11-16T22:32:19Z,OWNER,"I came up with the following query which seems to work! ```sql with cte as ( select rowid, country, country_long, name, owner, primary_fuel from [global-power-plants] ), truncated as ( select null as _facet, null as facet_name, null as facet_count, rowid, country, country_long, name, owner, primary_fuel from cte order by rowid limit 4 ), country_long_facet as ( select 'country_long' as _facet, country_long as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), owner_facet as ( select 'owner' as _facet, owner as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), primary_fuel_facet as ( select 'primary_fuel' as _facet, primary_fuel as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ) select * from truncated union all select * from country_long_facet union all select * from owner_facet union all select * from primary_fuel_facet ``` (Limits should be 101, 31, 31, 31 but I reduced size to get a shorter example table). Results [look like this](https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+country_long_facet%0D%0Aunion+all+select+*+from+owner_facet%0D%0Aunion+all+select+*+from+primary_fuel_facet): _facet | facet_name | facet_count | rowid | country | country_long | name | owner | primary_fuel -- | -- | -- | -- | -- | -- | -- | -- | --   |   |   | 1 | AFG | Afghanistan | Kajaki Hydroelectric Power Plant Afghanistan |   | Hydro   |   |   | 2 | AFG | Afghanistan | Kandahar DOG |   | Solar   |   |   | 3 | AFG | Afghanistan | Kandahar JOL |   | Solar   |   |   | 4 | AFG | Afghanistan | Mahipar Hydroelectric Power Plant Afghanistan |   | Hydro country_long | United States of America | 8688 |   |   |   |   |   |   country_long | China | 4235 |   |   |   |   |   |   country_long | United Kingdom | 2603 |   |   |   |   |   |   owner |   | 14112 |   |   |   |   |   |   owner | Lightsource Renewable Energy | 120 |   |   |   |   |   |   owner | Cypress Creek Renewables | 109 |   |   |   |   |   |   primary_fuel | Solar | 9662 |   |   |   |   |   |   primary_fuel | Hydro | 7155 |   |   |   |   |   |   primary_fuel | Wind | 5188 |   |   |   |   |   |   This is a neat proof of concept. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970742415,https://api.github.com/repos/simonw/datasette/issues/1513,970742415,IC_kwDOBm6k_c453FqP,9599,simonw,2021-11-16T22:37:14Z,2021-11-16T22:37:14Z,OWNER,"The query takes 42.794ms to run. Here's the equivalent page using separate queries: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1 Annoyingly I can't disable facet suggestions but keep facets. I'm going to turn on tracing so I can see how long the separate queries took.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970758179,https://api.github.com/repos/simonw/datasette/issues/1513,970758179,IC_kwDOBm6k_c453Jgj,9599,simonw,2021-11-16T22:47:38Z,2021-11-16T22:47:38Z,OWNER,"Trace now enabled: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1&_trace=1 Here are the relevant traces: ```json [ { ""type"": ""sql"", ""start"": 31.214430154, ""end"": 31.214817089, ""duration_ms"": 0.3869350000016425, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/base.py\"", line 262, in get\n return await self.view_get(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/base.py\"", line 477, in view_get\n response_or_template_contexts = await self.data(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 705, in data\n results = await db.execute(sql, params, truncate=True, **extra_args)\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] order by rowid limit 3"", ""params"": {} }, { ""type"": ""sql"", ""start"": 31.215234586, ""end"": 31.220110342, ""duration_ms"": 4.875756000000564, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 760, in data\n ) = await facet.facet_results()\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 212, in facet_results\n facet_rows_results = await self.ds.execute(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select country_long as value, count(*) as count from (\n select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n )\n where country_long is not null\n group by country_long order by count desc, value limit 4"", ""params"": [] }, { ""type"": ""sql"", ""start"": 31.221062485, ""end"": 31.228968364, ""duration_ms"": 7.905878999999061, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 760, in data\n ) = await facet.facet_results()\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 212, in facet_results\n facet_rows_results = await self.ds.execute(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select owner as value, count(*) as count from (\n select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n )\n where owner is not null\n group by owner order by count desc, value limit 4"", ""params"": [] }, { ""type"": ""sql"", ""start"": 31.229809757, ""end"": 31.253902162, ""duration_ms"": 24.09240499999754, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 760, in data\n ) = await facet.facet_results()\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 212, in facet_results\n facet_rows_results = await self.ds.execute(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select primary_fuel as value, count(*) as count from (\n select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n )\n where primary_fuel is not null\n group by primary_fuel order by count desc, value limit 4"", ""params"": [] }, { ""type"": ""sql"", ""start"": 31.255699745, ""end"": 31.256243889, ""duration_ms"": 0.544143999999136, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 145, in suggest\n row_count = await self.get_row_count()\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 132, in get_row_count\n await self.ds.execute(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select count(*) from (select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] )"", ""params"": [] } ] ``` ``` fetch rows: 0.3869350000016425 ms facet country_long: 4.875756000000564 ms facet owner: 7.905878999999061 ms facet primary_fuel: 24.09240499999754 ms count: 0.544143999999136 ms ``` Total = 37.8ms I modified the query to include the total count as well: https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+country_long_facet%0D%0Aunion+all+select+*+from+owner_facet%0D%0Aunion+all+select+*+from+primary_fuel_facet%0D%0Aunion+all+select+*+from+total_count&_trace=1 ```sql with cte as ( select rowid, country, country_long, name, owner, primary_fuel from [global-power-plants] ), truncated as ( select null as _facet, null as facet_name, null as facet_count, rowid, country, country_long, name, owner, primary_fuel from cte order by rowid limit 4 ), country_long_facet as ( select 'country_long' as _facet, country_long as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), owner_facet as ( select 'owner' as _facet, owner as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), primary_fuel_facet as ( select 'primary_fuel' as _facet, primary_fuel as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), total_count as ( select 'COUNT' as _facet, '' as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte ) select * from truncated union all select * from country_long_facet union all select * from owner_facet union all select * from primary_fuel_facet union all select * from total_count ``` The trace says that query took 34.801436999998714 ms. To my huge surprise, this convoluted optimization only shaves the sum query time down from 37.8ms to 34.8ms! That entire database file is just 11.1 MB though. Maybe it would make a meaningful difference on something larger?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970767952,https://api.github.com/repos/simonw/datasette/issues/1513,970767952,IC_kwDOBm6k_c453L5Q,9599,simonw,2021-11-16T22:53:52Z,2021-11-16T22:53:52Z,OWNER,It's going to take another 15 minutes for the build to finish and deploy the version with `_trace=1`: https://github.com/simonw/covid-19-datasette/actions/runs/1469150112,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970766486,https://api.github.com/repos/simonw/datasette/issues/1513,970766486,IC_kwDOBm6k_c453LiW,9599,simonw,2021-11-16T22:52:56Z,2021-11-16T22:56:07Z,OWNER,"https://covid-19.datasettes.com/covid is 805.2MB https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2 Equivalent SQL: https://covid-19.datasettes.com/covid?sql=with+cte+as+%28%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A%29%2C%0D%0Astate_facet+as+%28%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Afips_facet+as+%28%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Acounty_facet+as+%28%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet%0D%0Aunion+all+select+*+from+total_count ```sql with cte as ( select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties ), truncated as ( select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths from cte order by date desc limit 4 ), state_facet as ( select 'state' as _facet, state as facet_name, count(*) as facet_count, null, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), fips_facet as ( select 'fips' as _facet, fips as facet_name, count(*) as facet_count, null, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), county_facet as ( select 'county' as _facet, county as facet_name, count(*) as facet_count, null, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), total_count as ( select 'COUNT' as _facet, '' as facet_name, count(*) as facet_count, null, null, null, null, null, null, null from cte ) select * from truncated union all select * from state_facet union all select * from fips_facet union all select * from county_facet union all select * from total_count ``` _facet | facet_name | facet_count | rowid | date | county | state | fips | cases | deaths -- | -- | -- | -- | -- | -- | -- | -- | -- | --   |   |   | 1917344 | 2021-11-15 | Autauga | Alabama | 1001 | 10407 | 154   |   |   | 1917345 | 2021-11-15 | Baldwin | Alabama | 1003 | 37875 | 581   |   |   | 1917346 | 2021-11-15 | Barbour | Alabama | 1005 | 3648 | 79   |   |   | 1917347 | 2021-11-15 | Bibb | Alabama | 1007 | 4317 | 92 state | Texas | 148028 |   |   |   |   |   |   |   state | Georgia | 96249 |   |   |   |   |   |   |   state | Virginia | 79315 |   |   |   |   |   |   |   fips |   | 17580 |   |   |   |   |   |   |   fips | 53061 | 665 |   |   |   |   |   |   |   fips | 17031 | 662 |   |   |   |   |   |   |   county | Washington | 18666 |   |   |   |   |   |   |   county | Unknown | 15840 |   |   |   |   |   |   |   county | Jefferson | 15637 |   |   |   |   |   |   |   COUNT |   | 1920593 |   |   |   |   |   |   |  ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970770304,https://api.github.com/repos/simonw/datasette/issues/1513,970770304,IC_kwDOBm6k_c453MeA,9599,simonw,2021-11-16T22:55:19Z,2021-11-16T22:55:19Z,OWNER,(One thing I really like about this pattern is that it should work exactly the same when used to facet the results of arbitrary SQL queries as it does when faceting results from the table page.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970780866,https://api.github.com/repos/simonw/datasette/issues/1513,970780866,IC_kwDOBm6k_c453PDC,9599,simonw,2021-11-16T23:01:57Z,2021-11-16T23:01:57Z,OWNER,"One disadvantage to this approach: if you have a SQL time limit of 1s and it takes 0.9s to return the rows but then 0.5s to calculate each of the requested facets the entire query will exceed the time limit. Could work around this by catching that error and then re-running the query just for the rows, but that would result in the user having to wait longer for the results. Could try to remember if that has happened using an in-memory Python data structure and skip the faceting optimization if it's caused problems in the past? That seems a bit gross. Maybe this becomes an opt-in optimization you can request in your `metadata.json` setting for that table, which massively increases the time limit? That's a bit weird too - now there are two separate implementations of the faceting logic, which had better have a REALLY big pay-off to be worth maintaining. What if we kept the query that returns the rows to be displayed on the page separate from the facets, but then executed all of the facets together using this method such that the `cte` only (presumably) has to be calculated once? That would still lead to multiple facets potentially exceeding the SQL time limit when single facets would not have. Maybe a better optimization would be to move facets to happening via `fetch()` calls from the client, so the user gets to see their rows instantly and the facets then appear as and when they are available (though it would cause page jank). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970827674,https://api.github.com/repos/simonw/datasette/issues/1513,970827674,IC_kwDOBm6k_c453aea,9599,simonw,2021-11-16T23:26:58Z,2021-11-16T23:26:58Z,OWNER,"With trace. https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&_trace=1 shows the following: ``` fetch rows: 0.41762600005768036 ms facet state: 284.30423800000426 ms facet county: 273.2565999999679 ms facet fips: 197.80996999998024 ms ``` = 755.78843400001ms total It didn't run a count because that's the homepage and the count is cached. So I dropped the count from the query and ran it: https://covid-19.datasettes.com/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1 Shows 649.4359889999259 ms for the query - compared to 755.78843400001ms for the separate. So it saved about 100ms. Still not a huge difference though! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970828568,https://api.github.com/repos/simonw/datasette/issues/1513,970828568,IC_kwDOBm6k_c453asY,9599,simonw,2021-11-16T23:27:11Z,2021-11-16T23:27:11Z,OWNER,One last experiment: I'm going to try running an expensive query in the CTE portion.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970845844,https://api.github.com/repos/simonw/datasette/issues/1513,970845844,IC_kwDOBm6k_c453e6U,9599,simonw,2021-11-16T23:35:38Z,2021-11-16T23:35:38Z,OWNER,"I tried adding `cases > 10000` but the SQL query now takes too long - so moving this to my laptop. ``` cd /tmp wget https://covid-19.datasettes.com/covid.db datasette covid.db \ --setting facet_time_limit_ms 10000 \ --setting sql_time_limit_ms 10000 \ --setting trace_debug 1 ``` `http://127.0.0.1:8006/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&cases__gt=10000` shows in the traces: ```json [ { ""type"": ""sql"", ""start"": 12.693033525, ""end"": 12.694056904, ""duration_ms"": 1.0233789999993803, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 262, in get\n return await self.view_get(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 477, in view_get\n response_or_template_contexts = await self.data(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 705, in data\n results = await db.execute(sql, params, truncate=True, **extra_args)\n"" ], ""database"": ""covid"", ""sql"": ""select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 order by rowid limit 3"", ""params"": { ""p0"": 10000 } }, { ""type"": ""sql"", ""start"": 12.694285093, ""end"": 12.814936275, ""duration_ms"": 120.65118200000136, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 262, in get\n return await self.view_get(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 477, in view_get\n response_or_template_contexts = await self.data(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 723, in data\n count_rows = list(await db.execute(count_sql, from_sql_params))\n"" ], ""database"": ""covid"", ""sql"": ""select count(*) from ny_times_us_counties where \""cases\"" > :p0"", ""params"": { ""p0"": 10000 } }, { ""type"": ""sql"", ""start"": 12.818812089, ""end"": 12.851172544, ""duration_ms"": 32.360455000000954, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 856, in data\n suggested_facets.extend(await facet.suggest())\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\"", line 164, in suggest\n distinct_values = await self.ds.execute(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""covid"", ""sql"": ""select county, count(*) as n from (\n select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 \n ) where county is not null\n group by county\n limit 4"", ""params"": { ""p0"": 10000 } }, { ""type"": ""sql"", ""start"": 12.851418868, ""end"": 12.871268359, ""duration_ms"": 19.84949100000044, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 856, in data\n suggested_facets.extend(await facet.suggest())\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\"", line 164, in suggest\n distinct_values = await self.ds.execute(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""covid"", ""sql"": ""select state, count(*) as n from (\n select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 \n ) where state is not null\n group by state\n limit 4"", ""params"": { ""p0"": 10000 } }, { ""type"": ""sql"", ""start"": 12.871497655, ""end"": 12.897715027, ""duration_ms"": 26.217371999999628, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 856, in data\n suggested_facets.extend(await facet.suggest())\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\"", line 164, in suggest\n distinct_values = await self.ds.execute(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""covid"", ""sql"": ""select fips, count(*) as n from (\n select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 \n ) where fips is not null\n group by fips\n limit 4"", ""params"": { ""p0"": 10000 } } ] ``` So that's: ``` fetch rows: 1.0233789999993803 ms count: 120.65118200000136 ms facet county: 32.360455000000954 ms facet state: 19.84949100000044 ms facet fips: 26.217371999999628 ms ``` = 200.1 ms total Compared to: `http://127.0.0.1:8006/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1` Which is 353ms total. The separate queries ran faster! Really surprising result there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970853917,https://api.github.com/repos/simonw/datasette/issues/1513,970853917,IC_kwDOBm6k_c453g4d,9599,simonw,2021-11-16T23:41:01Z,2021-11-16T23:41:01Z,OWNER,"One very interesting difference between the two: on the single giant query page: ```json { ""request_duration_ms"": 376.4317020000476, ""sum_trace_duration_ms"": 370.0828700000329, ""num_traces"": 5 } ``` And on the page that uses separate queries: ```json { ""request_duration_ms"": 819.012272000009, ""sum_trace_duration_ms"": 201.52852100000018, ""num_traces"": 19 } ``` The separate pages page takes 819ms total to render the page, but spends 201ms across 19 SQL queries. The single big query takes 376ms total to render the page, spending 370ms in 5 queries
Those 5 queries, if you're interested ```sql select database_name, schema_version from databases PRAGMA schema_version PRAGMA schema_version explain with cte as (\r\n select rowid, date, county, state, fips, cases, deaths\r\n from ny_times_us_counties\r\n),\r\ntruncated as (\r\n select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet with cte as (\r\n select rowid, date, county, state, fips, cases, deaths\r\n from ny_times_us_counties\r\n),\r\ntruncated as (\r\n select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet ```
All of that additional non-SQL overhead must be stuff relating to Python and template rendering code running on the page. I'm really surprised at how much overhead that is! This is worth researching separately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970855084,https://api.github.com/repos/simonw/datasette/issues/1513,970855084,IC_kwDOBm6k_c453hKs,9599,simonw,2021-11-16T23:41:46Z,2021-11-16T23:41:46Z,OWNER,Conclusion: using a giant convoluted CTE and UNION ALL query to attempt to calculate facets at the same time as retrieving rows is a net LOSS for performance! Very surprised to see that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/pull/1512#issuecomment-970857411,https://api.github.com/repos/simonw/datasette/issues/1512,970857411,IC_kwDOBm6k_c453hvD,9599,simonw,2021-11-16T23:43:21Z,2021-11-16T23:43:21Z,OWNER,"``` E File ""/home/runner/work/datasette/datasette/datasette/utils/vendored_graphlib.py"", line 56 E if (result := self._node2info.get(node)) is None: E ^ E SyntaxError: invalid syntax ``` Oh no - the vendored code I use has `:=` so doesn't work on Python 3.6! Will have to backport it more thoroughly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/pull/1512#issuecomment-970861628,https://api.github.com/repos/simonw/datasette/issues/1512,970861628,IC_kwDOBm6k_c453iw8,9599,simonw,2021-11-16T23:46:07Z,2021-11-16T23:46:07Z,OWNER,"I made the changes locally and tested them with Python 3.6 like so: ``` cd /tmp mkdir v cd v pipenv shell --python=python3.6 cd ~/Dropbox/Development/datasette pip install -e '.[test]' pytest tests/test_asyncdi.py ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/issues/878#issuecomment-971209475,https://api.github.com/repos/simonw/datasette/issues/878,971209475,IC_kwDOBm6k_c4543sD,9599,simonw,2021-11-17T05:41:42Z,2021-11-17T05:41:42Z,OWNER,"I'm going to build a brand new implementation of the `TableView` class that doesn't subclass `BaseView` at all, instead using `asyncinject`. If I'm lucky that will clean up the grungiest part of the codebase. I can maybe even run the tests against old `TableView` and `TableView2` to check that they behave the same.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/pull/1512#issuecomment-971010724,https://api.github.com/repos/simonw/datasette/issues/1512,971010724,IC_kwDOBm6k_c454HKk,9599,simonw,2021-11-17T01:12:22Z,2021-11-17T01:12:22Z,OWNER,I'm going to extract out the `asyncinject` stuff into a separate library.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/pull/1512#issuecomment-971055677,https://api.github.com/repos/simonw/datasette/issues/1512,971055677,IC_kwDOBm6k_c454SI9,9599,simonw,2021-11-17T01:39:25Z,2021-11-17T01:39:25Z,OWNER,https://github.com/simonw/asyncinject version 0.1a0 is now live on PyPI: https://pypi.org/project/asyncinject/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/pull/1512#issuecomment-971056169,https://api.github.com/repos/simonw/datasette/issues/1512,971056169,IC_kwDOBm6k_c454SQp,9599,simonw,2021-11-17T01:39:44Z,2021-11-17T01:39:44Z,OWNER,Closing this PR because I shipped the code in it as a separate library instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/issues/878#issuecomment-971057553,https://api.github.com/repos/simonw/datasette/issues/878,971057553,IC_kwDOBm6k_c454SmR,9599,simonw,2021-11-17T01:40:45Z,2021-11-17T01:40:45Z,OWNER,"I shipped that code as a new library, `asyncinject`: https://pypi.org/project/asyncinject/ - I'll open a new PR to attempt to refactor `TableView` to use it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1506#issuecomment-968192980,https://api.github.com/repos/simonw/datasette/issues/1506,968192980,IC_kwDOBm6k_c45tXPU,9599,simonw,2021-11-14T02:22:40Z,2021-11-14T02:22:40Z,OWNER,"I think the answer is to spot this case and link to `?_item_exact=x` instead of `?_item=x` - it looks like that's already recommended in the documentation here: https://docs.datasette.io/en/stable/json_api.html#column-filter-arguments > **?column__exact=value** or **?_column=value** > Returns rows where the specified column exactly matches the value. So maybe the facet selection rendering logic needs to spot this and link correctly to it?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052826038,Columns beginning with an underscore do not facet correctly, https://github.com/simonw/datasette/issues/1503#issuecomment-968207906,https://api.github.com/repos/simonw/datasette/issues/1503,968207906,IC_kwDOBm6k_c45ta4i,9599,simonw,2021-11-14T05:08:26Z,2021-11-14T05:08:26Z,OWNER,"Error: ``` def test_table_html_filter_form_column_options( path, expected_column_options, app_client ): response = app_client.get(path) assert response.status == 200 form = Soup(response.body, ""html.parser"").find(""form"") column_options = [ o.attrs.get(""value"") or o.string for o in form.select(""select[name=_filter_column] option"") ] > assert expected_column_options == column_options E AssertionError: assert ['- column -'...wid', 'value'] == ['- column -', 'value'] E At index 1 diff: 'rowid' != 'value' E Left contains one more item: 'value' E Use -v to get the full diff ``` This is because `rowid` isn't a table column but IS returned by the query used on that page. My solution: start with the query columns, but then add any table columns that were not already returned by the query to the end of the `filter_columns` list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1050163432,`?_nocol=` removes that column from the filter interface, https://github.com/simonw/datasette/issues/1507#issuecomment-968209560,https://api.github.com/repos/simonw/datasette/issues/1507,968209560,IC_kwDOBm6k_c45tbSY,9599,simonw,2021-11-14T05:26:36Z,2021-11-14T05:26:36Z,OWNER,"It looks like my builds there still run on Python 2! ``` git clone --no-single-branch --depth 50 https://github.com/simonw/datasette . git checkout --force de1e031713f47fbd51eb7239db3e7e6025fbf81a git clean -d -f -f python2.7 -mvirtualenv /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2 /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/bin/python -m pip install --upgrade --no-cache-dir pip setuptools /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/bin/python -m pip install --upgrade --no-cache-dir mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 sphinx<2 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<2.2 cat docs/conf.py ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968209616,https://api.github.com/repos/simonw/datasette/issues/1507,968209616,IC_kwDOBm6k_c45tbTQ,9599,simonw,2021-11-14T05:27:22Z,2021-11-14T05:27:22Z,OWNER,https://blog.readthedocs.com/default-python-3/ they started defaulting new projects to Python 3 back in Feb 2019 but clearly my project was created before then.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968209731,https://api.github.com/repos/simonw/datasette/issues/1507,968209731,IC_kwDOBm6k_c45tbVD,9599,simonw,2021-11-14T05:28:41Z,2021-11-14T05:28:41Z,OWNER,"I will try adding a `.readthedocs.yml` file: https://docs.readthedocs.io/en/stable/config-file/v2.html#python-version This might work: ``` version: 2 build: os: ubuntu-20.04 tools: python: ""3.9"" sphinx: configuration: docs/conf.py ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968209957,https://api.github.com/repos/simonw/datasette/issues/1507,968209957,IC_kwDOBm6k_c45tbYl,9599,simonw,2021-11-14T05:31:07Z,2021-11-14T05:31:07Z,OWNER,"Looks like ReadTheDocs builds started failing for `latest` a few weeks ago: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968210222,https://api.github.com/repos/simonw/datasette/issues/1507,968210222,IC_kwDOBm6k_c45tbcu,9599,simonw,2021-11-14T05:34:14Z,2021-11-14T05:34:14Z,OWNER,"Here's the new build using Python 3: https://readthedocs.org/projects/datasette/builds/15268482/ It's still broken. Here's one of many issue threads about it, this one has a workaround fix: https://github.com/readthedocs/readthedocs.org/issues/8616#issuecomment-952034858 > For future readers, the solution for this problem is to pin `docutils<0.18` in your `requirements.txt` file, and have a `.readthedocs.yaml` file with these contents: > > ``` > version: 2 > > python: > install: > - requirements: docs/requirements.txt > ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968210842,https://api.github.com/repos/simonw/datasette/issues/1507,968210842,IC_kwDOBm6k_c45tbma,9599,simonw,2021-11-14T05:41:55Z,2021-11-14T05:41:55Z,OWNER,"Here's the build with that fix: https://readthedocs.org/projects/datasette/builds/15268498/ It passed and published the docs: https://docs.datasette.io/en/latest/changelog.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/448#issuecomment-969621662,https://api.github.com/repos/simonw/datasette/issues/448,969621662,IC_kwDOBm6k_c45y0Ce,9599,simonw,2021-11-16T01:32:04Z,2021-11-16T01:32:04Z,OWNER,"Tests are failing and I think it's because the facets come back in different orders, need a tie-breaker. https://github.com/simonw/datasette/runs/4219325197?check_suite_focus=true","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969436930,https://api.github.com/repos/simonw/datasette/issues/448,969436930,IC_kwDOBm6k_c45yG8C,9599,simonw,2021-11-15T23:31:58Z,2021-11-15T23:31:58Z,OWNER,"I think this SQL recipe may work instead: ```sql select * from ads_with_targets where 'people_who_match:interests:African-American Civil Rights Movement (1954—68)' in ( select value from json_each(target_names) ) and 'interests:Martin Luther King III' in ( select value from json_each(target_names) ) ``` https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select%0D%0A++*%0D%0Afrom%0D%0A++ads_with_targets%0D%0Awhere%0D%0A++%27people_who_match%3Ainterests%3AAfrican-American+Civil+Rights+Movement+%281954%E2%80%9468%29%27+in+%28%0D%0A++++select%0D%0A++++++value%0D%0A++++from%0D%0A++++++json_each%28target_names%29%0D%0A++%29%0D%0A++and+%27interests%3AMartin+Luther+King+III%27+in+%28%0D%0A++++select%0D%0A++++++value%0D%0A++++from%0D%0A++++++json_each%28target_names%29%0D%0A++%29&interests=&African=&Martin=","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/519#issuecomment-969433734,https://api.github.com/repos/simonw/datasette/issues/519,969433734,IC_kwDOBm6k_c45yGKG,9599,simonw,2021-11-15T23:26:11Z,2021-11-15T23:26:11Z,OWNER,"I'm happy with this as the goals for 1.0. I'm going to close this issue and create three tracking tickets for the three key themes: - https://github.com/simonw/datasette/issues/1509 - https://github.com/simonw/datasette/issues/1510 - https://github.com/simonw/datasette/issues/1511","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459590021,Decide what goes into Datasette 1.0, https://github.com/simonw/datasette/issues/448#issuecomment-969440918,https://api.github.com/repos/simonw/datasette/issues/448,969440918,IC_kwDOBm6k_c45yH6W,9599,simonw,2021-11-15T23:40:17Z,2021-11-15T23:40:35Z,OWNER,"Applied that fix to the `arraycontains` filter but I'm still getting bad results for the faceting: Should never get 182 results on a page that faceting against only 172 items. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969442215,https://api.github.com/repos/simonw/datasette/issues/448,969442215,IC_kwDOBm6k_c45yIOn,9599,simonw,2021-11-15T23:42:03Z,2021-11-15T23:42:03Z,OWNER,I think this code is wrong in the `ArrayFacet` class: https://github.com/simonw/datasette/blob/502c02fa6dde6a8bb840af6c4c8cf858aa1db687/datasette/facets.py#L357-L364,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969446972,https://api.github.com/repos/simonw/datasette/issues/448,969446972,IC_kwDOBm6k_c45yJY8,9599,simonw,2021-11-15T23:46:13Z,2021-11-15T23:46:13Z,OWNER,"It looks like the problem here is that some of the tags occur more than once in the documents: So they get counted more than once, hence the 182 count for something that couldn't possibly return more than 172 documents.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969449772,https://api.github.com/repos/simonw/datasette/issues/448,969449772,IC_kwDOBm6k_c45yKEs,9599,simonw,2021-11-15T23:48:37Z,2021-11-15T23:48:37Z,OWNER,"Given this query: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select%0D%0A++j.value+as+value%2C%0D%0A++count%28*%29+as+count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++id%2C%0D%0A++++++file%2C%0D%0A++++++clicks%2C%0D%0A++++++impressions%2C%0D%0A++++++text%2C%0D%0A++++++url%2C%0D%0A++++++spend_amount%2C%0D%0A++++++spend_currency%2C%0D%0A++++++created%2C%0D%0A++++++ended%2C%0D%0A++++++target_names%0D%0A++++from%0D%0A++++++ads_with_targets%0D%0A++++where%0D%0A++++++%3Ap0+in+%28%0D%0A++++++++select%0D%0A++++++++++value%0D%0A++++++++from%0D%0A++++++++++json_each%28%5Bads_with_targets%5D.%5Btarget_names%5D%29%0D%0A++++++%29%0D%0A++%29%0D%0A++join+json_each%28target_names%29+j%0D%0Agroup+by%0D%0A++j.value%0D%0Aorder+by%0D%0A++count+desc%2C%0D%0A++value%0D%0Alimit%0D%0A++31&p0=people_who_match%3Ainterests%3AAfrican-American+culture ```sql select j.value as value, count(*) as count from ( select id, file, clicks, impressions, text, url, spend_amount, spend_currency, created, ended, target_names from ads_with_targets where :p0 in ( select value from json_each([ads_with_targets].[target_names]) ) ) join json_each(target_names) j group by j.value order by count desc, value limit 31 ``` How can I return a count of the number of documents containing each tag, but not the number of total tags that match including duplicates?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969557008,https://api.github.com/repos/simonw/datasette/issues/448,969557008,IC_kwDOBm6k_c45ykQQ,9599,simonw,2021-11-16T00:56:09Z,2021-11-16T00:59:59Z,OWNER,"This looks like it might work: ```sql with inner as ( select * from ads_with_targets where :p0 in ( select value from json_each([ads_with_targets].[target_names]) ) ), deduped_array_items as ( select distinct j.value, inner.* from json_each([inner].[target_names]) j join inner ) select value, count(*) from deduped_array_items group by value order by count(*) desc ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969557972,https://api.github.com/repos/simonw/datasette/issues/448,969557972,IC_kwDOBm6k_c45ykfU,9599,simonw,2021-11-16T00:56:58Z,2021-11-16T00:56:58Z,OWNER,It uses a CTE which were introduced in SQLite 3.8 - and AWS Lambda Python 3.9 still provides 3.7 - but I've checked and I can use `pysqlite3-binary` to work around that there so I'm OK relying on CTEs for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969572281,https://api.github.com/repos/simonw/datasette/issues/448,969572281,IC_kwDOBm6k_c45yn-5,9599,simonw,2021-11-16T01:05:11Z,2021-11-16T01:05:11Z,OWNER,"I tried this and it seems to work correctly: ```python for source_and_config in self.get_configs(): config = source_and_config[""config""] source = source_and_config[""source""] column = config.get(""column"") or config[""simple""] facet_sql = """""" with inner as ({sql}), deduped_array_items as ( select distinct j.value, inner.* from json_each([inner].{col}) j join inner ) select value as value, count(*) as count from deduped_array_items group by value order by count(*) desc limit {limit} """""".format( col=escape_sqlite(column), sql=self.sql, limit=facet_size + 1 ) ``` The queries are _very_ slow though - I had to bump up to 2s time limit even against only a view returning 3,499 rows.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969578466,https://api.github.com/repos/simonw/datasette/issues/448,969578466,IC_kwDOBm6k_c45ypfi,9599,simonw,2021-11-16T01:08:29Z,2021-11-16T01:08:29Z,OWNER,Actually with the cache warmed up it looks like the facet query is taking 150ms which is good enough.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969582098,https://api.github.com/repos/simonw/datasette/issues/448,969582098,IC_kwDOBm6k_c45yqYS,9599,simonw,2021-11-16T01:10:28Z,2021-11-16T01:10:28Z,OWNER,"Also note that this demo data is using a SQL view to create the JSON arrays - the view is defined as such: ```sql CREATE VIEW ads_with_targets as select ads.*, json_group_array(targets.name) as target_names from ads join ad_targets on ad_targets.ad_id = ads.id join targets on ad_targets.target_id = targets.id group by ad_targets.ad_id; ``` So running JSON faceting on top of that view is a pretty big ask!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/1511#issuecomment-969600859,https://api.github.com/repos/simonw/datasette/issues/1511,969600859,IC_kwDOBm6k_c45yu9b,9599,simonw,2021-11-16T01:20:13Z,2021-11-16T01:20:13Z,OWNER,"See: - #830","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054246919,Review plugin hooks for Datasette 1.0, https://github.com/simonw/datasette/issues/1012#issuecomment-969602825,https://api.github.com/repos/simonw/datasette/issues/1012,969602825,IC_kwDOBm6k_c45yvcJ,9599,simonw,2021-11-16T01:21:14Z,2021-11-16T01:21:14Z,OWNER,"I'd been wondering how to get new classifiers into Trove - thanks, I'll give this a go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py, https://github.com/simonw/datasette/issues/1012#issuecomment-969613166,https://api.github.com/repos/simonw/datasette/issues/1012,969613166,IC_kwDOBm6k_c45yx9u,9599,simonw,2021-11-16T01:27:25Z,2021-11-16T01:27:25Z,OWNER,"Requested here: - https://github.com/pypa/trove-classifiers/pull/85","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py, https://github.com/simonw/datasette/issues/1176#issuecomment-969616626,https://api.github.com/repos/simonw/datasette/issues/1176,969616626,IC_kwDOBm6k_c45yyzy,9599,simonw,2021-11-16T01:29:13Z,2021-11-16T01:29:13Z,OWNER,"I'm inclined to create a Sphinx reference documentation page for this, as I did for `sqlite-utils` here: https://sqlite-utils.datasette.io/en/stable/reference.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1527#issuecomment-988154238,https://api.github.com/repos/simonw/datasette/issues/1527,988154238,IC_kwDOBm6k_c465gl-,9599,simonw,2021-12-07T18:05:26Z,2021-12-07T18:05:26Z,OWNER,"Found a new case of this bug: click the ""Apply"" button on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1 ![apply-bug](https://user-images.githubusercontent.com/9599/145082760-6947c769-480f-45c7-9916-b6cc7f5834f8.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1544#issuecomment-988226523,https://api.github.com/repos/simonw/datasette/issues/1544,988226523,IC_kwDOBm6k_c465yPb,9599,simonw,2021-12-07T20:02:00Z,2021-12-07T20:02:00Z,OWNER,Here's the code at fault: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/database.py#L288-L291,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1073712378,Code that detects the label column for a table is case-sensitive, https://github.com/simonw/datasette/issues/1544#issuecomment-988226938,https://api.github.com/repos/simonw/datasette/issues/1544,988226938,IC_kwDOBm6k_c465yV6,9599,simonw,2021-12-07T20:02:44Z,2021-12-07T20:02:44Z,OWNER,I'm feeling rushed today so I'm going to fix this without adding a test!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1073712378,Code that detects the label column for a table is case-sensitive, https://github.com/simonw/datasette/issues/878#issuecomment-973554024,https://api.github.com/repos/simonw/datasette/issues/878,973554024,IC_kwDOBm6k_c46B0Fo,9599,simonw,2021-11-19T00:21:20Z,2021-11-19T00:21:20Z,OWNER,"That's annoying: it looks like plugins can't use `register_routes()` to over-ride default routes within Datasette itself. This didn't work: ```python from datasette.utils.asgi import Response from datasette import hookimpl import html async def table(request): return Response.html(""Hello from {}"".format( html.escape(repr(request.url_vars)) )) @hookimpl def register_routes(): return [ (r""/(?P[^/]+)/(?P[^/]+?$)"", table), ] ``` I'll use a `/t/` prefix for the moment, but this is probably something I'll fix in Datasette itself later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973564260,https://api.github.com/repos/simonw/datasette/issues/878,973564260,IC_kwDOBm6k_c46B2lk,9599,simonw,2021-11-19T00:27:06Z,2021-11-19T00:27:06Z,OWNER,"Problem: the fancy `asyncinject` stuff inteferes with the fancy Datasette thing that introspects view functions to look for what parameters they take: ```python class Table(asyncinject.AsyncInjectAll): async def view(self, request): return Response.html(""Hello from {}"".format( html.escape(repr(request.url_vars)) )) @hookimpl def register_routes(): return [ (r""/t/(?P[^/]+)/(?P[^/]+?$)"", Table().view), ] ``` This failed with error: ""Table.view() takes 1 positional argument but 2 were given"" So I'm going to use `AsyncInject` and have the `view` function NOT use the `@inject` decorator.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973568285,https://api.github.com/repos/simonw/datasette/issues/878,973568285,IC_kwDOBm6k_c46B3kd,9599,simonw,2021-11-19T00:29:20Z,2021-11-19T00:29:20Z,OWNER,"This is working! ```python from datasette.utils.asgi import Response from datasette import hookimpl import html from asyncinject import AsyncInject, inject class Table(AsyncInject): @inject async def database(self, request): return request.url_vars[""db_name""] @inject async def main(self, request, database): return Response.html(""Database: {}"".format( html.escape(database) )) async def view(self, request): return await self.main(request=request) @hookimpl def register_routes(): return [ (r""/t/(?P[^/]+)/(?P[^/]+?$)"", Table().view), ] ``` This project will definitely show me if I actually like the `asyncinject` patterns or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973527870,https://api.github.com/repos/simonw/datasette/issues/878,973527870,IC_kwDOBm6k_c46Bts-,9599,simonw,2021-11-19T00:13:43Z,2021-11-19T00:13:43Z,OWNER,"New plan: I'm going to build a brand new implementation of `TableView` starting out as a plugin, using the `register_routes()` plugin hook. It will reuse the existing HTML template but will be a completely new Python implementation, based on `asyncinject`. I'm going to start by just getting the table to show up on the page - then I'll add faceting, suggested facets, filters and so-on. Bonus: I'm going to see if I can get it to work for arbitrary SQL queries too (stretch goal).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973542284,https://api.github.com/repos/simonw/datasette/issues/878,973542284,IC_kwDOBm6k_c46BxOM,9599,simonw,2021-11-19T00:16:44Z,2021-11-19T00:16:44Z,OWNER,"``` Development % cookiecutter gh:simonw/datasette-plugin You've downloaded /Users/simon/.cookiecutters/datasette-plugin before. Is it okay to delete and re-download it? [yes]: yes plugin_name []: table-new description []: New implementation of TableView, see https://github.com/simonw/datasette/issues/878 hyphenated [table-new]: underscored [table_new]: github_username []: simonw author_name []: Simon Willison include_static_directory []: include_templates_directory []: ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973635157,https://api.github.com/repos/simonw/datasette/issues/878,973635157,IC_kwDOBm6k_c46CH5V,9599,simonw,2021-11-19T01:07:08Z,2021-11-19T01:07:08Z,OWNER,"This exercise is proving so useful in getting my head around how the enormous and complex `TableView` class works again. Here's where I've got to now - I'm systematically working through the variables that are returned for HTML and for JSON copying across code to get it to work: ```python from datasette.database import QueryInterrupted from datasette.utils import escape_sqlite from datasette.utils.asgi import Response, NotFound, Forbidden from datasette.views.base import DatasetteError from datasette import hookimpl from asyncinject import AsyncInject, inject from pprint import pformat class Table(AsyncInject): @inject async def database(self, request, datasette): # TODO: all that nasty hash resolving stuff can go here db_name = request.url_vars[""db_name""] try: db = datasette.databases[db_name] except KeyError: raise NotFound(f""Database '{db_name}' does not exist"") return db @inject async def table_and_format(self, request, database, datasette): table_and_format = request.url_vars[""table_and_format""] # TODO: be a lot smarter here if ""."" in table_and_format: return table_and_format.split(""."", 2) else: return table_and_format, ""html"" @inject async def main(self, request, database, table_and_format, datasette): # TODO: if this is actually a canned query, dispatch to it table, format = table_and_format is_view = bool(await database.get_view_definition(table)) table_exists = bool(await database.table_exists(table)) if not is_view and not table_exists: raise NotFound(f""Table not found: {table}"") await check_permissions( datasette, request, [ (""view-table"", (database.name, table)), (""view-database"", database.name), ""view-instance"", ], ) private = not await datasette.permission_allowed( None, ""view-table"", (database.name, table), default=True ) pks = await database.primary_keys(table) table_columns = await database.table_columns(table) specified_columns = await columns_to_select(datasette, database, table, request) select_specified_columns = "", "".join( escape_sqlite(t) for t in specified_columns ) select_all_columns = "", "".join(escape_sqlite(t) for t in table_columns) use_rowid = not pks and not is_view if use_rowid: select_specified_columns = f""rowid, {select_specified_columns}"" select_all_columns = f""rowid, {select_all_columns}"" order_by = ""rowid"" order_by_pks = ""rowid"" else: order_by_pks = "", "".join([escape_sqlite(pk) for pk in pks]) order_by = order_by_pks if is_view: order_by = """" nocount = request.args.get(""_nocount"") nofacet = request.args.get(""_nofacet"") if request.args.get(""_shape"") in (""array"", ""object""): nocount = True nofacet = True # Next, a TON of SQL to build where_params and filters and suchlike # skipping that and jumping straight to... where_clauses = [] where_clause = """" if where_clauses: where_clause = f""where {' and '.join(where_clauses)} "" from_sql = ""from {table_name} {where}"".format( table_name=escape_sqlite(table), where=(""where {} "".format("" and "".join(where_clauses))) if where_clauses else """", ) from_sql_params ={} params = {} count_sql = f""select count(*) {from_sql}"" sql_no_order_no_limit = ( ""select {select_all_columns} from {table_name} {where}"".format( select_all_columns=select_all_columns, table_name=escape_sqlite(table), where=where_clause, ) ) page_size = 100 offset = "" offset 0"" sql = ""select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}"".format( select_specified_columns=select_specified_columns, table_name=escape_sqlite(table), where=where_clause, order_by=order_by, page_size=page_size + 1, offset=offset, ) # Fetch rows results = await database.execute(sql, params, truncate=True) columns = [r[0] for r in results.description] rows = list(results.rows) # Fetch count filtered_table_rows_count = None if count_sql: try: count_rows = list(await database.execute(count_sql, from_sql_params)) filtered_table_rows_count = count_rows[0][0] except QueryInterrupted: pass vars = { ""json"": { # THIS STUFF is from the regular JSON ""database"": database.name, ""table"": table, ""is_view"": is_view, # ""human_description_en"": human_description_en, ""rows"": rows[:page_size], ""truncated"": results.truncated, ""filtered_table_rows_count"": filtered_table_rows_count, # ""expanded_columns"": expanded_columns, # ""expandable_columns"": expandable_columns, ""columns"": columns, ""primary_keys"": pks, # ""units"": units, ""query"": {""sql"": sql, ""params"": params}, # ""facet_results"": facet_results, # ""suggested_facets"": suggested_facets, # ""next"": next_value and str(next_value) or None, # ""next_url"": next_url, ""private"": private, ""allow_execute_sql"": await datasette.permission_allowed( request.actor, ""execute-sql"", database, default=True ), }, ""html"": { # ... this is the HTML special stuff # ""table_actions"": table_actions, # ""supports_search"": bool(fts_table), # ""search"": search or """", ""use_rowid"": use_rowid, # ""filters"": filters, # ""display_columns"": display_columns, # ""filter_columns"": filter_columns, # ""display_rows"": display_rows, # ""facets_timed_out"": facets_timed_out, # ""sorted_facet_results"": sorted( # facet_results.values(), # key=lambda f: (len(f[""results""]), f[""name""]), # reverse=True, # ), # ""show_facet_counts"": special_args.get(""_facet_size"") == ""max"", # ""extra_wheres_for_ui"": extra_wheres_for_ui, # ""form_hidden_args"": form_hidden_args, # ""is_sortable"": any(c[""sortable""] for c in display_columns), # ""path_with_replaced_args"": path_with_replaced_args, # ""path_with_removed_args"": path_with_removed_args, # ""append_querystring"": append_querystring, ""request"": request, # ""sort"": sort, # ""sort_desc"": sort_desc, ""disable_sort"": is_view, # ""custom_table_templates"": [ # f""_table-{to_css_class(database)}-{to_css_class(table)}.html"", # f""_table-table-{to_css_class(database)}-{to_css_class(table)}.html"", # ""_table.html"", # ], # ""metadata"": metadata, # ""view_definition"": await db.get_view_definition(table), # ""table_definition"": await db.get_table_definition(table), }, } # I'm just trying to get HTML to work for the moment if format == ""json"": return Response.json(dict(vars, locals=locals()), default=repr) else: return Response.html(repr(vars[""html""])) async def view(self, request, datasette): return await self.main(request=request, datasette=datasette) @hookimpl def register_routes(): return [ (r""/t/(?P[^/]+)/(?P[^/]+?$)"", Table().view), ] async def check_permissions(datasette, request, permissions): """"""permissions is a list of (action, resource) tuples or 'action' strings"""""" for permission in permissions: if isinstance(permission, str): action = permission resource = None elif isinstance(permission, (tuple, list)) and len(permission) == 2: action, resource = permission else: assert ( False ), ""permission should be string or tuple of two items: {}"".format( repr(permission) ) ok = await datasette.permission_allowed( request.actor, action, resource=resource, default=None, ) if ok is not None: if ok: return else: raise Forbidden(action) async def columns_to_select(datasette, database, table, request): table_columns = await database.table_columns(table) pks = await database.primary_keys(table) columns = list(table_columns) if ""_col"" in request.args: columns = list(pks) _cols = request.args.getlist(""_col"") bad_columns = [column for column in _cols if column not in table_columns] if bad_columns: raise DatasetteError( ""_col={} - invalid columns"".format("", "".join(bad_columns)), status=400, ) # De-duplicate maintaining order: columns.extend(dict.fromkeys(_cols)) if ""_nocol"" in request.args: # Return all columns EXCEPT these bad_columns = [ column for column in request.args.getlist(""_nocol"") if (column not in table_columns) or (column in pks) ] if bad_columns: raise DatasetteError( ""_nocol={} - invalid columns"".format("", "".join(bad_columns)), status=400, ) tmp_columns = [ column for column in columns if column not in request.args.getlist(""_nocol"") ] columns = tmp_columns return columns ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973678931,https://api.github.com/repos/simonw/datasette/issues/878,973678931,IC_kwDOBm6k_c46CSlT,9599,simonw,2021-11-19T02:51:17Z,2021-11-19T02:51:17Z,OWNER,"OK, I managed to get a table to render! Here's the code I used - I had to copy a LOT of stuff. https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 I'm going to move this work into a new, separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1518#issuecomment-973681970,https://api.github.com/repos/simonw/datasette/issues/1518,973681970,IC_kwDOBm6k_c46CTUy,9599,simonw,2021-11-19T02:56:31Z,2021-11-19T02:56:53Z,OWNER,"Here's where I got to with my hacked-together initial plugin prototype - it managed to render the table page with some rows on it (and a bunch of missing functionality such as filters): https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-973682389,https://api.github.com/repos/simonw/datasette/issues/1518,973682389,IC_kwDOBm6k_c46CTbV,9599,simonw,2021-11-19T02:57:39Z,2021-11-19T02:57:39Z,OWNER,"Ideally I'd like to execute the existing test suite against the new implementation - that would require me to solve this so I can replace the view with the plugin version though: - #1517 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1517#issuecomment-973686874,https://api.github.com/repos/simonw/datasette/issues/1517,973686874,IC_kwDOBm6k_c46CUha,9599,simonw,2021-11-19T03:06:58Z,2021-11-19T03:06:58Z,OWNER,"I made a mistake: I just wrote a test that proves that plugins CAN over-ride default routes, plus if you look at the code here the plugins get to register themselves first: https://github.com/simonw/datasette/blob/0156c6b5e52d541e93f0d68e9245f20ae83bc933/datasette/app.py#L965-L981","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1057996111,Let `register_routes()` over-ride default routes within Datasette, https://github.com/simonw/datasette/issues/1518#issuecomment-973687978,https://api.github.com/repos/simonw/datasette/issues/1518,973687978,IC_kwDOBm6k_c46CUyq,9599,simonw,2021-11-19T03:07:47Z,2021-11-19T03:07:47Z,OWNER,"I was wrong about that, you CAN over-ride default routes already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1517#issuecomment-973696604,https://api.github.com/repos/simonw/datasette/issues/1517,973696604,IC_kwDOBm6k_c46CW5c,9599,simonw,2021-11-19T03:20:00Z,2021-11-19T03:20:00Z,OWNER,Confirmed - my test plugin is indeed correctly over-riding the table page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1057996111,Let `register_routes()` over-ride default routes within Datasette, https://github.com/simonw/datasette/issues/1518#issuecomment-973700549,https://api.github.com/repos/simonw/datasette/issues/1518,973700549,IC_kwDOBm6k_c46CX3F,9599,simonw,2021-11-19T03:31:20Z,2021-11-19T03:31:26Z,OWNER,"... and while I'm doing all of this I can rewrite the templates to not use those cheating magical functions AND document the template context at the same time, refs: - #1510.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-973698917,https://api.github.com/repos/simonw/datasette/issues/1518,973698917,IC_kwDOBm6k_c46CXdl,9599,simonw,2021-11-19T03:26:18Z,2021-11-19T03:29:03Z,OWNER,"A (likely incomplete) list of features on the table page: - [ ] Display table/database/instance metadata - [ ] Show count of all results - [ ] Display table of results - [ ] Special table display treatment for URLs, numbers - [ ] Allow plugins to modify table cells - [ ] Respect `?_col=` and `?_nocol=` - [ ] Show interface for filtering by columns and operations - [ ] Show search box, support executing FTS searches - [ ] Sort table by specified column - [ ] Paginate table - [ ] Show facet results - [ ] Show suggested facets - [ ] Link to available exports - [ ] Display schema for table - [ ] Maybe it should show the SQL for the query too? - [ ] Handle various non-obvious querystring options, like `?_where=` and `?_through=`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-973699424,https://api.github.com/repos/simonw/datasette/issues/1518,973699424,IC_kwDOBm6k_c46CXlg,9599,simonw,2021-11-19T03:27:49Z,2021-11-19T03:27:49Z,OWNER,"My goal is to break up a lot of this functionality into separate methods. These methods can be executed in parallel by `asyncinject`, but more importantly they can be used to build a much better JSON representation, where the default representation is lighter and `?_extra=x` options can be used to execute more expensive portions and add them to the response. So the HTML version itself needs to be re-written to use those JSON extras.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-973700322,https://api.github.com/repos/simonw/datasette/issues/1518,973700322,IC_kwDOBm6k_c46CXzi,9599,simonw,2021-11-19T03:30:30Z,2021-11-19T03:30:30Z,OWNER,"Right now the HTML version gets to cheat - it passes through objects that are not JSON serializable, including custom functions that can then be called by Jinja. I'm interested in maybe removing this cheating - if the HTML version could only request JSON-serializable extras those could be exposed in the API as well. It would also help cleanup the kind-of-nasty pattern I use in the current `BaseView` where everything returns both a bunch of JSON-serializable data AND an awaitable function that then gets to add extra things to the HTML context.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1521#issuecomment-974371116,https://api.github.com/repos/simonw/datasette/issues/1521,974371116,IC_kwDOBm6k_c46E7ks,9599,simonw,2021-11-19T19:45:47Z,2021-11-19T19:45:47Z,OWNER,"https://github.com/krallin/tini says: > *NOTE: If you are using Docker 1.13 or greater, Tini is included in Docker itself. This includes all versions of Docker CE. To enable Tini, just [pass the `--init` flag to `docker run`](https://docs.docker.com/engine/reference/commandline/run/).*","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974380798,https://api.github.com/repos/simonw/datasette/issues/1521,974380798,IC_kwDOBm6k_c46E97-,9599,simonw,2021-11-19T19:54:26Z,2021-11-19T19:54:26Z,OWNER,"Got it working! Here's a `Dockerfile` which runs completely stand-alone (thanks to using the `echo $'` trick to write out the config files it needs) and successfully serves Datasette behind Apache and `mod_proxy`: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini # Append this to the end of the default httpd.conf file RUN echo $'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPass / http://localhost:9000/\n\ ProxyPassReverse / http://localhost:9000/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf WORKDIR /app RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette -p 9000 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""] ``` Run it like this: ``` docker build -t datasette-apache2-proxy . docker run -p 5000:80 --rm datasette-apache2-proxy ``` Then run this to confirm: ``` ~ % curl -i 'http://localhost:5000/-/versions.json' HTTP/1.1 200 OK Date: Fri, 19 Nov 2021 19:54:05 GMT Server: uvicorn content-type: application/json; charset=utf-8 X-Proxied-By: Apache2 Transfer-Encoding: chunked {""python"": {""version"": ""3.10.0"", ""full"": ""3.10.0 (default, Nov 13 2021, 03:23:03) [GCC 10.3.1 20210424]""}, ""datasette"": {""version"": ""0.59.2""}, ""asgi"": ""3.0"", ""uvicorn"": ""0.15.0"", ""sqlite"": {""version"": ""3.35.5"", ""fts_versions"": [""FTS5"", ""FTS4"", ""FTS3""], ""extensions"": {""json1"": null}, ""compile_options"": [""COMPILER=gcc-10.3.1 20210424"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_DBSTAT_VTAB"", ""ENABLE_FTS3"", ""ENABLE_FTS3_PARENTHESIS"", ""ENABLE_FTS4"", ""ENABLE_FTS5"", ""ENABLE_GEOPOLY"", ""ENABLE_JSON1"", ""ENABLE_MATH_FUNCTIONS"", ""ENABLE_RTREE"", ""ENABLE_UNLOCK_NOTIFY"", ""MAX_VARIABLE_NUMBER=250000"", ""SECURE_DELETE"", ""THREADSAFE=1"", ""USE_URI""]}} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974388295,https://api.github.com/repos/simonw/datasette/issues/1521,974388295,IC_kwDOBm6k_c46E_xH,9599,simonw,2021-11-19T20:00:06Z,2021-11-19T20:00:06Z,OWNER,"And this is the version that proxies to a `base_url` of `/foo/bar/`: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini # Append this to the end of the default httpd.conf file RUN echo $'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPass /foo/bar/ http://localhost:9000/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo $'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette fixtures.db --setting base_url ""/foo/bar/"" -p 9000 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1518#issuecomment-974285803,https://api.github.com/repos/simonw/datasette/issues/1518,974285803,IC_kwDOBm6k_c46Emvr,9599,simonw,2021-11-19T17:56:48Z,2021-11-19T18:14:30Z,OWNER,"Very confused by this piece of code here: https://github.com/simonw/datasette/blob/1c13e1af0664a4dfb1e69714c56523279cae09e4/datasette/views/table.py#L37-L63 I added it in https://github.com/simonw/datasette/commit/754836eef043676e84626c4fd3cb993eed0d2976 - in the new world that should probably be replaced by pure JSON. Aha - this comment explains it: https://github.com/simonw/datasette/issues/521#issuecomment-505279560 > I think the trick is to redefine what a ""cell_row"" is. Each row is currently a list of cells: > > https://github.com/simonw/datasette/blob/6341f8cbc7833022012804dea120b838ec1f6558/datasette/views/table.py#L159-L163 > > I can redefine the row (the `cells` variable in the above example) as a thing-that-iterates-cells (hence behaving like a list) but that also supports `__getitem__` access for looking up cell values if you know the name of the column. The goal was to support neater custom templates like this: ```html+jinja {% for row in display_rows %}

{{ row[""First_Name""] }} {{ row[""Last_Name""] }}

... ``` This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-974287570,https://api.github.com/repos/simonw/datasette/issues/1518,974287570,IC_kwDOBm6k_c46EnLS,9599,simonw,2021-11-19T17:59:33Z,2021-11-19T17:59:33Z,OWNER,"I'm going to try leaning into the `asyncinject` mechanism a bit here. One method can execute and return the raw rows. Another can turn that into the default minimal JSON representation. Then a third can take that (or take both) and use it to inflate out the JSON that the HTML template needs, with those extras and with the rendered cells from plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-974300823,https://api.github.com/repos/simonw/datasette/issues/1518,974300823,IC_kwDOBm6k_c46EqaX,9599,simonw,2021-11-19T18:18:32Z,2021-11-19T18:18:32Z,OWNER,"> This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more. I can definitely support this using pure-JSON - I could make two versions of the row available, one that's an array of cell objects and the other that's an object mapping column names to column raw values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1520#issuecomment-974308215,https://api.github.com/repos/simonw/datasette/issues/1520,974308215,IC_kwDOBm6k_c46EsN3,9599,simonw,2021-11-19T18:29:26Z,2021-11-19T18:29:26Z,OWNER,"The solution that jumps to mind first is that it would be neat if routes could return something that meant ""actually my bad, I can't handle this after all - move to the next one in the list"". A related idea: it might be useful for custom views like my one here to say ""no actually call the default view for this, but give me back the response so I can modify it in some way"". Kind of like Django or ASGI middleware.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058803238,Pattern for avoiding accidental URL over-rides, https://github.com/simonw/datasette/issues/1519#issuecomment-974309591,https://api.github.com/repos/simonw/datasette/issues/1519,974309591,IC_kwDOBm6k_c46EsjX,9599,simonw,2021-11-19T18:31:32Z,2021-11-19T18:31:32Z,OWNER,"`base_url` has been a source of so many bugs like this! I often find them quite hard to replicate, likely because I haven't made myself a good Apache `mod_proxy` testing environment yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974310208,https://api.github.com/repos/simonw/datasette/issues/1519,974310208,IC_kwDOBm6k_c46EstA,9599,simonw,2021-11-19T18:32:31Z,2021-11-19T18:32:31Z,OWNER,Having a live demo running on Cloud Run that proxies through Apache and uses `base_url` would be incredibly useful for replicating and debugging this kind of thing. I wonder how hard it is to run Apache and `mod_proxy` in the same Docker container as Datasette?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1521#issuecomment-974321391,https://api.github.com/repos/simonw/datasette/issues/1521,974321391,IC_kwDOBm6k_c46Evbv,9599,simonw,2021-11-19T18:49:15Z,2021-11-19T18:57:18Z,OWNER,"This pattern looks like it can help: https://ahmet.im/blog/cloud-run-multiple-processes-easy-way/ - see example in https://github.com/ahmetb/multi-process-container-lazy-solution I got that demo working locally like this: ```bash cd /tmp git clone https://github.com/ahmetb/multi-process-container-lazy-solution cd multi-process-container-lazy-solution docker build -t multi-process-container-lazy-solution . docker run -p 5000:8080 --rm multi-process-container-lazy-solution ``` I want to use `apache2` rather than `nginx` though. I found a few relevant examples of Apache in Alpine: - https://github.com/Hacking-Lab/alpine-apache2-reverse-proxy/blob/master/Dockerfile - https://www.sentiatechblog.com/running-apache-in-a-docker-container - https://github.com/search?l=Dockerfile&q=alpine+apache2&type=code ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974322178,https://api.github.com/repos/simonw/datasette/issues/1521,974322178,IC_kwDOBm6k_c46EvoC,9599,simonw,2021-11-19T18:50:22Z,2021-11-19T18:50:22Z,OWNER,"I'll get this working on my laptop first, but then I want to get it up and running on Cloud Run - maybe with a GitHub Actions workflow in this repo that re-deploys it on manual execution.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974327812,https://api.github.com/repos/simonw/datasette/issues/1521,974327812,IC_kwDOBm6k_c46ExAE,9599,simonw,2021-11-19T18:58:49Z,2021-11-19T18:59:55Z,OWNER,"From this example: https://github.com/tigelane/dockerfiles/blob/06cff2ac8cdc920ebd64f50965115eaa3d0afb84/Alpine-Apache2/Dockerfile#L25-L31 it looks like running `apk add apache2` installs a config file at `/etc/apache2/httpd.conf` - so one approach is to then modify that file. ``` # APACHE - Alpine ################# RUN apk --update add apache2 php5-apache2 && \ #apk add openrc --no-cache && \ rm -rf /var/cache/apk/* && \ sed -i 's/#ServerName www.example.com:80/ServerName localhost/' /etc/apache2/httpd.conf && \ mkdir -p /run/apache2/ # Upload our files from folder ""dist"". COPY dist /var/www/localhost/htdocs # Manually set up the apache environment variables ENV APACHE_RUN_USER www-data ENV APACHE_RUN_GROUP www-data ENV APACHE_LOG_DIR /var/log/apache2 ENV APACHE_LOCK_DIR /var/lock/apache2 ENV APACHE_PID_FILE /var/run/apache2.pid # Execute apache2 on run ######################## EXPOSE 80 ENTRYPOINT [""httpd""] CMD [""-D"", ""FOREGROUND""] ``` I think I'll create my own separate copy and modify that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974332787,https://api.github.com/repos/simonw/datasette/issues/1521,974332787,IC_kwDOBm6k_c46EyNz,9599,simonw,2021-11-19T19:05:52Z,2021-11-19T19:05:52Z,OWNER,"Made myself this Dockerfile to let me explore a bit: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 CMD [""sh""] ``` Then: ``` % docker run alpine-apache2-sh % docker run -it alpine-apache2-sh / # ls /etc/apache2/httpd.conf /etc/apache2/httpd.conf / # cat /etc/apache2/httpd.conf # # This is the main Apache HTTP server configuration file. It contains the # configuration directives that give the server its instructions. ... ``` Copying that into a GIST like so: ``` docker run -it --entrypoint sh alpine-apache2-sh -c ""cat /etc/apache2/httpd.conf"" | pbcopy ``` Gist here: https://gist.github.com/simonw/5ea0db6049192cb9f761fbd6beb3a84a","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974334278,https://api.github.com/repos/simonw/datasette/issues/1521,974334278,IC_kwDOBm6k_c46EylG,9599,simonw,2021-11-19T19:08:09Z,2021-11-19T19:08:09Z,OWNER,"Stripping comments using this StackOverflow recipe: https://unix.stackexchange.com/a/157619 docker run -it --entrypoint sh alpine-apache2-sh \ -c ""cat /etc/apache2/httpd.conf"" | sed '/^[[:blank:]]*#/d;s/#.*//' Result is here: https://gist.github.com/simonw/0a05090df5fcff8e8b3334621fa17976","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974336020,https://api.github.com/repos/simonw/datasette/issues/1521,974336020,IC_kwDOBm6k_c46EzAU,9599,simonw,2021-11-19T19:10:48Z,2021-11-19T19:10:48Z,OWNER,"There's a promising looking minimal Apache 2 proxy config here: https://stackoverflow.com/questions/26474476/minimal-configuration-for-apache-reverse-proxy-in-docker-container ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974602459,https://api.github.com/repos/simonw/datasette/issues/1522,974602459,IC_kwDOBm6k_c46F0Db,9599,simonw,2021-11-20T06:15:58Z,2021-11-20T06:15:58Z,OWNER,First I'm going to try using Debian Buster as the base image instead of Alpine.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974605128,https://api.github.com/repos/simonw/datasette/issues/1522,974605128,IC_kwDOBm6k_c46F0tI,9599,simonw,2021-11-20T06:47:59Z,2021-11-20T06:47:59Z,OWNER,"I managed to port the whole thing over to Debian - which took a lot of work because their packaged Apache 2 works very differently from the Alpine one. Once again... I got it working fine on my laptop, but the image deployed to Cloud Run throws 503 errors! ```dockerfile FROM python:3.9.7-slim-bullseye RUN apt-get update && \ apt-get install -y apache2 supervisor && \ apt clean && \ rm -rf /var/lib/apt && \ rm -rf /var/lib/dpkg/info/* # Apache environment, copied from # https://github.com/ijklim/laravel-benfords-law-app/blob/e9bf385dcaddb62ea466a7b245ab6e4ef708c313/docker/os/Dockerfile ENV APACHE_DOCUMENT_ROOT=/var/www/html/public ENV APACHE_RUN_USER www-data ENV APACHE_RUN_GROUP www-data ENV APACHE_PID_FILE /var/run/apache2.pid ENV APACHE_RUN_DIR /var/run/apache2 ENV APACHE_LOCK_DIR /var/lock/apache2 ENV APACHE_LOG_DIR /var/log RUN ln -sf /dev/stdout /var/log/apache2-access.log RUN ln -sf /dev/stderr /var/log/apache2-error.log RUN mkdir -p $APACHE_RUN_DIR $APACHE_LOCK_DIR RUN a2enmod proxy RUN a2enmod proxy_http RUN a2enmod headers ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip # Append this to the end of the default httpd.conf file RUN echo '\n\ \n\ Options Indexes FollowSymLinks\n\ AllowOverride None\n\ Require all granted\n\ \n\ \n\ \n\ ServerName localhost\n\ DocumentRoot /app/html\n\ ProxyPreserveHost On\n\ ProxyPass /prefix/ http://127.0.0.1:8001/\n\ Header add X-Proxied-By ""Apache2""\n\ \n\ ' > /etc/apache2/sites-enabled/000-default.conf WORKDIR /app RUN mkdir -p /app/html RUN echo 'Datasette' > /app/html/index.html ADD https://latest.datasette.io/fixtures.db /app/fixtures.db EXPOSE 80 RUN echo ""[supervisord]"" >> /app/supervisord.conf RUN echo ""nodaemon=true"" >> /app/supervisord.conf RUN echo """" >> /app/supervisord.conf RUN echo ""[program:apache2]"" >> /app/supervisord.conf RUN echo ""command=apache2 -D FOREGROUND"" >> /app/supervisord.conf RUN echo """" >> /app/supervisord.conf RUN echo ""[program:datasette]"" >> /app/supervisord.conf RUN echo ""command=datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001"" >> /app/supervisord.conf CMD [""/usr/bin/supervisord"", ""-c"", ""/app/supervisord.conf""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974605529,https://api.github.com/repos/simonw/datasette/issues/1522,974605529,IC_kwDOBm6k_c46F0zZ,9599,simonw,2021-11-20T06:52:21Z,2021-11-20T06:52:21Z,OWNER,"I've now tried both Debian and Alpine, and I've tried both `tini` and `supervisord`. Each time I get the same result - I get 503 errors for the first dozen or so refreshes of `/prefix/` followed by it intermittently working. Absolutely stumped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974389472,https://api.github.com/repos/simonw/datasette/issues/1519,974389472,IC_kwDOBm6k_c46FADg,9599,simonw,2021-11-19T20:01:02Z,2021-11-19T20:01:02Z,OWNER,I now have a `Dockerfile` in https://github.com/simonw/datasette/issues/1521#issuecomment-974388295 that I can use to run a local Apache 2 with `mod_proxy` to investigate this class of bugs!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974391204,https://api.github.com/repos/simonw/datasette/issues/1519,974391204,IC_kwDOBm6k_c46FAek,9599,simonw,2021-11-19T20:02:41Z,2021-11-19T20:02:41Z,OWNER,"Bug confirmed: ![proxy-bug](https://user-images.githubusercontent.com/9599/142684666-112136bf-9243-4b6e-8202-339fcfe91bcc.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974398399,https://api.github.com/repos/simonw/datasette/issues/1519,974398399,IC_kwDOBm6k_c46FCO_,9599,simonw,2021-11-19T20:08:20Z,2021-11-19T20:22:02Z,OWNER,"The relevant test is this one: https://github.com/simonw/datasette/blob/30255055150d7bc0affc8156adc18295495020ff/tests/test_html.py#L1608-L1649 I modified that test to add `""/fixtures/facetable?sql=select+1""` as one of the tested paths, and dropped in an `assert False` to pause it in the debugger: ``` @pytest.mark.parametrize( ""path"", [ ""/"", ""/fixtures"", ""/fixtures/compound_three_primary_keys"", ""/fixtures/compound_three_primary_keys/a,a,a"", ""/fixtures/paginated_view"", ""/fixtures/facetable"", ""/fixtures?sql=select+1"", ], ) def test_base_url_config(app_client_base_url_prefix, path): client = app_client_base_url_prefix response = client.get(""/prefix/"" + path.lstrip(""/"")) soup = Soup(response.body, ""html.parser"") if path == ""/fixtures?sql=select+1"": > assert False E assert False ``` BUT... in the debugger: ``` (Pdb) print(soup) ...

This data as json, testall, testnone, testresponse, CSV

``` Those all have the correct prefix! But that's not what I'm seeing in my `Dockerfile` reproduction of the issue. Something very weird is going on here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974405016,https://api.github.com/repos/simonw/datasette/issues/1519,974405016,IC_kwDOBm6k_c46FD2Y,9599,simonw,2021-11-19T20:14:19Z,2021-11-19T20:15:05Z,OWNER,"I added `template_debug` in the Dockerfile: ``` datasette fixtures.db --setting template_debug 1 --setting base_url ""/foo/bar/"" -p 9000 &\n\ ``` And then hit `http://localhost:5000/foo/bar/fixtures?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1` to view the template context - and it showed the bug, output edited to just show relevant keys: ```json { ""edit_sql_url"": ""/foo/bar/fixtures?sql=select+%2A+from+compound_three_primary_keys+limit+1"", ""settings"": { ""force_https_urls"": false, ""template_debug"": true, ""trace_debug"": false, ""base_url"": ""/foo/bar/"" }, ""show_hide_link"": ""/fixtures?sql=select+%2A+from+compound_three_primary_keys+limit+1&_context=1&_hide_sql=1"", ""show_hide_text"": ""hide"", ""show_hide_hidden"": """", ""renderers"": { ""json"": ""/fixtures.json?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1"" }, ""url_csv"": ""/fixtures.csv?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1&_size=max"", ""url_csv_path"": ""/fixtures.csv"", ""base_url"": ""/foo/bar/"" } ``` This is so strange. `edit_sql_url` and `base_url` are correct, but `show_hide_link` and `url_csv` and `renderers.json` are not. And it's _really strange_ that the bug doesn't show up in the tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974418496,https://api.github.com/repos/simonw/datasette/issues/1519,974418496,IC_kwDOBm6k_c46FHJA,9599,simonw,2021-11-19T20:24:16Z,2021-11-19T20:24:16Z,OWNER,"Here's the code that generates `edit_sql_url` correctly: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/database.py#L416-L420 And here's the code for `show_hide_link`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/database.py#L432-L433 And for `url_csv`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/base.py#L600-L602","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974420619,https://api.github.com/repos/simonw/datasette/issues/1519,974420619,IC_kwDOBm6k_c46FHqL,9599,simonw,2021-11-19T20:25:19Z,2021-11-19T20:25:19Z,OWNER,"The implementations of `path_with_removed_args` and `path_with_format`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L228-L254 https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L710-L729","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974422829,https://api.github.com/repos/simonw/datasette/issues/1519,974422829,IC_kwDOBm6k_c46FIMt,9599,simonw,2021-11-19T20:26:35Z,2021-11-19T20:26:35Z,OWNER,"In the `?_context=` debug view the request looks like this: ``` ""request"": """", ``` I'm going to add a `repr()` to it such that it's a bit more useful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974433206,https://api.github.com/repos/simonw/datasette/issues/1519,974433206,IC_kwDOBm6k_c46FKu2,9599,simonw,2021-11-19T20:31:52Z,2021-11-19T20:31:52Z,OWNER,"Modified my `Dockerfile` to do this: RUN pip install https://github.com/simonw/datasette/archive/ff0dd4da38d48c2fa9250ecf336002c9ed724e36.zip And now the `request` in that debug `?_context=1` looks like this: ``` ""request"": """" ``` That explains the bug - that request doesn't maintain the original path prefix of `http://localhost:5000/foo/bar/fixtures?sql=` (also it's been rewritten to `localhost:9000` instead of `localhost:5000`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974433320,https://api.github.com/repos/simonw/datasette/issues/1519,974433320,IC_kwDOBm6k_c46FKwo,9599,simonw,2021-11-19T20:32:04Z,2021-11-19T20:32:04Z,OWNER,Still not clear why the tests pass but the live example fails.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1521#issuecomment-974433520,https://api.github.com/repos/simonw/datasette/issues/1521,974433520,IC_kwDOBm6k_c46FKzw,9599,simonw,2021-11-19T20:32:29Z,2021-11-19T20:32:29Z,OWNER,This configuration works great.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974435661,https://api.github.com/repos/simonw/datasette/issues/1522,974435661,IC_kwDOBm6k_c46FLVN,9599,simonw,2021-11-19T20:33:42Z,2021-11-19T20:33:42Z,OWNER,"Should just be a case of deploying this `Dockerfile`: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini # Append this to the end of the default httpd.conf file RUN echo $'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPass /foo/bar/ http://localhost:9000/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo $'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette fixtures.db --setting base_url ""/foo/bar/"" -p 9000 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""] ``` I can follow this TIL: https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974450232,https://api.github.com/repos/simonw/datasette/issues/1519,974450232,IC_kwDOBm6k_c46FO44,9599,simonw,2021-11-19T20:41:53Z,2021-11-19T20:42:19Z,OWNER,https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration says I should use `ProxyPreserveHost on`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974447950,https://api.github.com/repos/simonw/datasette/issues/1519,974447950,IC_kwDOBm6k_c46FOVO,9599,simonw,2021-11-19T20:40:19Z,2021-11-19T20:40:19Z,OWNER,"Figured it out! The test is not an accurate recreation of what is happening, because it doesn't simulate a request with a path of `/fixtures` that has been redirected by the proxy to `/prefix/fixtures`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974477465,https://api.github.com/repos/simonw/datasette/issues/1519,974477465,IC_kwDOBm6k_c46FViZ,9599,simonw,2021-11-19T21:15:30Z,2021-11-19T21:15:30Z,OWNER,"I think what's happening here is Apache is actually making a request to `/fixtures` rather than making a request to `/prefix/fixtures` - and Datasette is replying to requests on both the prefixed and the non-prefixed paths. This is pretty confusing! I think Datasette should ONLY reply to `/prefix/fixtures` instead and return a 404 for `/fixtures` - this would make things a whole lot easier to debug. But shipping that change could break existing deployments. Maybe that should be a breaking change for 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974478126,https://api.github.com/repos/simonw/datasette/issues/1519,974478126,IC_kwDOBm6k_c46FVsu,9599,simonw,2021-11-19T21:16:36Z,2021-11-19T21:16:36Z,OWNER,"In the meantime I can catch these errors by changing the test to run each path twice, once with and once without the prefix. This should accurately simulate how Apache is working here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1522#issuecomment-974506401,https://api.github.com/repos/simonw/datasette/issues/1522,974506401,IC_kwDOBm6k_c46Fcmh,9599,simonw,2021-11-19T22:11:51Z,2021-11-19T22:11:51Z,OWNER,"This is frustrating: I have the following Dockerfile: ```dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini # Append this to the end of the default httpd.conf file RUN echo $'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPass /prefix/ http://localhost:8001/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo $'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette fixtures.db --setting base_url ""/prefix/"" -h 0.0.0.0 -p 8001 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""] ``` It works fine when I run it locally: ``` docker build -t datasette-apache-proxy-demo . docker run -p 5000:80 datasette-apache-proxy-demo ``` But when I deploy it to Cloud Run with the following script: ```bash #!/bin/bash # https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run NAME=""datasette-apache-proxy-demo"" PROJECT=$(gcloud config get-value project) IMAGE=""gcr.io/$PROJECT/$NAME"" gcloud builds submit --tag $IMAGE gcloud run deploy \ --allow-unauthenticated \ --platform=managed \ --image $IMAGE $NAME \ --port 80 ``` It serves the `/` page successfully, but hits to `/prefix/` return the following 503 error: > Service Unavailable > > The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. > > Apache/2.4.51 (Unix) Server at datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app Port 80 Cloud Run logs: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974521687,https://api.github.com/repos/simonw/datasette/issues/1522,974521687,IC_kwDOBm6k_c46FgVX,9599,simonw,2021-11-19T22:46:26Z,2021-11-19T22:46:26Z,OWNER,"Oh weird, it started working: https://datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app/prefix/fixtures/sortable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974523297,https://api.github.com/repos/simonw/datasette/issues/1522,974523297,IC_kwDOBm6k_c46Fguh,9599,simonw,2021-11-19T22:50:31Z,2021-11-19T22:50:31Z,OWNER,Demo code is now at: https://github.com/simonw/datasette/tree/main/demos/apache-proxy,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974523569,https://api.github.com/repos/simonw/datasette/issues/1522,974523569,IC_kwDOBm6k_c46Fgyx,9599,simonw,2021-11-19T22:51:10Z,2021-11-19T22:51:10Z,OWNER,I wan a GitHub Action which I can manually activate to deploy a new version of that demo... and I want it to bake in the latest release of Datasette so I can use it to demonstrate bug fixes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974541971,https://api.github.com/repos/simonw/datasette/issues/1522,974541971,IC_kwDOBm6k_c46FlST,9599,simonw,2021-11-19T23:40:32Z,2021-11-19T23:40:32Z,OWNER,"I want to be able to use build arguments to specify which commit version or branch of Datasette to deploy. This is proving hard to work out. I have this in my Dockerfile now: ``` ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip ``` Which works locally: docker build -t datasette-apache-proxy-demo . \ --build-arg DATASETTE_REF=c617e1769ea27e045b0f2907ef49a9a1244e577d But I can't figure out the right incantation to pass to `gcloud build submit`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974542348,https://api.github.com/repos/simonw/datasette/issues/1522,974542348,IC_kwDOBm6k_c46FlYM,9599,simonw,2021-11-19T23:41:47Z,2021-11-19T23:44:07Z,OWNER,Do I have to use `cloudbuild.yml` to specify these? https://stackoverflow.com/a/58327340/6083 and https://stackoverflow.com/a/66232670/6083 suggest I do.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974557766,https://api.github.com/repos/simonw/datasette/issues/1522,974557766,IC_kwDOBm6k_c46FpJG,9599,simonw,2021-11-20T00:35:25Z,2021-11-20T00:35:25Z,OWNER,Wrote a TIL about `--build-arg` and Cloud Run: https://til.simonwillison.net/cloudrun/using-build-args-with-cloud-run,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974558076,https://api.github.com/repos/simonw/datasette/issues/1522,974558076,IC_kwDOBm6k_c46FpN8,9599,simonw,2021-11-20T00:36:56Z,2021-11-20T00:36:56Z,OWNER,That 503 error is _really_ frustrating: I have a deploy running at https://apache-proxy-demo.datasette.io/prefix/ and after a fresh deploy it serves 503 errors for quite a while - then eventually starts working.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974558267,https://api.github.com/repos/simonw/datasette/issues/1519,974558267,IC_kwDOBm6k_c46FpQ7,9599,simonw,2021-11-20T00:37:57Z,2021-11-20T00:37:57Z,OWNER,Thanks to #1522 I have a live demo that exhibits this bug now: https://apache-proxy-demo.datasette.io/prefix/fixtures/attraction_characteristic,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974559176,https://api.github.com/repos/simonw/datasette/issues/1519,974559176,IC_kwDOBm6k_c46FpfI,9599,simonw,2021-11-20T00:42:08Z,2021-11-20T00:42:08Z,OWNER,"> In the meantime I can catch these errors by changing the test to run each path twice, once with and once without the prefix. This should accurately simulate how Apache is working here. This worked, I managed to get the tests to fail! Here's the change I made: ```diff diff --git a/tests/test_html.py b/tests/test_html.py index f24165b..dbdfe59 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -1614,12 +1614,19 @@ def test_metadata_sort_desc(app_client): ""/fixtures/compound_three_primary_keys/a,a,a"", ""/fixtures/paginated_view"", ""/fixtures/facetable"", + ""/fixtures?sql=select+1"", ], ) -def test_base_url_config(app_client_base_url_prefix, path): +@pytest.mark.parametrize(""use_prefix"", (True, False)) +def test_base_url_config(app_client_base_url_prefix, path, use_prefix): client = app_client_base_url_prefix - response = client.get(""/prefix/"" + path.lstrip(""/"")) + path_to_get = path + if use_prefix: + path_to_get = ""/prefix/"" + path.lstrip(""/"") + response = client.get(path_to_get) soup = Soup(response.body, ""html.parser"") + if path == ""/fixtures?sql=select+1"": + assert False for el in soup.findAll([""a"", ""link"", ""script""]): if ""href"" in el.attrs: href = el[""href""] @@ -1642,11 +1649,12 @@ def test_base_url_config(app_client_base_url_prefix, path): # If this has been made absolute it may start http://localhost/ if href.startswith(""http://localhost/""): href = href[len(""http://localost/"") :] - assert href.startswith(""/prefix/""), { + assert href.startswith(""/prefix/""), json.dumps({ ""path"": path, + ""path_to_get"": path_to_get, ""href_or_src"": href, ""element_parent"": str(el.parent), - } + }, indent=4, default=repr) def test_base_url_affects_metadata_extra_css_urls(app_client_base_url_prefix): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1522#issuecomment-974564712,https://api.github.com/repos/simonw/datasette/issues/1522,974564712,IC_kwDOBm6k_c46Fq1o,9599,simonw,2021-11-20T01:07:49Z,2021-11-20T01:10:48Z,OWNER,https://apache-proxy-demo.datasette.io/prefix/fixtures/compound_three_primary_keys has broken suggested facet links - they go to `https://localhost:8001/prefix/fixtures/compound_three_primary_keys?_facet=pk1#facet-pk1` - but I think that's because I'm missing the `ProxyPreserveHost On` setting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974561593,https://api.github.com/repos/simonw/datasette/issues/1519,974561593,IC_kwDOBm6k_c46FqE5,9599,simonw,2021-11-20T00:53:19Z,2021-11-20T00:53:19Z,OWNER,Adding that test found (I hope!) all of the remaining `base_url` bugs. There were a bunch! I think I finally get to close #838 too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974562942,https://api.github.com/repos/simonw/datasette/issues/1519,974562942,IC_kwDOBm6k_c46FqZ-,9599,simonw,2021-11-20T00:59:32Z,2021-11-20T00:59:32Z,OWNER,"Ouch a nasty bug crept through there - https://datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app/prefix/fixtures/compound_three_primary_keys says > 500: name 'ds' is not defined","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1522#issuecomment-974565392,https://api.github.com/repos/simonw/datasette/issues/1522,974565392,IC_kwDOBm6k_c46FrAQ,9599,simonw,2021-11-20T01:11:20Z,2021-11-20T01:11:20Z,OWNER,"Yup, that fixed it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974565816,https://api.github.com/repos/simonw/datasette/issues/1522,974565816,IC_kwDOBm6k_c46FrG4,9599,simonw,2021-11-20T01:13:39Z,2021-11-20T01:13:39Z,OWNER,"I have a hunch that running `httpd -D FOREGROUND` doesn't show error logs, which would explain why I can't use the Cloud Run logs to figure out the reason for the 503s.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974572505,https://api.github.com/repos/simonw/datasette/issues/1522,974572505,IC_kwDOBm6k_c46FsvZ,9599,simonw,2021-11-20T01:50:48Z,2021-11-20T01:51:41Z,OWNER,"I figured out a recipe to run `httpd` as a service inside Alpine - works great on my laptop, here's my new `Dockerfile`: ```dockerfile FROM python:3-alpine # openrc gives us rc-service RUN apk add --no-cache \ openrc \ apache2 \ apache2-proxy \ bash ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip # Append this to the end of the default httpd.conf file RUN echo -e 'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPreserveHost On\n\ ProxyPass /prefix/ http://127.0.0.1:8001/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo 'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db EXPOSE 80 # RUN echo -e ""#!/bin/bash\nopenrc default\nrc-service apache2 start;\ndatasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001"" > /app/start.sh RUN echo ""#!/bin/bash"" >> start.sh RUN echo ""openrc default"" >> start.sh RUN echo ""rc-service apache2 start"" >> start.sh RUN echo ""datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001"" >> /app/start.sh RUN chmod +x /app/start.sh CMD /app/start.sh ``` I'm going to try this on Cloud Run and see if it fixes the 503s One annoying thing about this: Ctrl+C on my laptop no longer stops the container, I have to `docker ps` and then `docker kill xxx` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974577565,https://api.github.com/repos/simonw/datasette/issues/1522,974577565,IC_kwDOBm6k_c46Ft-d,9599,simonw,2021-11-20T02:23:07Z,2021-11-20T02:23:07Z,OWNER,"OK, that works on my laptop - and Ctrl+C quits it, which is nice: ``` apache-proxy % docker run -p 5000:80 --rm datasette-build-arg-demo 2021-11-20 02:22:13,925 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2021-11-20 02:22:13,927 INFO supervisord started with pid 1 2021-11-20 02:22:14,931 INFO spawned: 'datasette' with pid 7 2021-11-20 02:22:14,934 INFO spawned: 'httpd' with pid 8 2021-11-20 02:22:16,484 INFO success: datasette entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-11-20 02:22:16,484 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) ^C 2021-11-20 02:22:26,285 WARN received SIGINT indicating exit request 2021-11-20 02:22:26,286 INFO waiting for datasette, httpd to die 2021-11-20 02:22:26,315 INFO stopped: httpd (exit status 0) 2021-11-20 02:22:26,540 INFO stopped: datasette (exit status 0) ``` Here's my new Dockerfile: ```dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ supervisor \ bash ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip # Append this to the end of the default httpd.conf file RUN echo -e 'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPreserveHost On\n\ ProxyPass /prefix/ http://127.0.0.1:8001/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo 'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db EXPOSE 80 RUN echo ""[supervisord]"" >> /app/supervisord.conf RUN echo ""nodaemon=true"" >> /app/supervisord.conf RUN echo """" >> /app/supervisord.conf RUN echo ""[program:httpd]"" >> /app/supervisord.conf RUN echo ""command=httpd -D FOREGROUND"" >> /app/supervisord.conf RUN echo """" >> /app/supervisord.conf RUN echo ""[program:datasette]"" >> /app/supervisord.conf RUN echo ""command=datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001"" >> /app/supervisord.conf CMD [""/usr/bin/supervisord"", ""-c"", ""/app/supervisord.conf""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974577082,https://api.github.com/repos/simonw/datasette/issues/1522,974577082,IC_kwDOBm6k_c46Ft26,9599,simonw,2021-11-20T02:19:27Z,2021-11-20T02:19:27Z,OWNER,"https://docs.docker.com/config/containers/multi-service_container/ suggests `supervisord` as a last resort. https://stackoverflow.com/a/49100302/6083 has a neat looking recipe for than in Alpine: > **1.** `Dockerfile` is: > > FROM alpine:latest > RUN apk update && apk add --no-cache supervisor openssh nginx > COPY supervisord.conf /etc/supervisord.conf > CMD [""/usr/bin/supervisord"", ""-c"", ""/etc/supervisord.conf""] > > **2.** `supervisord.conf` is: > > [supervisord] > nodaemon=true > > [program:sshd] > command=/usr/sbin/sshd -D > > [program:nginx] > command=nginx -c /etc/nginx/nginx.conf ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974573616,https://api.github.com/repos/simonw/datasette/issues/1522,974573616,IC_kwDOBm6k_c46FtAw,9599,simonw,2021-11-20T01:58:44Z,2021-11-20T01:58:44Z,OWNER,"Deploy to Cloud Run appears to hang here: ``` Deploying container to Cloud Run service [datasette-apache-proxy-demo] in project [datasette-222320] region [us-central1] ⠧ Deploying... Revision deployment finished. Waiting for health check to begin. ⠧ Creating Revision... . Routing traffic... ✓ Setting IAM Policy... ``` **Waiting for health check to begin** makes it sound like the container didn't start properly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974575512,https://api.github.com/repos/simonw/datasette/issues/1522,974575512,IC_kwDOBm6k_c46FteY,9599,simonw,2021-11-20T02:09:20Z,2021-11-20T02:09:20Z,OWNER,"> **Waiting for health check to begin** makes it sound like the container didn't start properly. That eventually failed, but I did get these in the build logs: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974576436,https://api.github.com/repos/simonw/datasette/issues/1522,974576436,IC_kwDOBm6k_c46Fts0,9599,simonw,2021-11-20T02:14:45Z,2021-11-20T02:14:45Z,OWNER,I'm going to try running Apache with `httpd -D FOREGROUND` while running `datasette &`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974576624,https://api.github.com/repos/simonw/datasette/issues/1522,974576624,IC_kwDOBm6k_c46Ftvw,9599,simonw,2021-11-20T02:16:12Z,2021-11-20T02:16:12Z,OWNER,"Again, that approach worked on my laptop but when deployed to Cloud Run mostly gave me 503 errors for the `/prefix/` page, with the occasional 200. I did this: ```Dockerfile RUN echo ""#!/bin/bash"" >> start.sh # Start Datasette running in background with & RUN echo ""datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001 &"" >> /app/start.sh RUN echo ""httpd -D FOREGROUND"" >> /app/start.sh RUN chmod +x /app/start.sh CMD /app/start.sh ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974577949,https://api.github.com/repos/simonw/datasette/issues/1522,974577949,IC_kwDOBm6k_c46FuEd,9599,simonw,2021-11-20T02:26:09Z,2021-11-20T02:26:17Z,OWNER,"So frustrating, that's giving me the same problem after being deployed! 503 errors for the first while, then it starts working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974578141,https://api.github.com/repos/simonw/datasette/issues/1522,974578141,IC_kwDOBm6k_c46FuHd,9599,simonw,2021-11-20T02:27:23Z,2021-11-20T02:27:23Z,OWNER,"Aha! This could be the clue I was looking for: https://www.reddit.com/r/googlecloud/comments/fmkx63/comment/fl5csty/?utm_source=reddit&utm_medium=web2x&context=3 > Are you processing on a background thread in your container? If so, it's likely your problem, because cloud run will put your app into a low power state between http requests. For long running tasks in cloud run, you need to keep the http connection open, and not return until you are done. Maybe the `datasette &` process is being affected by that in some way?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974585374,https://api.github.com/repos/simonw/datasette/issues/1522,974585374,IC_kwDOBm6k_c46Fv4e,9599,simonw,2021-11-20T03:28:58Z,2021-11-20T03:28:58Z,OWNER,Based on https://medium.com/google-cloud/init-process-for-containers-d03a471fa0cc I might try s6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974682507,https://api.github.com/repos/simonw/datasette/issues/1522,974682507,IC_kwDOBm6k_c46GHmL,9599,simonw,2021-11-20T17:24:13Z,2021-11-20T17:24:13Z,OWNER,"I'm going to leave this issue open, tag it as ""help wanted"" and cross my fingers that someone with Cloud Run deep expertise takes an interest in figuring out what's going wrong here!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974683220,https://api.github.com/repos/simonw/datasette/issues/1522,974683220,IC_kwDOBm6k_c46GHxU,9599,simonw,2021-11-20T17:29:12Z,2021-11-20T17:29:12Z,OWNER,"> As a a sanity check, would it be worth looking at trying to push the multi-process container on another provider of a knative / cloud run / tekton ? I have a somewhat similar use case for a future proejct, so i'm been very grateful to you sharing all the progress in this issue. That's a great idea. I'll try running on a non-Knative host too (probably Fly - though they actually run containers using Firecracker which ends up being completely different). Cloud Run are the only Knative host I've used, know of any others aside from Scaleway? They look like they're worth getting familiar with.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974685095,https://api.github.com/repos/simonw/datasette/issues/1522,974685095,IC_kwDOBm6k_c46GIOn,9599,simonw,2021-11-20T17:42:25Z,2021-11-20T17:42:25Z,OWNER,"I tried to deploy it to Fly - initially using `flyctl launch` but then switching to `flyctl deploy` so I could use the `--build-arg` option (posted [a feature request](https://community.fly.io/t/feature-request-flyctl-launch-build-arg/3209) here). Almost got it working, but it failed the health check: ``` % cd datasette/demos/apache-proxy apache-proxy % flyctl launch Creating app in /Users/simon/Dropbox/Development/datasette/demos/apache-proxy Scanning source code Detected Dockerfile app Automatically selected personal organization: Simon Willison ? Select region: sjc (Sunnyvale, California (US)) Created app floral-dust-4577 in organization personal Wrote config file fly.toml Your app is ready. Deploy with `flyctl deploy` ? Would you like to deploy now? Yes Deploying floral-dust-4577 ==> Validating app configuration --> Validating app configuration done Services TCP 80/443 ⇢ 8080 ==> Creating build context --> Creating build context done ==> Building image with Docker Sending build context to Docker daemon 8.704kB ... Error error building: executor failed running [/bin/sh -c pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip]: exit code: 1 # I didn't pass the build argument, trying again with flyctl deploy apache-proxy % flyctl deploy --build-arg DATASETTE_REF=main Update available 0.0.229 -> v0.0.255 Run ""flyctl version update"" to upgrade Deploying floral-dust-4577 ==> Validating app configuration --> Validating app configuration done Services TCP 80/443 ⇢ 8080 ==> Creating build context --> Creating build context done ==> Building image with Docker Sending build context to Docker daemon 8.704kB [+] Building 15.7s (27/27) ... 0.0s ==> Pushing image to fly The push refers to repository [registry.fly.io/floral-dust-4577] 9bf88c92aa2a: Pushed 3d61728b8391: Pushed ... --> Pushing image done Image: registry.fly.io/floral-dust-4577:deployment-1637429501 Image size: 276 MB ==> Creating release Release v2 created You can detach the terminal anytime without stopping the deployment Monitoring Deployment 1 desired, 1 placed, 0 healthy, 0 unhealthy [health checks: 1 total, 1 critical] 1 desired, 1 placed, 0 healthy, 1 unhealthy [health checks: 1 total, 1 critical] v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to Failed Instances ==> Failure #1 Instance ID = 36adac86 Version = 0 Region = sjc Desired = run Status = running Health Checks = 1 total, 1 critical Restarts = 0 Created = 4m52s ago Recent Events TIMESTAMP TYPE MESSAGE 2021-11-20T17:32:52Z Received Task received by client 2021-11-20T17:32:52Z Task Setup Building Task Directory 2021-11-20T17:33:02Z Started Task started by client Recent Logs 2021-11-20T17:32:56Z [info] Unpacking image 2021-11-20T17:33:01Z [info] Preparing kernel init 2021-11-20T17:33:01Z [info] Configuring firecracker 2021-11-20T17:33:02Z [info] Starting virtual machine 2021-11-20T17:33:02Z [info] Starting init (commit: 7943db6)... 2021-11-20T17:33:02Z [info] Preparing to run: `/usr/bin/supervisord -c /app/supervisord.conf` as root 2021-11-20T17:33:02Z [info] 2021/11/20 17:33:02 listening on [fdaa:0:4ef:a7b:2295:36ad:ac86:2]:22 (DNS: [fdaa::3]:53) 2021-11-20T17:33:02Z [info] 2021-11-20 17:33:02,374 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2021-11-20T17:33:02Z [info] 2021-11-20 17:33:02,376 INFO supervisord started with pid 510 2021-11-20T17:33:03Z [info] 2021-11-20 17:33:03,379 INFO spawned: 'apache2' with pid 515 2021-11-20T17:33:03Z [info] 2021-11-20 17:33:03,381 INFO spawned: 'datasette' with pid 516 2021-11-20T17:33:05Z [info] 2021-11-20 17:33:05,068 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-11-20T17:33:05Z [info] 2021-11-20 17:33:05,068 INFO success: datasette entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-11-20T17:33:28Z [error] Health check status changed 'warning' => 'critical' ***v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974692546,https://api.github.com/repos/simonw/datasette/issues/1522,974692546,IC_kwDOBm6k_c46GKDC,9599,simonw,2021-11-20T18:34:13Z,2021-11-20T18:34:13Z,OWNER,"Here's why it failed - the `fly.toml` file that was generated when I ran `flyctl launch` had this section: ```toml [[services]] http_checks = [] internal_port = 8080 processes = [""app""] protocol = ""tcp"" script_checks = [] ``` But I need `internal_port` to be 80 for Apache, so I changed that and ran `flyctl deploy --build-arg DATASETTE_REF=main` again - and it worked! https://floral-dust-4577.fly.dev/prefix/ - not seeing any 503 errors there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974693350,https://api.github.com/repos/simonw/datasette/issues/1522,974693350,IC_kwDOBm6k_c46GKPm,9599,simonw,2021-11-20T18:39:27Z,2021-11-20T18:39:27Z,OWNER,"I'm going to go with Fly instead for this, especially as I can keep it within their free tier (and iI want to get more familiar with their platform).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974695111,https://api.github.com/repos/simonw/datasette/issues/1522,974695111,IC_kwDOBm6k_c46GKrH,9599,simonw,2021-11-20T18:52:11Z,2021-11-20T18:52:11Z,OWNER,The demo is now live on https://datasette-apache-proxy-demo.fly.dev/prefix/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974697824,https://api.github.com/repos/simonw/datasette/issues/1519,974697824,IC_kwDOBm6k_c46GLVg,9599,simonw,2021-11-20T19:11:21Z,2021-11-20T19:11:21Z,OWNER,"OK, i think I got all of them this time! The latest demo is now live at https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures/sortable?_facet=pk2 I'm closing this issue, but feel free to re-open it if you spot any that I missed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1524#issuecomment-974704254,https://api.github.com/repos/simonw/datasette/issues/1524,974704254,IC_kwDOBm6k_c46GM5-,9599,simonw,2021-11-20T20:03:51Z,2021-11-20T20:22:52Z,OWNER,"I'm also going to extract the Apache config files from https://github.com/simonw/datasette/blob/250db8192cb8aba5eb8cd301ccc2a49525bc3d24/demos/apache-proxy/Dockerfile into a separate file to make it easier to read. (The supervisor config needs to be dynamically constructed to include $DATASETTE_REF so I will leave it where it is.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059219106,"Improve Apache proxy documentation, link to demo", https://github.com/simonw/datasette/issues/1519#issuecomment-974701788,https://api.github.com/repos/simonw/datasette/issues/1519,974701788,IC_kwDOBm6k_c46GMTc,9599,simonw,2021-11-20T19:42:29Z,2021-11-20T19:42:29Z,OWNER,"> I think what's happening here is Apache is actually making a request to `/fixtures` rather than making a request to `/prefix/fixtures` - and Datasette is replying to requests on both the prefixed and the non-prefixed paths. > > This is pretty confusing! I think Datasette should ONLY reply to `/prefix/fixtures` instead and return a 404 for `/fixtures` - this would make things a whole lot easier to debug. > > But shipping that change could break existing deployments. Maybe that should be a breaking change for 1.0. On further thought I'm not going to do this. Having Datasette work behind a proxy the way it does right now is clearly easy for people to deploy (now that I've fixed the bugs) and I trust my improved tests to catch problems in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1524#issuecomment-974707878,https://api.github.com/repos/simonw/datasette/issues/1524,974707878,IC_kwDOBm6k_c46GNym,9599,simonw,2021-11-20T20:34:51Z,2021-11-20T20:38:29Z,OWNER,"I pointed `CNAME` of `datasette-apache-proxy-demo.datasette.io` at `datasette-apache-proxy-demo.fly.dev.` using Vercel DNS: Then I asked Fly to issue a LetsEncrypt certificate for that: ``` % flyctl certs create datasette-apache-proxy-demo.datasette.io # About 53 seconds later: % flyctl certs show datasette-apache-proxy-demo.datasette.io The certificate for datasette-apache-proxy-demo.datasette.io has been issued. Hostname = datasette-apache-proxy-demo.datasette.io DNS Provider = constellix Certificate Authority = Let's Encrypt Issued = ecdsa,rsa Added to App = 53 seconds ago Source = fly ``` https://datasette-apache-proxy-demo.datasette.io/ works now - I'll use that in the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059219106,"Improve Apache proxy documentation, link to demo", https://github.com/simonw/datasette/issues/1524#issuecomment-974721652,https://api.github.com/repos/simonw/datasette/issues/1524,974721652,IC_kwDOBm6k_c46GRJ0,9599,simonw,2021-11-20T22:41:03Z,2021-11-20T22:41:03Z,OWNER,New TIL: https://til.simonwillison.net/fly/custom-subdomain-fly,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059219106,"Improve Apache proxy documentation, link to demo", https://github.com/simonw/datasette/issues/1524#issuecomment-974725814,https://api.github.com/repos/simonw/datasette/issues/1524,974725814,IC_kwDOBm6k_c46GSK2,9599,simonw,2021-11-20T23:24:01Z,2021-11-20T23:24:01Z,OWNER,"I noticed that `http://datasette-apache-proxy-demo.datasette.io/` wasn't redirecting to `https` so I built a new plugin: https://github.com/simonw/datasette-redirect-to-https ``` % curl -i 'http://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key' HTTP/1.1 301 Moved Permanently date: Sat, 20 Nov 2021 23:22:50 GMT server: Fly/51d150d (2021-11-19) location: https://datasette-apache-proxy-demo.datasette.io/fixtures/no_primary_key x-proxied-by: Apache2 Debian transfer-encoding: chunked via: 1.1 fly.io fly-request-id: 01FMZTHTHVPC8BZY0625D7JV4B ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059219106,"Improve Apache proxy documentation, link to demo", https://github.com/simonw/datasette/issues/1525#issuecomment-974912985,https://api.github.com/repos/simonw/datasette/issues/1525,974912985,IC_kwDOBm6k_c46G_3Z,9599,simonw,2021-11-21T22:55:42Z,2021-11-21T22:55:42Z,OWNER,Here's the template: https://github.com/simonw/datasette/blob/48f11998b73350057b74fe6ab464d4ac3071637c/datasette/templates/row.html#L39-L41,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059509927,"""Links from other tables"" broken for columns starting with underscore", https://github.com/simonw/datasette/issues/1525#issuecomment-974913180,https://api.github.com/repos/simonw/datasette/issues/1525,974913180,IC_kwDOBm6k_c46G_6c,9599,simonw,2021-11-21T22:57:08Z,2021-11-21T22:57:08Z,OWNER,https://latest.datasette.io/fixtures/facetable can't quite demonstrate the bug because `_neighborhood` isn't a foreign key - I should rename `city_id` to `_city_id`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059509927,"""Links from other tables"" broken for columns starting with underscore", https://github.com/simonw/datasette/issues/93#issuecomment-974765825,https://api.github.com/repos/simonw/datasette/issues/93,974765825,IC_kwDOBm6k_c46Gb8B,9599,simonw,2021-11-21T07:00:21Z,2021-11-21T07:00:21Z,OWNER,Closing this in favour of Datasette Desktop: https://datasette.io/desktop,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary, https://github.com/simonw/datasette/issues/1527#issuecomment-974979785,https://api.github.com/repos/simonw/datasette/issues/1527,974979785,IC_kwDOBm6k_c46HQLJ,9599,simonw,2021-11-22T01:02:57Z,2021-11-22T01:03:19Z,OWNER,"I think the root cause is this hidden form field on https://latest.datasette.io/fixtures/facetable?_facet=_neighborhood&_neighborhood__exact=Downtown ```html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1526#issuecomment-975073308,https://api.github.com/repos/simonw/datasette/issues/1526,975073308,IC_kwDOBm6k_c46HnAc,9599,simonw,2021-11-22T04:13:46Z,2021-11-22T04:13:46Z,OWNER,"Addressing that over here (hadn't seen that issue yet, thanks for the prod): https://github.com/simonw/datasette-publish-vercel/issues/51#issuecomment-975073026","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059549523,"Add to vercel.json, rather than overwriting it.", https://github.com/simonw/datasette/issues/1526#issuecomment-975110692,https://api.github.com/repos/simonw/datasette/issues/1526,975110692,IC_kwDOBm6k_c46HwIk,9599,simonw,2021-11-22T04:49:44Z,2021-11-22T04:49:44Z,OWNER,Fixed in the 0.12 release of that plugin: https://github.com/simonw/datasette-publish-vercel/releases/tag/0.12,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059549523,"Add to vercel.json, rather than overwriting it.", https://github.com/simonw/datasette/issues/1518#issuecomment-981172385,https://api.github.com/repos/simonw/datasette/issues/1518,981172385,IC_kwDOBm6k_c46e4Ch,9599,simonw,2021-11-28T23:21:26Z,2021-11-28T23:21:26Z,OWNER,"Aside: is there any reason this work can't complete the long-running goal of merging the TableView and QueryView, such that most of the features available for tables become available for arbitrary queries too? I had already mentally committed to implementing facets for queries, but I just realized that filters could work too - using either a CTE or a nested query. Pagination is the one holdout here, since table pagination uses keyset pagination over a known order. But maybe arbitrary queries can only be paginated off you order them first?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-981172801,https://api.github.com/repos/simonw/datasette/issues/1518,981172801,IC_kwDOBm6k_c46e4JB,9599,simonw,2021-11-28T23:23:51Z,2021-11-28T23:23:51Z,OWNER,"(I could experiment with merging the two tables by adding a temporary undocumented `?_sql=` parameter to the in-progress table view that sets an alternative query instead of `select cols from table` - added bonus, this will force me to use introspection against the returned columns rather than mixing in the known columns for the specified table)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1533#issuecomment-981149039,https://api.github.com/repos/simonw/datasette/issues/1533,981149039,IC_kwDOBm6k_c46eyVv,9599,simonw,2021-11-28T20:45:36Z,2021-11-28T20:45:36Z,OWNER,I built an initial prototype of this in a branch: https://github.com/simonw/datasette/commit/e0a84691c2959f2d1d76948574c9c4a910c7556c - which exposed even more flaws in the way `TableView` is structured (adding custom HTTP headers to the response is way harder than it should be) which I should address in the refactor in #617.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065431383,"Add `Link: rel=""alternate""` header pointing to JSON for a table/query", https://github.com/simonw/datasette/issues/1534#issuecomment-981149531,https://api.github.com/repos/simonw/datasette/issues/1534,981149531,IC_kwDOBm6k_c46eydb,9599,simonw,2021-11-28T20:48:54Z,2021-11-28T20:48:54Z,OWNER,"If I'm going to do this, is there value in also spotting `Accept: text/csv` and returning CSV for that? I'm pretty sure no client has EVER implemented this though, so it feels like it would be showboating.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/issues/1518#issuecomment-981153060,https://api.github.com/repos/simonw/datasette/issues/1518,981153060,IC_kwDOBm6k_c46ezUk,9599,simonw,2021-11-28T21:13:09Z,2021-12-17T23:37:08Z,OWNER,"Two new requirements inspired by work on the `datasette-table` (and `datasette-notebook`) projects: - #1533 - #1534","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-981153186,https://api.github.com/repos/simonw/datasette/issues/1518,981153186,IC_kwDOBm6k_c46ezWi,9599,simonw,2021-11-28T21:13:50Z,2021-11-28T21:13:50Z,OWNER,"I'm also going to use the new `datasette-table` Web Component to help guide the design of the new API, which relates directly to this issue too: - #1532","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1538#issuecomment-981849494,https://api.github.com/repos/simonw/datasette/issues/1538,981849494,IC_kwDOBm6k_c46hdWW,9599,simonw,2021-11-29T17:23:52Z,2021-11-29T17:23:52Z,OWNER,"Just trying to use `git_history_file` produces this error: > `TypeError: Attempted to convert a callback into a command twice.` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066288689,Research pattern for re-registering existing Click tools with register_commands, https://github.com/simonw/datasette/issues/1538#issuecomment-981852280,https://api.github.com/repos/simonw/datasette/issues/1538,981852280,IC_kwDOBm6k_c46heB4,9599,simonw,2021-11-29T17:27:12Z,2021-11-29T17:27:12Z,OWNER,"Thanks to https://stackoverflow.com/a/45514541/6083 I found the right pattern: ```python from datasette import hookimpl from git_history.cli import cli as git_history_cli @hookimpl def register_commands(cli): cli.add_command(git_history_cli, name=""git-history"") ``` I think this is a little bit too obscure to add to the Datasette documentation - I'll turn it into a TIL instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066288689,Research pattern for re-registering existing Click tools with register_commands, https://github.com/simonw/datasette/issues/1538#issuecomment-981856895,https://api.github.com/repos/simonw/datasette/issues/1538,981856895,IC_kwDOBm6k_c46hfJ_,9599,simonw,2021-11-29T17:32:44Z,2021-11-29T17:32:44Z,OWNER,TIL: https://til.simonwillison.net/datasette/reuse-click-for-register-commands,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066288689,Research pattern for re-registering existing Click tools with register_commands, https://github.com/simonw/datasette/issues/1527#issuecomment-982235541,https://api.github.com/repos/simonw/datasette/issues/1527,982235541,IC_kwDOBm6k_c46i7mV,9599,simonw,2021-11-30T02:57:34Z,2021-11-30T02:58:44Z,OWNER,"I started fiddling with a test for this which extracts the `` fields, but I probably won't use it: ```python def test_exact_parameter_results_in_correct_hidden_fields(app_client): # https://github.com/simonw/datasette/issues/1527 response = app_client.get( ""/fixtures/facetable?_facet=_neighborhood&_neighborhood__exact=Downtown"" ) # In this case we should NOT have a hidden _neighborhood__exact=Downtown field form = Soup(response.body, ""html.parser"").find(""form"") selects = [ { ""name"": select[""name""], ""value"": select.select(""option[selected]"")[0].text if select.select(""option[selected]"") else """", } for select in form.findAll(""select"") ] inputs = [input.attrs for input in form.findAll(""input"")] # Turn those both into a {name: (value, type)} array form_inputs = {} form_inputs.update( {select[""name""]: (select[""value""], ""select"") for select in selects} ) form_inputs.update( { input[""name""]: (input.get(""value""), input[""type""]) for input in inputs if input.get(""name"") } ) assert form_inputs == { ""_filter_column_1"": (""_neighborhood"", ""select""), ""_filter_op_1"": (""="", ""select""), ""_filter_value_1"": (""Downtown"", ""text""), ""_filter_column"": ("""", ""select""), ""_filter_op"": ("""", ""select""), ""_filter_value"": (None, ""text""), ""_sort"": (""Sort by pk"", ""select""), ""_sort_by_desc"": (None, ""checkbox""), ""_facet"": (""_neighborhood"", ""hidden""), ""_neighborhood__exact"": (""Downtown"", ""hidden""), } ``` The problem is that last hidden field, `_neighborhood__exact=Downtown` - which should not be there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1527#issuecomment-982318745,https://api.github.com/repos/simonw/datasette/issues/1527,982318745,IC_kwDOBm6k_c46jP6Z,9599,simonw,2021-11-30T06:11:21Z,2021-11-30T06:11:21Z,OWNER,"Manually tested this too, looks like that fixed it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1532#issuecomment-982319210,https://api.github.com/repos/simonw/datasette/issues/1532,982319210,IC_kwDOBm6k_c46jQBq,9599,simonw,2021-11-30T06:12:19Z,2021-11-30T06:12:19Z,OWNER,That's really cool to hear - I've not seen many people actively building on top of the JSON API.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936,Use datasette-table Web Component to guide the design of the JSON API for 1.0, https://github.com/simonw/datasette/issues/1525#issuecomment-982331602,https://api.github.com/repos/simonw/datasette/issues/1525,982331602,IC_kwDOBm6k_c46jTDS,9599,simonw,2021-11-30T06:39:00Z,2021-11-30T06:39:00Z,OWNER,"These two pages now help demonstrate the fix: - https://latest.datasette.io/fixtures/facet_cities/1 - https://latest.datasette.io/fixtures/attraction_characteristic/2 I added a new test for these here: https://github.com/simonw/datasette/blob/35b12746ba2bf9f254791bddac03d25b19be9b77/tests/test_html.py#L823-L848 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059509927,"""Links from other tables"" broken for columns starting with underscore", https://github.com/simonw/datasette/issues/1540#issuecomment-984048965,https://api.github.com/repos/simonw/datasette/issues/1540,984048965,IC_kwDOBm6k_c46p2VF,9599,simonw,2021-12-01T20:59:26Z,2021-12-01T21:02:58Z,OWNER,"This is a bit of a mess but it does keep the hovercard around for a moment and then fade it away when you mouse out of it: ```html+jinja {% extends ""base.html"" %} {% block content %}

Hovercards demo

Here is a link to a row {% endblock %} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1540#issuecomment-984051925,https://api.github.com/repos/simonw/datasette/issues/1540,984051925,IC_kwDOBm6k_c46p3DV,9599,simonw,2021-12-01T21:03:16Z,2021-12-01T21:03:16Z,OWNER,Needs `pageX` not `clientX` because otherwise it doesn't work when you scroll down the page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1540#issuecomment-984053760,https://api.github.com/repos/simonw/datasette/issues/1540,984053760,IC_kwDOBm6k_c46p3gA,9599,simonw,2021-12-01T21:05:20Z,2021-12-01T21:05:20Z,OWNER,"I realized you couldn't click the links any more because the hovercard overlapped them, so I changed it to this instead. Need to reconsider the when-to-hide logic though. ```javascript hovercard.style.top = (ev.pageY + 15) + 'px'; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1540#issuecomment-983985330,https://api.github.com/repos/simonw/datasette/issues/1540,983985330,IC_kwDOBm6k_c46pmyy,9599,simonw,2021-12-01T19:29:05Z,2021-12-01T19:29:05Z,OWNER,"The layout of the hover card could be similar to the one used by `datasette-cluster-map`: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1540#issuecomment-984037711,https://api.github.com/repos/simonw/datasette/issues/1540,984037711,IC_kwDOBm6k_c46pzlP,9599,simonw,2021-12-01T20:42:17Z,2021-12-01T20:43:14Z,OWNER,"A first prototype (saved as `templates/pages/hovercard.html` and run with `datasette fixtures.db --template-dir=templates`): ```html+jinja {% extends ""base.html"" %} {% block content %}

Hovercards demo

Here is a link to a row {% endblock %} ``` ![hovercard](https://user-images.githubusercontent.com/9599/144310888-6db71bad-b6f6-4d8a-a737-81a618022bbe.gif) Lots of decisions to make here. Most importantly, when should it be hidden again?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1540#issuecomment-984801331,https://api.github.com/repos/simonw/datasette/issues/1540,984801331,IC_kwDOBm6k_c46suAz,9599,simonw,2021-12-02T16:42:02Z,2021-12-09T23:38:39Z,OWNER,"I'm going to wrap this up in a plugin for the moment - I want it in Datasette core but I'd like to improve the implementation first with things like support for `base_url` which will likely depend on #1533 or similar. Here's the plugin: https://github.com/simonw/datasette-hovercards","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1541#issuecomment-984908185,https://api.github.com/repos/simonw/datasette/issues/1541,984908185,IC_kwDOBm6k_c46tIGZ,9599,simonw,2021-12-02T18:56:54Z,2021-12-02T18:56:54Z,OWNER,Also it should link to foreign keys like the table page does.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1069881276,Different default layout for row page, https://github.com/simonw/datasette/issues/1585#issuecomment-1003575286,https://api.github.com/repos/simonw/datasette/issues/1585,1003575286,IC_kwDOBm6k_c470Vf2,9599,simonw,2022-01-01T15:40:38Z,2022-01-01T15:40:38Z,OWNER,API tutorial: https://firebase.google.com/docs/hosting/api-deploy,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1091838742,Fire base caching for `publish cloudrun`, https://github.com/simonw/datasette/issues/1534#issuecomment-1005975080,https://api.github.com/repos/simonw/datasette/issues/1534,1005975080,IC_kwDOBm6k_c479fYo,9599,simonw,2022-01-05T18:29:06Z,2022-01-05T18:29:06Z,OWNER,"A really big downside to this is that it turns out many CDNs - apparently including Cloudflare - don't support the Vary header at all! More in this thread: https://twitter.com/simonw/status/1478470282931163137","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/issues/1518#issuecomment-991285527,https://api.github.com/repos/simonw/datasette/issues/1518,991285527,IC_kwDOBm6k_c47FdEX,9599,simonw,2021-12-10T20:52:00Z,2021-12-10T20:52:00Z,OWNER,"If I break this up into `@inject` methods, what methods could I have and what would they do? - `resolve_path`: Use request path to resolve the database and table. Could handle hash URLs too (if I don't manage to extract those to a plugin) - would be nice if this could raise a redirect, but I think that will instead have to be one of the things it returns - `build_sql`: Builds the SQL query based on the querystring (and some DB introspection) - `execute_count`: Execute the `count(*)` - `execute_rows`: Execute the `limit 101` to fetch the rows - `execute_facets`: Execute all requested facets (could this do its own `asyncio.gather()` to run facets in parallel?) - `suggest_facets`: Execute facet suggestions Are there any plugin hooks that would make sense to execute in parallel? Actually there might be: I don't think `extra_template_vars`, `extra_css_urls`, `extra_js_urls`, `extra_body_script` depend on each other so it might be possible to execute them in a parallel chunk (at least any of them that return awaitables).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1549#issuecomment-991752486,https://api.github.com/repos/simonw/datasette/issues/1549,991752486,IC_kwDOBm6k_c47HPEm,9599,simonw,2021-12-11T19:09:15Z,2021-12-11T19:09:15Z,OWNER,"That's what this option does: ![EAB1B9E8-38E9-4C6D-8854-BD1935F163D9](https://user-images.githubusercontent.com/9599/145688531-668bafa1-e287-4bbd-84d6-157241fb1f68.jpeg) The usability of this is pretty terrible though (including ""stream all rows"" - how are people meant to understand what that does?) so it can definitely do with some rethinking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability, https://github.com/simonw/datasette/issues/1549#issuecomment-991754794,https://api.github.com/repos/simonw/datasette/issues/1549,991754794,IC_kwDOBm6k_c47HPoq,9599,simonw,2021-12-11T19:16:33Z,2021-12-11T19:16:33Z,OWNER,Good call! I'm doing a refactor #1518 right now which will hopefully bring the functionality of those two much closer - I'll make a note to consider this there too.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability, https://github.com/simonw/datasette/issues/617#issuecomment-991755013,https://api.github.com/repos/simonw/datasette/issues/617,991755013,IC_kwDOBm6k_c47HPsF,9599,simonw,2021-12-11T19:17:11Z,2021-12-11T19:17:11Z,OWNER,This work is now happening in #1518 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519613116,Refactor TableView.data() method, https://github.com/simonw/datasette/issues/1549#issuecomment-991755245,https://api.github.com/repos/simonw/datasette/issues/1549,991755245,IC_kwDOBm6k_c47HPvt,9599,simonw,2021-12-11T19:17:54Z,2021-12-11T19:17:54Z,OWNER,"Also relevant: - #1062 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability, https://github.com/simonw/datasette/issues/1550#issuecomment-991761635,https://api.github.com/repos/simonw/datasette/issues/1550,991761635,IC_kwDOBm6k_c47HRTj,9599,simonw,2021-12-11T19:39:01Z,2021-12-11T19:39:01Z,OWNER,"I wonder if this could work for public instances too with some kind of queuing mechanism? I really need to use benchmarking to figure out what the right number of maximum SQLite connections is. I'm just guessing at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077628073,Research option for returning all rows from arbitrary query, https://github.com/simonw/datasette/issues/1550#issuecomment-991805516,https://api.github.com/repos/simonw/datasette/issues/1550,991805516,IC_kwDOBm6k_c47HcBM,9599,simonw,2021-12-11T23:43:24Z,2021-12-11T23:43:24Z,OWNER,"I built a tiny Starlette app to experiment with this a bit: ```python import asyncio import janus from starlette.applications import Starlette from starlette.responses import JSONResponse, HTMLResponse, StreamingResponse from starlette.routing import Route import sqlite3 from concurrent import futures executor = futures.ThreadPoolExecutor(max_workers=10) async def homepage(request): return HTMLResponse( """""" SQL CSV Server

SQL CSV Server

"""""" ) def run_query_in_thread(sql, sync_q): db = sqlite3.connect(""../datasette/covid.db"") cursor = db.cursor() cursor.arraysize = 100 # Default is 1 apparently? cursor.execute(sql) columns = [d[0] for d in cursor.description] sync_q.put([columns]) # Now start putting batches of rows while True: rows = cursor.fetchmany() if rows: sync_q.put(rows) else: break # Let queue know we are finished\ sync_q.put(None) async def csv_query(request): sql = request.query_params[""sql""] queue = janus.Queue() loop = asyncio.get_running_loop() async def csv_generator(): loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q) while True: rows = await queue.async_q.get() if rows is not None: for row in rows: yield "","".join(map(str, row)) + ""\n "" queue.async_q.task_done() else: # Cleanup queue.close() await queue.wait_closed() break return StreamingResponse(csv_generator(), media_type='text/plain') app = Starlette( debug=True, routes=[ Route(""/"", homepage), Route(""/csv"", csv_query), ], ) ``` But.. if I run this in a terminal window: ``` /tmp % wget 'http://127.0.0.1:8000/csv?sql=select+*+from+ny_times_us_counties' ``` it takes about 20 seconds to run and returns a 50MB file - but while it is running no other requests can be served by that server - not even the homepage! So something is blocking the event loop. Maybe I should be using `fut = loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)` and then awaiting `fut` somewhere, like in the Janus documentation? Don't think that's needed though. Needs more work to figure out why this is blocking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077628073,Research option for returning all rows from arbitrary query, https://github.com/simonw/datasette/issues/1518#issuecomment-991819781,https://api.github.com/repos/simonw/datasette/issues/1518,991819781,IC_kwDOBm6k_c47HfgF,9599,simonw,2021-12-12T01:53:10Z,2021-12-12T01:53:10Z,OWNER,"I have a hunch that the conclusion of this experiment may end up being that the `asyncinject` trick is kinda neat but the code will be easier to maintain (while still executing in parallel) if it's written using `asyncio.gather` directly instead. It's possible `asyncinject` will end up being neat enough that I'll want to keep it though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991822853,https://api.github.com/repos/simonw/datasette/issues/1518,991822853,IC_kwDOBm6k_c47HgQF,9599,simonw,2021-12-12T02:24:00Z,2021-12-12T02:24:00Z,OWNER,Rebuilding `TableView` from the ground up is proving not to be much fun. I'm going to explore starting the refactor of the existing code by separating out the bit that generates the SQL query from the rest of it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991823001,https://api.github.com/repos/simonw/datasette/issues/1518,991823001,IC_kwDOBm6k_c47HgSZ,9599,simonw,2021-12-12T02:25:32Z,2021-12-12T02:25:32Z,OWNER,The tests for `TableView` are currently mixed in with everything else in `tests/test_api.py` and `tests/html.py` - might be good to split those out into `test_table_html.py` and `test_table_api.py` since they're such a key part of how Datasette works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991827468,https://api.github.com/repos/simonw/datasette/issues/1518,991827468,IC_kwDOBm6k_c47HhYM,9599,simonw,2021-12-12T03:15:00Z,2021-12-12T03:15:00Z,OWNER," I don't think this code is necessary any more: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/views/table.py#L396-L399 That dates back from when Datasette was built on top of Sanic and Sanic didn't preserve those query parameters the way I needed it to: https://github.com/simonw/datasette/blob/1f69269fe93e4cd42e56890126cc0dbcf719c6cb/datasette/views/table.py#L202-L206","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991828014,https://api.github.com/repos/simonw/datasette/issues/1518,991828014,IC_kwDOBm6k_c47Hhgu,9599,simonw,2021-12-12T03:21:35Z,2021-12-12T03:21:35Z,OWNER,"No, removing that gave me the following test failure: ``` tests/test_table_api.py::test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] FAILED [100%] =============================================================================== FAILURES ================================================================================ ______________________________________ test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] ______________________________________ app_client = , path = '/fixtures/simple_primary_key.json?content__exact=', expected_rows = [['3', '']] @pytest.mark.parametrize( ""path,expected_rows"", [ (""/fixtures/simple_primary_key.json?content=hello"", [[""1"", ""hello""]]), ( ""/fixtures/simple_primary_key.json?content__contains=o"", [ [""1"", ""hello""], [""2"", ""world""], [""4"", ""RENDER_CELL_DEMO""], ], ), (""/fixtures/simple_primary_key.json?content__exact="", [[""3"", """"]]), ( ""/fixtures/simple_primary_key.json?content__not=world"", [ [""1"", ""hello""], [""3"", """"], [""4"", ""RENDER_CELL_DEMO""], [""5"", ""RENDER_CELL_ASYNC""], ], ), ], ) def test_table_filter_queries(app_client, path, expected_rows): response = app_client.get(path) > assert expected_rows == response.json[""rows""] E AssertionError: assert [['3', '']] == [['1', 'hello'],\n ['2', 'world'],\n ['3', ''],\n ['4', 'RENDER_CELL_DEMO'],\n ['5', 'RENDER_CELL_ASYNC']] E At index 0 diff: ['3', ''] != ['1', 'hello'] E Right contains 4 more items, first extra item: ['2', 'world'] E Full diff: E [ E - ['1', E - 'hello'], E - ['2', E - 'world'], E ['3', E ''], E - ['4', E - 'RENDER_CELL_DEMO'], E - ['5', E - 'RENDER_CELL_ASYNC'], E ] /Users/simon/Dropbox/Development/datasette/tests/test_table_api.py:511: AssertionError ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1551#issuecomment-991960416,https://api.github.com/repos/simonw/datasette/issues/1551,991960416,IC_kwDOBm6k_c47IB1g,9599,simonw,2021-12-12T19:56:12Z,2021-12-12T19:56:12Z,OWNER,"Python documentation for `parse_qs`: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.parse_qs > The optional argument *keep_blank_values* is a flag indicating whether blank values in percent-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`, https://github.com/simonw/datasette/issues/1551#issuecomment-991960719,https://api.github.com/repos/simonw/datasette/issues/1551,991960719,IC_kwDOBm6k_c47IB6P,9599,simonw,2021-12-12T19:58:17Z,2021-12-12T19:58:17Z,OWNER,"Here's an example of the difference that causes: ```pycon >>> import urllib.parse >>> urllib.parse.parse_qs(""foo=bar"") {'foo': ['bar']} >>> urllib.parse.parse_qs(""foo=bar&baz="") {'foo': ['bar']} >>> urllib.parse.parse_qs(""foo=bar&baz="", keep_blank_values=True) {'foo': ['bar'], 'baz': ['']} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`, https://github.com/simonw/datasette/issues/1551#issuecomment-991960179,https://api.github.com/repos/simonw/datasette/issues/1551,991960179,IC_kwDOBm6k_c47IBxz,9599,simonw,2021-12-12T19:54:45Z,2021-12-12T19:54:45Z,OWNER,This is technically a backwards-incompatible for any plugins that use `request.args` - but it's unlikely to break anything. At any rate this needs to happen before Datasette 1.0!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`, https://github.com/simonw/datasette/issues/1518#issuecomment-991978789,https://api.github.com/repos/simonw/datasette/issues/1518,991978789,IC_kwDOBm6k_c47IGUl,9599,simonw,2021-12-12T22:04:19Z,2021-12-12T22:04:19Z,OWNER,Idea: in JSON output include a `warnings` block listing any _ parameters that were not recognized.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-992833868,https://api.github.com/repos/simonw/datasette/issues/1518,992833868,IC_kwDOBm6k_c47LXFM,9599,simonw,2021-12-13T19:59:17Z,2021-12-13T19:59:17Z,OWNER,"Built a new plugin to help with this work by improving the display of `?_trace=1` output: https://datasette.io/plugins/datasette-pretty-traces ![image](https://user-images.githubusercontent.com/9599/145879751-36621f43-ba68-4ccd-b14b-379ed8f2111a.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-993000787,https://api.github.com/repos/simonw/datasette/issues/1518,993000787,IC_kwDOBm6k_c47L_1T,9599,simonw,2021-12-13T23:19:20Z,2021-12-14T17:06:05Z,OWNER,"Useful old comment here: https://github.com/simonw/datasette/issues/617#issuecomment-552253893 > As noted in [#621 (comment)](https://github.com/simonw/datasette/issues/621#issuecomment-552253208) a common pattern in this method is blocks of code that append new items to the `where_clauses`, `params` and `extra_human_descriptions` arrays. This is a useful refactoring opportunity. > > Code that fits this pattern: > > * The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)` > * Code that handles `?_where=`: `where_clauses.extend(request.args[""_where""])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses > * The `_through=` code, see [Syntax for ?_through= that works as a form field #621](https://github.com/simonw/datasette/issues/621) for details > * The code that deals with `?_search=` FTS > > The keyset pagination code modifies `where_clauses` and `params` too, but I don't think it's quite going to work with the same abstraction that would cover the above examples. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/pull/1554#issuecomment-993006521,https://api.github.com/repos/simonw/datasette/issues/1554,993006521,IC_kwDOBm6k_c47MBO5,9599,simonw,2021-12-13T23:28:47Z,2021-12-13T23:28:47Z,OWNER,"That's frustrating: you can only attach comments to lines that were changed in the PR or are within about 3-4 lines of them: ![comments](https://user-images.githubusercontent.com/9599/145905357-5d8873f5-99c9-4b46-b4d5-35d38f5cb686.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079129258,TableView refactor, https://github.com/simonw/datasette/issues/1518#issuecomment-994042389,https://api.github.com/repos/simonw/datasette/issues/1518,994042389,IC_kwDOBm6k_c47P-IV,9599,simonw,2021-12-14T21:35:53Z,2021-12-14T21:35:53Z,OWNER,"Maybe a better way to approach this would be to focus on the JSON side of things - try to get a basic JSON version with `?_extra=` support working, then eventually build that up to the point where it can power the HTML version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/621#issuecomment-994005634,https://api.github.com/repos/simonw/datasette/issues/621,994005634,IC_kwDOBm6k_c47P1KC,9599,simonw,2021-12-14T21:02:50Z,2021-12-14T21:02:50Z,OWNER,"This would also mean that an extra text input box could be easily shown on the page. https://latest-with-plugins.datasette.io/fixtures/roadside_attractions?_through={""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id"",""value"":""1""} but with the annotated box added (and made to look good): ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/1518#issuecomment-993794247,https://api.github.com/repos/simonw/datasette/issues/1518,993794247,IC_kwDOBm6k_c47PBjH,9599,simonw,2021-12-14T17:09:40Z,2021-12-14T17:09:40Z,OWNER,- `table_actions` should be an extra.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/621#issuecomment-993813210,https://api.github.com/repos/simonw/datasette/issues/621,993813210,IC_kwDOBm6k_c47PGLa,9599,simonw,2021-12-14T17:30:13Z,2021-12-14T20:23:57Z,OWNER,"Might be able to create a web form that's unambiguous using: `https://latest.datasette.io/fixtures/roadside_attractions?_through.[""roadside_attraction_characteristics"",""characteristic_id""]=1` So: ```html ``` I'm pretty confident this is allowed by the HTML specification. This works: ```html
``` ASGI parsing seems to work too: https://latest-with-plugins.datasette.io/-/asgi-scope?_through.[%22roadside_attraction_characteristics%22%2C%22characteristic_id%22]=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/621#issuecomment-993958242,https://api.github.com/repos/simonw/datasette/issues/621,993958242,IC_kwDOBm6k_c47Ppli,9599,simonw,2021-12-14T20:33:25Z,2021-12-14T20:33:56Z,OWNER,"Alternative idea: since current syntax is: `?_through={""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id"",""value"":""1""}` The form-encoding-friendly syntax could be: `?_through.{""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id""}=1` Which is more consistent than the array proposal: `?_through.[""roadside_attraction_characteristics"",""characteristic_id""]=1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/1518#issuecomment-994085710,https://api.github.com/repos/simonw/datasette/issues/1518,994085710,IC_kwDOBm6k_c47QItO,9599,simonw,2021-12-14T22:03:16Z,2021-12-14T22:04:28Z,OWNER,"There are actually four forms of SQL query used by the table page: - `from_sql` - just the `from table_name where ...` - `sql_no_order_no_limit` - used for faceting, `""select {select_all_columns} from {table_name} {where}""` - `sql` - the above but with order and limit clauses: `""select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}""` - `count_sql` used for the count, built out of `from_sql`: `""select count(*) {from_sql}""` I'm tempted to encapsulate those in a `Query` class.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1542#issuecomment-995022217,https://api.github.com/repos/simonw/datasette/issues/1542,995022217,IC_kwDOBm6k_c47TtWJ,9599,simonw,2021-12-15T17:47:07Z,2021-12-15T17:47:07Z,OWNER,"This does make sense to me. I've been hoping to significantly improve the way JavaScript plugins work - there are some notes on that here: - #983 Encouraging plugins such as `datasette-cluster-map` to emit events that can then be listened to by other plugins is a really interesting idea that I hadn't considered.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072106103,feature request: order and dependency of plugins (that use js), https://github.com/simonw/datasette/issues/1423#issuecomment-995023410,https://api.github.com/repos/simonw/datasette/issues/1423,995023410,IC_kwDOBm6k_c47Ttoy,9599,simonw,2021-12-15T17:48:40Z,2021-12-15T17:48:40Z,OWNER,You've caused me to rethink this feature - I no longer think there's value in only showing these numbers if `?_facet_size=max` as opposed to all of the time. New issue coming up.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1552#issuecomment-995034143,https://api.github.com/repos/simonw/datasette/issues/1552,995034143,IC_kwDOBm6k_c47TwQf,9599,simonw,2021-12-15T18:02:53Z,2021-12-15T18:02:53Z,OWNER,"This is definitely a missing feature. The ""different types of facet"" stuff feels incomplete to me generally - this is one issue, but this one as well: - #625","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/262#issuecomment-995034911,https://api.github.com/repos/simonw/datasette/issues/262,995034911,IC_kwDOBm6k_c47Twcf,9599,simonw,2021-12-15T18:03:46Z,2021-12-15T18:03:56Z,OWNER,"This is relevant to the big refactor in: - #1518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/issues/625#issuecomment-996100774,https://api.github.com/repos/simonw/datasette/issues/625,996100774,IC_kwDOBm6k_c47X0qm,9599,simonw,2021-12-16T19:10:01Z,2021-12-16T19:10:48Z,OWNER,"I think the problem here may be in the design of the JSON returned by facets. It looks like this: ``` ""facet_results"": { ""tags"": { ""name"": ""tags"", ""type"": ""array"", ""results"": [...], ""hideable"": false, ""toggle_url"": ""/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1"", ""truncated"": false }, ""created"": { ""name"": ""created"", ""type"": ""date"", ""results"": [...] ``` The problem then is that the `tags` key is over-ridden by the second facet with a different type against the same column name! https://latest-with-plugins.datasette.io/fixtures/facetable?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags confirms that the SQL queries for those facets are being executed - but the final JSON doesn't show them on https://latest-with-plugins.datasette.io/fixtures/facetable.json?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags They're not available in the template context either: https://latest-with-plugins.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet_array=tags&_facet=tags&_context=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1553#issuecomment-996103956,https://api.github.com/repos/simonw/datasette/issues/1553,996103956,IC_kwDOBm6k_c47X1cU,9599,simonw,2021-12-16T19:14:38Z,2021-12-16T19:14:38Z,OWNER,This is a really interesting idea - kind of similar to how many APIs include custom HTTP headers informing of rate-limits.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1079111498,if csv export is truncated in non streaming mode set informative response header, https://github.com/simonw/datasette/issues/1556#issuecomment-996104214,https://api.github.com/repos/simonw/datasette/issues/1556,996104214,IC_kwDOBm6k_c47X1gW,9599,simonw,2021-12-16T19:15:00Z,2021-12-16T19:15:28Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet=planet_int&_facet=_city_id&_facet=created#facet-created ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1081318247,"Show count of facet values always, not just for `?_facet_size=max`", https://github.com/simonw/datasette/issues/625#issuecomment-996118401,https://api.github.com/repos/simonw/datasette/issues/625,996118401,IC_kwDOBm6k_c47X4-B,9599,simonw,2021-12-16T19:34:28Z,2021-12-16T19:34:55Z,OWNER,"The big question here is do I break any existing clients of the `""facet_results""` JSON API? It's still pre-1.0 so I could break them, but I've also built my own code against this in the past so it's likely other people have too. If I don't break them, I will instead need to come up with a naming convention for those keys - something like `""tags__array""` for example. As well as a way to ensure that a column called `tags__array` doesn't end up conflicting with the `tags__array` key!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1557#issuecomment-996115949,https://api.github.com/repos/simonw/datasette/issues/1557,996115949,IC_kwDOBm6k_c47X4Xt,9599,simonw,2021-12-16T19:30:55Z,2021-12-16T19:30:55Z,OWNER,"Demo: compare https://latest.datasette.io/fixtures/facetable?_facet=_city_id&_nosuggest=1 to https://latest.datasette.io/fixtures/facetable?_facet=_city_id Documentation: bottom of https://docs.datasette.io/en/latest/json_api.html#special-table-arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082564912,`?_nosuggest=1` parameter for disabling facet suggestions on table view, https://github.com/simonw/datasette/issues/625#issuecomment-996119954,https://api.github.com/repos/simonw/datasette/issues/625,996119954,IC_kwDOBm6k_c47X5WS,9599,simonw,2021-12-16T19:36:01Z,2021-12-16T19:36:11Z,OWNER,"Datasette's own HTML rendering code doesn't actually use the keys in `facet_results` - it instead loops through `sorted_facet_results` which is defined like this: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L937-L941 And used like this: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/templates/table.html#L154-L156","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996121736,https://api.github.com/repos/simonw/datasette/issues/625,996121736,IC_kwDOBm6k_c47X5yI,9599,simonw,2021-12-16T19:37:08Z,2021-12-16T19:37:08Z,OWNER,"Really `facet_results` here should be an array of objects, not an object that maps poorly designed string keys to those objects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1558#issuecomment-996134716,https://api.github.com/repos/simonw/datasette/issues/1558,996134716,IC_kwDOBm6k_c47X888,9599,simonw,2021-12-16T19:46:21Z,2021-12-16T19:46:21Z,OWNER,"The flaw in the current design is illustrated by this example: ``` ""facet_results"": { ""tags"": { ""name"": ""tags"", ""type"": ""array"", ""results"": [...], ""hideable"": false, ""toggle_url"": ""/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1"", ""truncated"": false }, ""created"": { ""name"": ""created"", ""type"": ""date"", ""results"": [...] ``` This was the cause of the bug in #625 - the each of those objects is keyed by the name of the column, which left no room for faceting the same column once by date and once by column value.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082584499,Redesign `facet_results` JSON structure prior to Datasette 1.0, https://github.com/simonw/datasette/issues/625#issuecomment-996130862,https://api.github.com/repos/simonw/datasette/issues/625,996130862,IC_kwDOBm6k_c47X8Au,9599,simonw,2021-12-16T19:44:48Z,2021-12-16T19:44:48Z,OWNER,"Decision: as an initial fix I'm going to de-duplicate those keys by using `tags__array` etc - with a `_2` on the end if that key is already used. I'll open a separate issue to redesign this better for Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996146762,https://api.github.com/repos/simonw/datasette/issues/625,996146762,IC_kwDOBm6k_c47X_5K,9599,simonw,2021-12-16T19:51:44Z,2021-12-16T19:51:44Z,OWNER,"Here's where `facet_results` is built up: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L752-L758 So the decision to key things based on column name is actually embedded deep in the existing facet classes here: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L224-L226 https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L395-L397 https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L510-L512","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1552#issuecomment-996034408,https://api.github.com/repos/simonw/datasette/issues/1552,996034408,IC_kwDOBm6k_c47Xkdo,9599,simonw,2021-12-16T17:37:37Z,2021-12-16T17:37:37Z,OWNER,"I think you're right! I had completely forgotten that piece of code. This just turned into a bug fix and a documentation update. Thanks for the research!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996045776,https://api.github.com/repos/simonw/datasette/issues/1552,996045776,IC_kwDOBm6k_c47XnPQ,9599,simonw,2021-12-16T17:52:54Z,2021-12-16T17:52:54Z,OWNER,"I tried that fix you suggested and now this `metadata.json` does the right thing: ```json { ""databases"": { ""fixtures"": { ""tables"": { ""facetable"": { ""facets"": [ { ""array"": ""tags"" } ] } } } } } ``` It does further highlight the bug in #625 though - since then if you try to add `?_facet=tags` to facet by tags treating them NOT as an array your request to do so is ignored.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996046304,https://api.github.com/repos/simonw/datasette/issues/1552,996046304,IC_kwDOBm6k_c47XnXg,9599,simonw,2021-12-16T17:53:40Z,2021-12-16T18:16:12Z,OWNER,"I'm also not convinced that this configuration syntax is right. It's a bit weird having a `""facets""` list that can either by column-name-strings or `{""type-of-facet"": ""column-name""}` objects. Maybe there's a better design for this? Part of the problem here is that facets were designed to accept optional extra configuration - partly to support `m2m` facets in #495 - but I haven't actually shipped any facets that use that ability. Facet by delimiter would be a good one to exercise that ability: - #510","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996077053,https://api.github.com/repos/simonw/datasette/issues/1552,996077053,IC_kwDOBm6k_c47Xu39,9599,simonw,2021-12-16T18:36:41Z,2021-12-16T18:36:41Z,OWNER,"... actually no, I WILL document this, because not documenting this is what got us to this point in the first place!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996076373,https://api.github.com/repos/simonw/datasette/issues/1552,996076373,IC_kwDOBm6k_c47XutV,9599,simonw,2021-12-16T18:35:40Z,2021-12-16T18:35:40Z,OWNER,"I'm going to ship your fix now, but I'm not going to add this to the documentation yet because I hope to improve the design prior to Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996084899,https://api.github.com/repos/simonw/datasette/issues/1552,996084899,IC_kwDOBm6k_c47Xwyj,9599,simonw,2021-12-16T18:48:14Z,2021-12-16T18:48:14Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/20a2ed6bec367d2f6759be4a879364a72780b59d/docs/facets.rst#facets-in-metadatajson,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/625#issuecomment-996093884,https://api.github.com/repos/simonw/datasette/issues/625,996093884,IC_kwDOBm6k_c47Xy-8,9599,simonw,2021-12-16T19:00:28Z,2021-12-16T19:00:28Z,OWNER,Implementing #1552 has made a fix for this bug even more important.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996150904,https://api.github.com/repos/simonw/datasette/issues/625,996150904,IC_kwDOBm6k_c47YA54,9599,simonw,2021-12-16T19:57:52Z,2021-12-16T19:57:52Z,OWNER,Good news - GitHub's new code search doesn't show ANYONE using that plugin hook - not surprising since it has that documentation warning plus it's just not a very clearly usable hook: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/830#issuecomment-996151246,https://api.github.com/repos/simonw/datasette/issues/830,996151246,IC_kwDOBm6k_c47YA_O,9599,simonw,2021-12-16T19:58:22Z,2021-12-16T19:58:22Z,OWNER,"As of today, 16 December 2021, I'm still not seeing any evidence that anyone is using this hook (yet) according to GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636511683,Redesign register_facet_classes plugin hook, https://github.com/simonw/datasette/issues/625#issuecomment-996149720,https://api.github.com/repos/simonw/datasette/issues/625,996149720,IC_kwDOBm6k_c47YAnY,9599,simonw,2021-12-16T19:56:14Z,2021-12-16T19:56:14Z,OWNER,"This bad design is even covered in the plugin hooks documentation: https://docs.datasette.io/en/0.59.4/plugin_hooks.html#register-facet-classes It does at least have the following warning: > **Warning** > > The design of this plugin hook is unstable and may change. See [issue 830](https://github.com/simonw/datasette/issues/830).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996152213,https://api.github.com/repos/simonw/datasette/issues/625,996152213,IC_kwDOBm6k_c47YBOV,9599,simonw,2021-12-16T19:59:46Z,2021-12-16T20:00:05Z,OWNER,"Since no-one is using that plugin hook I'm going to alter its contract slightly. I'll still keep the existing JSON format working though (until 1.0), since it's much more likely that people are using that JSON somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996161380,https://api.github.com/repos/simonw/datasette/issues/625,996161380,IC_kwDOBm6k_c47YDdk,9599,simonw,2021-12-16T20:13:05Z,2021-12-16T20:13:05Z,OWNER,I updated the example code in the facet plugin hook documentation: https://github.com/simonw/datasette/blob/95d0dd7a1cf6be6b7da41e1404184217eb93f64a/docs/plugin_hooks.rst#register_facet_classes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996165659,https://api.github.com/repos/simonw/datasette/issues/625,996165659,IC_kwDOBm6k_c47YEgb,9599,simonw,2021-12-16T20:19:53Z,2021-12-16T20:19:53Z,OWNER,Demo of the fix: https://latest.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags#facet-tags,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996170510,https://api.github.com/repos/simonw/datasette/issues/625,996170510,IC_kwDOBm6k_c47YFsO,9599,simonw,2021-12-16T20:27:41Z,2021-12-16T20:27:41Z,OWNER,"And here's the new JSON: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1 ``` { ""database"": ""fixtures"", ""table"": ""facetable"", ""is_view"": false, ""human_description_en"": """", ... ""facet_results"": { ""created"": { ""name"": ""created"", ""type"": ""column"", ... }, ""tags"": { ""name"": ""tags"", ""type"": ""column"", ... }, ""created_2"": { ""name"": ""created"", ""type"": ""date"", ... }, ""tags_2"": { ""name"": ""tags"", ""type"": ""array"", ... } } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1558#issuecomment-996204369,https://api.github.com/repos/simonw/datasette/issues/1558,996204369,IC_kwDOBm6k_c47YN9R,9599,simonw,2021-12-16T21:23:25Z,2021-12-16T21:23:25Z,OWNER,"Related: Following the fix for #625 I noticed that `facets_timed_out` gives you just the column name, but doesn't let you know which particular type of facet (`date` or `array` for example) suffered the timeout: https://github.com/simonw/datasette/blob/0d4145d0f4d8b2a7edc1ba4aac1be56cd536a10a/datasette/facets.py#L269-L270 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082584499,Redesign `facet_results` JSON structure prior to Datasette 1.0, https://github.com/simonw/datasette/issues/1518#issuecomment-996219117,https://api.github.com/repos/simonw/datasette/issues/1518,996219117,IC_kwDOBm6k_c47YRjt,9599,simonw,2021-12-16T21:47:51Z,2021-12-16T21:49:24Z,OWNER,"Should facets really not be displayed on pages past page one (where `?_next=` is set)? That made sense to me at the time, but I'm now having second thoughts about it. I guess it's a useful performance tweak for when crawlers keep hitting the `?_next=` link. Actually it looks like facets DO display on subsequent pages, e.g. on https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 - but facet suggestions do not, thanks to this code: https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/views/table.py#L777-L785 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996225235,https://api.github.com/repos/simonw/datasette/issues/1518,996225235,IC_kwDOBm6k_c47YTDT,9599,simonw,2021-12-16T21:58:24Z,2021-12-16T21:58:41Z,OWNER,"A fundamental operation of this view is to construct the SQL query and accompanying human description based on the incoming query string parameters. The human description is the bit at the top of https://latest.datasette.io/fixtures/searchable?_search=dog&_sort=pk&_facet=text2&text2=sara+weasel that says: > 1 row where search matches ""dog"" and text2 = ""sara weasel"" sorted by pk (Also used in the page ``). The code actually gathers three things: - Fragments of the `where` clause, for example ` ""text2"" = :p0` - Parameters, e.g. `{""p0"": ""sara weasel""}` - Human description components, e.g. `text2 = ""sara weasel""` Some operations such as `?_where=` don't currently provide an extra human description component. `_where=` also doesn't populate a parameter, but maybe it could? Would be neat if in the future `?_where=foo+=+:bar` worked and added a `bar` input field to the screen, as seen with custom queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996225889,https://api.github.com/repos/simonw/datasette/issues/1518,996225889,IC_kwDOBm6k_c47YTNh,9599,simonw,2021-12-16T21:59:32Z,2021-12-16T22:00:42Z,OWNER,I added a ton of comments to the `data()` method which really helps get a better feel for how this all works: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L322,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996227713,https://api.github.com/repos/simonw/datasette/issues/1518,996227713,IC_kwDOBm6k_c47YTqB,9599,simonw,2021-12-16T22:02:35Z,2021-12-16T22:03:55Z,OWNER,"Is there an opportunity to refactor things using a new plugin hook here? Maybe the `register_filters` hook from #473, where the hook becomes responsible for building where clauses (and human descriptions of them) based on the incoming query string. That version dealt with `Filter` classes, but those might be a bit too low-level for this. `?_spatial_within=GEOJSON` was an interesting idea attached to that issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996240802,https://api.github.com/repos/simonw/datasette/issues/1518,996240802,IC_kwDOBm6k_c47YW2i,9599,simonw,2021-12-16T22:25:00Z,2021-12-16T22:36:04Z,OWNER,"I think that plugin hook would get given the `request` object (and `datasette` and the name of the database and table) and returns a list of SQL fragments, a dictionary of lookup arguments and a list of human-description fragments - or an awaitable. `filters_from_request(request, database, table, datasette)` perhaps? (Similar in name to `actor_from_request`). ```python @hookspec def filters_from_request(request, database, table, datasette): """"""Return (where_clauses, params_dict, human_descriptions) based on the request"""""" ``` Turns out that's pretty much exactly what I implemented in 5116c4ec8aed5091e1f75415424b80f613518dc6 for #473: ```python @hookspec def table_filter(): ""Custom filtering of the current table based on the request"" ``` ```python TableFilter = namedtuple(""TableFilter"", ( ""human_description_extras"", ""where_clauses"", ""params"") ) ``` ```python # filter_arguments plugin hook support for awaitable_fn in pm.hook.table_filter(): extras = await awaitable_fn( view=self, name=name, table=table, request=request ) human_description_extras.extend(extras.human_description_extras) where_clauses.extend(extras.where_clauses) params.update(extras.params) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996248713,https://api.github.com/repos/simonw/datasette/issues/1518,996248713,IC_kwDOBm6k_c47YYyJ,9599,simonw,2021-12-16T22:39:47Z,2021-12-16T22:39:47Z,OWNER,"The hook could return a named tuple like this one: ```python from typing import NamedTuple, List, Optional, Union, Dict class FilterArguments(NamedTuple): where_clauses: List[str] params: Dict[str, Union[str, int, float]] human_descriptions: List[str] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996250585,https://api.github.com/repos/simonw/datasette/issues/1518,996250585,IC_kwDOBm6k_c47YZPZ,9599,simonw,2021-12-16T22:43:37Z,2021-12-16T22:45:07Z,OWNER,"Ran into a problem prototyping that hook up for handling `?_where=` - that feature also adds a little bit of extra template context in order to show the interface for removing wheres - the `extra_wheres_for_ui` variable: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L457-L463 Maybe change to this? ```python class FilterArguments(NamedTuple): where_clauses: List[str] params: Dict[str, Union[str, int, float]] human_descriptions: List[str] extra_context: Dict[str, Any] ``` That might be necessary for `_search` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996264617,https://api.github.com/repos/simonw/datasette/issues/1518,996264617,IC_kwDOBm6k_c47Ycqp,9599,simonw,2021-12-16T23:11:12Z,2021-12-16T23:11:12Z,OWNER,I managed to extract both `_search=` and `_where=` out using a prototype of that hook. I wonder if it could extract the complex code for `?_next` too?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/473#issuecomment-996267817,https://api.github.com/repos/simonw/datasette/issues/473,996267817,IC_kwDOBm6k_c47Ydcp,9599,simonw,2021-12-16T23:17:52Z,2021-12-16T23:19:00Z,OWNER,"I revisited this idea in #1518 and came up with a slightly different name and design for the hook: ```python @hookspec def filters_from_request(request, database, table, datasette): """""" Return FilterArguments( where_clauses=[str, str, str], params={}, human_descriptions=[str, str, str], extra_context={} ) based on the request"""""" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/1518#issuecomment-996272906,https://api.github.com/repos/simonw/datasette/issues/1518,996272906,IC_kwDOBm6k_c47YesK,9599,simonw,2021-12-16T23:27:42Z,2021-12-16T23:27:42Z,OWNER,Got a TIL out of this: https://til.simonwillison.net/pluggy/multiple-hooks-same-file,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/473#issuecomment-996275108,https://api.github.com/repos/simonw/datasette/issues/473,996275108,IC_kwDOBm6k_c47YfOk,9599,simonw,2021-12-16T23:32:22Z,2021-12-16T23:32:30Z,OWNER,This filter design can only influence the `where` component of the SQL clause - it's not able to modify the `SELECT` columns or adjust the `ORDER BY` or `OFFSET LIMIT` parts. I think that's OK.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/1518#issuecomment-996286104,https://api.github.com/repos/simonw/datasette/issues/1518,996286104,IC_kwDOBm6k_c47Yh6Y,9599,simonw,2021-12-17T00:00:07Z,2021-12-17T00:00:07Z,OWNER,Documentation of the new hook in the PR: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/473#issuecomment-996286199,https://api.github.com/repos/simonw/datasette/issues/473,996286199,IC_kwDOBm6k_c47Yh73,9599,simonw,2021-12-17T00:00:22Z,2021-12-17T00:00:22Z,OWNER,Documentation for that hook in the PR branch: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/pull/1559#issuecomment-996286808,https://api.github.com/repos/simonw/datasette/issues/1559,996286808,IC_kwDOBm6k_c47YiFY,9599,simonw,2021-12-17T00:01:43Z,2021-12-17T00:01:43Z,OWNER,"This already has tests and documentation, and I've used it to refactor out the logic for `?_where=` and `?_search=` and `?_through=`. Do I like this enough to land it on `main`? Also, I think I can still use it to refactor out the `Filters` code that implements `?col=x` and `?col__lt=5` and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/issues/473#issuecomment-996345233,https://api.github.com/repos/simonw/datasette/issues/473,996345233,IC_kwDOBm6k_c47YwWR,9599,simonw,2021-12-17T01:20:31Z,2021-12-17T18:13:01Z,OWNER,I could use this hook to add table filtering on a map to the existing `datasette-leaflet-freedraw` plugin.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/473#issuecomment-996484551,https://api.github.com/repos/simonw/datasette/issues/473,996484551,IC_kwDOBm6k_c47ZSXH,9599,simonw,2021-12-17T07:02:21Z,2021-12-17T07:04:23Z,OWNER,"The one slightly weird thing about this hook is how it adds `extra_context` without an obvious way for plugins to add extra HTML to the templates based on that context. Maybe I need the proposed mechanism from - #1191 Which has an in-progress PR: - #1204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/pull/1204#issuecomment-996488925,https://api.github.com/repos/simonw/datasette/issues/1204,996488925,IC_kwDOBm6k_c47ZTbd,9599,simonw,2021-12-17T07:10:48Z,2021-12-17T07:10:48Z,OWNER,I think this is missing the `_macro.html` template file but I have that in my Dropbox.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793002853,WIP: Plugin includes, https://github.com/simonw/datasette/pull/1559#issuecomment-996895423,https://api.github.com/repos/simonw/datasette/issues/1559,996895423,IC_kwDOBm6k_c47a2q_,9599,simonw,2021-12-17T17:28:44Z,2021-12-17T17:28:44Z,OWNER,"Before I land this I'm going to build one prototype plugin against it to confirm that the new hook is useful in its current shape. I'll add support for filtering a table by drawing on a map to https://datasette.io/plugins/datasette-leaflet-freedraw","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/issues/1568#issuecomment-997153253,https://api.github.com/repos/simonw/datasette/issues/1568,997153253,IC_kwDOBm6k_c47b1nl,9599,simonw,2021-12-18T06:20:23Z,2021-12-18T06:20:23Z,OWNER,Now running at https://latest-with-plugins.datasette.io/github/commits?_trace=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083726550,Trace should show queries on the write connection too, https://github.com/simonw/datasette/issues/473#issuecomment-996958442,https://api.github.com/repos/simonw/datasette/issues/473,996958442,IC_kwDOBm6k_c47bGDq,9599,simonw,2021-12-17T18:59:27Z,2021-12-17T18:59:27Z,OWNER,I'm happy with how the prototype that used this plugin in `datasette-leaflet-freedraw` turned out: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/pull/1559#issuecomment-996959325,https://api.github.com/repos/simonw/datasette/issues/1559,996959325,IC_kwDOBm6k_c47bGRd,9599,simonw,2021-12-17T18:59:54Z,2021-12-17T18:59:54Z,OWNER,I've convinced myself that this plugin hook design is good through this `datasette-leaflet-freedraw` prototype: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/pull/1559#issuecomment-996961196,https://api.github.com/repos/simonw/datasette/issues/1559,996961196,IC_kwDOBm6k_c47bGus,9599,simonw,2021-12-17T19:00:53Z,2021-12-17T19:00:53Z,OWNER,"I'm going to merge this to `main` now. I can continue the refactoring there, but having it in `main` means I can put out an alpha release with the new hook which will unblock me from running tests against it in this repo: https://github.com/simonw/datasette-leaflet-freedraw/pull/8","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/issues/1565#issuecomment-997069128,https://api.github.com/repos/simonw/datasette/issues/1565,997069128,IC_kwDOBm6k_c47bhFI,9599,simonw,2021-12-17T22:31:18Z,2021-12-17T22:31:18Z,OWNER,This should aim to be as consistent as possible with the various arguments to hooks on https://docs.datasette.io/en/stable/plugin_hooks.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/pull/1562#issuecomment-997080352,https://api.github.com/repos/simonw/datasette/issues/1562,997080352,IC_kwDOBm6k_c47bj0g,9599,simonw,2021-12-17T23:03:08Z,2021-12-17T23:03:08Z,OWNER,"They say they've dropped 3.6 support, but Datasette's tests against 3.6 are still passing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/issues/1565#issuecomment-997077410,https://api.github.com/repos/simonw/datasette/issues/1565,997077410,IC_kwDOBm6k_c47bjGi,9599,simonw,2021-12-17T22:54:45Z,2021-12-17T22:54:45Z,OWNER,"The table page should expose the query both with and without the `limit` clause. The above gave me back: ```sql select id, ACCESS_TYP, UNIT_ID, UNIT_NAME, SUID_NMA, AGNCY_ID, AGNCY_NAME, AGNCY_LEV, AGNCY_TYP, AGNCY_WEB, LAYER, MNG_AG_ID, MNG_AGENCY, MNG_AG_LEV, MNG_AG_TYP, PARK_URL, COUNTY, ACRES, LABEL_NAME, YR_EST, DES_TP, GAP_STS, geometry from CPAD_2020a_Units where ""AGNCY_LEV"" = :p0 order by id limit 101 ``` But I actually wanted to run a `fetch()` against a version of that without the `order by id limit 101` bit (I wanted to figure out the `Extent()` of the `geometry` column) - so I need something like `datasette.table_sql_no_order_no_limit`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1566#issuecomment-997078812,https://api.github.com/repos/simonw/datasette/issues/1566,997078812,IC_kwDOBm6k_c47bjcc,9599,simonw,2021-12-17T22:58:55Z,2021-12-17T22:58:55Z,OWNER,The release notes for the 0.60a0 alpha will be useful here: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/pull/1562#issuecomment-997081673,https://api.github.com/repos/simonw/datasette/issues/1562,997081673,IC_kwDOBm6k_c47bkJJ,9599,simonw,2021-12-17T23:06:38Z,2021-12-17T23:06:38Z,OWNER,"From this diff between `0.7.0` and `1.0`: https://github.com/aio-libs/janus/compare/v0.7.0...v1.0.0 It looks like the only change relevant to compatibility is `loop = asyncio.get_running_loop()` directly instead of falling back to `asyncio.get_event_loop()` if `get_running_loop` isn't available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/pull/1562#issuecomment-997082189,https://api.github.com/repos/simonw/datasette/issues/1562,997082189,IC_kwDOBm6k_c47bkRN,9599,simonw,2021-12-17T23:08:14Z,2021-12-17T23:08:14Z,OWNER,"Oh that makes sense: In Python 3.6 this happens: ``` Collecting janus<1.1,>=0.6.2 Using cached janus-0.7.0-py3-none-any.whl (6.9 kB) ``` While in Python 3.7 or higher this happens: ``` Collecting janus<1.1,>=0.6.2 Downloading janus-1.0.0-py3-none-any.whl (6.9 kB) ``` So this is safe to apply because `pip` is smart enough to pick the version of Janus that works for that Python version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/pull/1559#issuecomment-997082676,https://api.github.com/repos/simonw/datasette/issues/1559,997082676,IC_kwDOBm6k_c47bkY0,9599,simonw,2021-12-17T23:09:41Z,2021-12-17T23:09:41Z,OWNER,This is now available to try out in Datasette 0.60a0: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/issues/1518#issuecomment-997082845,https://api.github.com/repos/simonw/datasette/issues/1518,997082845,IC_kwDOBm6k_c47bkbd,9599,simonw,2021-12-17T23:10:09Z,2021-12-17T23:10:17Z,OWNER,These changes so far are now in the 0.60a0 alpha: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/621#issuecomment-997120723,https://api.github.com/repos/simonw/datasette/issues/621,997120723,IC_kwDOBm6k_c47btrT,9599,simonw,2021-12-18T01:42:33Z,2021-12-18T01:42:33Z,OWNER,I refactored this code out into the `filters.py` module in aa7f0037a46eb76ae6fe9bf2a1f616c58738ecdf,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/1565#issuecomment-997121215,https://api.github.com/repos/simonw/datasette/issues/1565,997121215,IC_kwDOBm6k_c47bty_,9599,simonw,2021-12-18T01:45:44Z,2021-12-18T01:45:44Z,OWNER,I want to get this into Datasette 0.60 - #1566 - it's a small change that can unlock a lot of potential.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1564#issuecomment-997122938,https://api.github.com/repos/simonw/datasette/issues/1564,997122938,IC_kwDOBm6k_c47buN6,9599,simonw,2021-12-18T01:55:25Z,2021-12-18T01:55:46Z,OWNER,"Made this change while working on this issue: - #1567 I'm going to write a test for this that uses that `sleep()` SQL function from c35b84a2aabe2f14aeacf6cda4110ae1e94d6059.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083581011,_prepare_connection not called on write connections, https://github.com/simonw/datasette/issues/1546#issuecomment-997124280,https://api.github.com/repos/simonw/datasette/issues/1546,997124280,IC_kwDOBm6k_c47bui4,9599,simonw,2021-12-18T02:05:16Z,2021-12-18T02:05:16Z,OWNER,"Sure - there are actually several levels to this. The code that creates connections to the database is this: https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/database.py#L72-L95 For files on disk, it does this: ```python # For read-only connections conn = sqlite3.connect( ""file:my.db?mode=ro"", uri=True, check_same_thread=False) # For connections that should be treated as immutable: conn = sqlite3.connect( ""file:my.db?immutable=1"", uri=True, check_same_thread=False) ``` For in-memory databases it runs this after the connection has been created: ```python conn.execute(""PRAGMA query_only=1"") ``` SQLite `PRAGMA` queries are treated as dangerous: someone could run `PRAGMA query_only=0` to turn that previous option off for example. So this function runs against any incoming SQL to verify that it looks like a `SELECT ...` and doesn't have anything like that in it. https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/utils/__init__.py#L195-L204 You can see the tests for that here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/tests/test_utils.py#L136-L170","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1076057610,validating the sql, https://github.com/simonw/datasette/issues/1563#issuecomment-997125191,https://api.github.com/repos/simonw/datasette/issues/1563,997125191,IC_kwDOBm6k_c47buxH,9599,simonw,2021-12-18T02:10:20Z,2021-12-18T02:10:20Z,OWNER,I should document the usage of this constructor in https://docs.datasette.io/en/stable/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083573206,Datasette(... files=) should not be a required argument, https://github.com/simonw/datasette/issues/1563#issuecomment-997127084,https://api.github.com/repos/simonw/datasette/issues/1563,997127084,IC_kwDOBm6k_c47bvOs,9599,simonw,2021-12-18T02:22:30Z,2021-12-18T02:22:30Z,OWNER,Docs here: https://docs.datasette.io/en/latest/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083573206,Datasette(... files=) should not be a required argument, https://github.com/simonw/datasette/issues/1561#issuecomment-997127784,https://api.github.com/repos/simonw/datasette/issues/1561,997127784,IC_kwDOBm6k_c47bvZo,9599,simonw,2021-12-18T02:27:56Z,2021-12-18T02:27:56Z,OWNER,"Oh that's an interesting solution, combining the hashes of all of the individual databases. I'm actually not a big fan of `hashed_url` mode - I implemented it right at the start of the project because it felt like a clever hack, and then ended up making it not-the-default a few years ago: - #418 - #419 - #421 I've since not found myself wanting to use it at all for any of my projects - which makes me nervous, because it means there's a pretty complex feature that I'm not using at all, so it's only really protected by the existing unit tests for it. What I'd really like to do is figure out how to have hashed URL mode work entirely as a plugin - then I could extract it from Datasette core entirely (which would simplify a bunch of stuff) but people who find the optimization useful would be able to access it. I'm not sure that the existing plugin hooks are robust enough to do that yet though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082765654,"add hash id to ""_memory"" url if hashed url mode is turned on and crossdb is also turned on", https://github.com/simonw/datasette/issues/1555#issuecomment-997128080,https://api.github.com/repos/simonw/datasette/issues/1555,997128080,IC_kwDOBm6k_c47bveQ,9599,simonw,2021-12-18T02:30:19Z,2021-12-18T02:30:19Z,OWNER,"I think all of these queries happen in one place - in the `populate_schema_tables()` function - so optimizing them might be localized to just that area of the code, which would be nice: https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L97-L183","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997128251,https://api.github.com/repos/simonw/datasette/issues/1555,997128251,IC_kwDOBm6k_c47bvg7,9599,simonw,2021-12-18T02:31:51Z,2021-12-18T02:31:51Z,OWNER,"I was thinking it might even be possible to convert this into a `insert into tables select from ...` query: https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L102-L112 But the `SELECT` runs against a separate database from the `INSERT INTO`, so I would have to setup a cross-database connection for this which feels a little too complicated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997128368,https://api.github.com/repos/simonw/datasette/issues/1555,997128368,IC_kwDOBm6k_c47bviw,9599,simonw,2021-12-18T02:32:43Z,2021-12-18T02:32:43Z,OWNER,I wonder why the `INSERT INTO` queries don't show up in that `?trace=1` view?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997128508,https://api.github.com/repos/simonw/datasette/issues/1555,997128508,IC_kwDOBm6k_c47bvk8,9599,simonw,2021-12-18T02:33:57Z,2021-12-18T02:33:57Z,OWNER,"Here's why - `trace` only applies to read, not write SQL operations: https://github.com/simonw/datasette/blob/7c8f8aa209e4ba7bf83976f8495d67c28fbfca24/datasette/database.py#L209-L211","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1568#issuecomment-997128950,https://api.github.com/repos/simonw/datasette/issues/1568,997128950,IC_kwDOBm6k_c47bvr2,9599,simonw,2021-12-18T02:38:01Z,2021-12-18T02:38:01Z,OWNER,"Prototype: ```diff diff --git a/datasette/database.py b/datasette/database.py index 0a0c104..468e936 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -99,7 +99,9 @@ class Database: with conn: return conn.execute(sql, params or []) - return await self.execute_write_fn(_inner, block=block) + with trace(""sql"", database=self.name, sql=sql.strip(), params=params): + results = await self.execute_write_fn(_inner, block=block) + return results async def execute_write_fn(self, fn, block=False): task_id = uuid.uuid5(uuid.NAMESPACE_DNS, ""datasette.io"") ``` <img width=""1292"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/146626249-cc8609f6-590c-49e3-abab-0cbc132916d1.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083726550,Trace should show queries on the write connection too, https://github.com/simonw/datasette/issues/1555#issuecomment-997234858,https://api.github.com/repos/simonw/datasette/issues/1555,997234858,IC_kwDOBm6k_c47cJiq,9599,simonw,2021-12-18T17:28:44Z,2021-12-18T17:28:44Z,OWNER,"Maybe it would be worth exploring attaching each DB in turn to the _internal connection in order to perform these queries faster. I'm a bit worried about leaks though: the internal database isn't meant to be visible, even temporarily attaching another DB to it could cause SQL queries against that DB to be able to access the internal data. So maybe instead the _internal connection gets to connect to the other DBs? There's a maximum of ten there I think, which is good for most but not all cases. But the cases with the most connected databases will see the worst performance!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997235086,https://api.github.com/repos/simonw/datasette/issues/1555,997235086,IC_kwDOBm6k_c47cJmO,9599,simonw,2021-12-18T17:30:13Z,2021-12-18T17:30:13Z,OWNER,"Now that trace sees write queries (#1568) it's clear that there is a whole lot more DB activity then I had realized: <img width=""1292"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/146626249-cc8609f6-590c-49e3-abab-0cbc132916d1.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1566#issuecomment-997235388,https://api.github.com/repos/simonw/datasette/issues/1566,997235388,IC_kwDOBm6k_c47cJq8,9599,simonw,2021-12-18T17:32:07Z,2021-12-18T17:32:07Z,OWNER,I can release a new version of `datasette-leaflet-freedraw` as soon as this is out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1555#issuecomment-997241645,https://api.github.com/repos/simonw/datasette/issues/1555,997241645,IC_kwDOBm6k_c47cLMt,9599,simonw,2021-12-18T18:12:26Z,2021-12-18T18:12:26Z,OWNER,"A simpler optimization would be just to turn all of those column and index reads into a single efficient UNION query against each database, then figure out the most efficient pattern to send them all as writes in one go as opposed to calling `.execute_write()` in a loop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997241969,https://api.github.com/repos/simonw/datasette/issues/1555,997241969,IC_kwDOBm6k_c47cLRx,9599,simonw,2021-12-18T18:13:04Z,2021-12-18T18:13:04Z,OWNER,Also: running all of those `CREATE TABLE IF NOT EXISTS` in a single call to `conn.executescript()` rather than as separate queries may speed things up too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997248364,https://api.github.com/repos/simonw/datasette/issues/1555,997248364,IC_kwDOBm6k_c47cM1s,9599,simonw,2021-12-18T18:20:10Z,2021-12-18T18:20:10Z,OWNER,"Idea: teach `execute_write` to accept an optional `executescript=True` parameter, like this: ```diff diff --git a/datasette/database.py b/datasette/database.py index 468e936..1a424f5 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -94,10 +94,14 @@ class Database: f""file:{self.path}{qs}"", uri=True, check_same_thread=False ) - async def execute_write(self, sql, params=None, block=False): + async def execute_write(self, sql, params=None, executescript=False, block=False): + assert not executescript and params, ""Cannot use params with executescript=True"" def _inner(conn): with conn: - return conn.execute(sql, params or []) + if executescript: + return conn.executescript(sql) + else: + return conn.execute(sql, params or []) with trace(""sql"", database=self.name, sql=sql.strip(), params=params): results = await self.execute_write_fn(_inner, block=block) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997245301,https://api.github.com/repos/simonw/datasette/issues/1555,997245301,IC_kwDOBm6k_c47cMF1,9599,simonw,2021-12-18T18:17:04Z,2021-12-18T18:17:04Z,OWNER,"One downside of `conn.executescript()` is that it won't be picked up by the tracing mechanism - in fact nothing that uses `await db.execute_write_fn(fn, block=True)` or `await db.execute_fn(fn, block=True)` gets picked up by tracing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1569#issuecomment-997249563,https://api.github.com/repos/simonw/datasette/issues/1569,997249563,IC_kwDOBm6k_c47cNIb,9599,simonw,2021-12-18T18:21:23Z,2021-12-18T18:21:23Z,OWNER,"Goal here is to gain the ability to use `conn.executescript()` and still have it show up in the tracer. https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.executescript","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083895395,"db.execute_write(..., executescript=True) parameter", https://github.com/simonw/datasette/issues/1555#issuecomment-997262475,https://api.github.com/repos/simonw/datasette/issues/1555,997262475,IC_kwDOBm6k_c47cQSL,9599,simonw,2021-12-18T18:34:18Z,2021-12-18T18:34:18Z,OWNER,"<img width=""1055"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/146652142-1c0bc34e-4a18-407d-bd59-28d565b631a6.png""> Using `executescript=True` that call now takes 1.89ms to create all of those tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997266100,https://api.github.com/repos/simonw/datasette/issues/1555,997266100,IC_kwDOBm6k_c47cRK0,9599,simonw,2021-12-18T18:40:02Z,2021-12-18T18:40:02Z,OWNER,The implementation of `cursor.executemany()` looks very efficient - it turns into a call to this C function with `multiple` set to `1`: https://github.com/python/cpython/blob/e002bbc6cce637171fb2b1391ffeca8643a13843/Modules/_sqlite/cursor.c#L468-L469,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1569#issuecomment-997266687,https://api.github.com/repos/simonw/datasette/issues/1569,997266687,IC_kwDOBm6k_c47cRT_,9599,simonw,2021-12-18T18:41:40Z,2021-12-18T18:41:40Z,OWNER,Updated documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-executescript-false-block-false,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083895395,"db.execute_write(..., executescript=True) parameter", https://github.com/simonw/datasette/issues/1555#issuecomment-997267416,https://api.github.com/repos/simonw/datasette/issues/1555,997267416,IC_kwDOBm6k_c47cRfY,9599,simonw,2021-12-18T18:44:53Z,2021-12-18T18:45:28Z,OWNER,"Rather than adding a `executemany=True` parameter, I'm now thinking a better design might be to have three methods: - `db.execute_write(sql, params=None, block=False)` - `db.execute_writescript(sql, block=False)` - `db.execute_writemany(sql, params_seq, block=False)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1570#issuecomment-997267583,https://api.github.com/repos/simonw/datasette/issues/1570,997267583,IC_kwDOBm6k_c47cRh_,9599,simonw,2021-12-18T18:46:05Z,2021-12-18T18:46:12Z,OWNER,This will replace the work done in #1569.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083921371,Separate db.execute_write() into three methods, https://github.com/simonw/datasette/issues/1555#issuecomment-997272223,https://api.github.com/repos/simonw/datasette/issues/1555,997272223,IC_kwDOBm6k_c47cSqf,9599,simonw,2021-12-18T19:17:13Z,2021-12-18T19:17:13Z,OWNER,That's a good optimization. Still need to deal with the huge flurry of `PRAGMA` queries though before I can consider this done.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1566#issuecomment-997272328,https://api.github.com/repos/simonw/datasette/issues/1566,997272328,IC_kwDOBm6k_c47cSsI,9599,simonw,2021-12-18T19:18:01Z,2021-12-18T19:18:01Z,OWNER,"Added some useful new documented internal methods in: - #1570","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1566#issuecomment-997457790,https://api.github.com/repos/simonw/datasette/issues/1566,997457790,IC_kwDOBm6k_c47c_9-,9599,simonw,2021-12-19T20:40:50Z,2021-12-19T20:40:57Z,OWNER,"Also release new version of `datasette-pretty-traces` with this feature: - https://github.com/simonw/datasette-pretty-traces/issues/7","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1555#issuecomment-997320824,https://api.github.com/repos/simonw/datasette/issues/1555,997320824,IC_kwDOBm6k_c47ceh4,9599,simonw,2021-12-19T02:59:57Z,2021-12-19T03:00:44Z,OWNER,"To list all indexes: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++index_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_index_list%28sqlite_master.name%29+AS+index_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, index_list.* FROM sqlite_master, pragma_index_list(sqlite_master.name) AS index_list WHERE sqlite_master.type = 'table' ``` Foreign keys: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++foreign_key_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_foreign_key_list%28sqlite_master.name%29+AS+foreign_key_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, foreign_key_list.* FROM sqlite_master, pragma_foreign_key_list(sqlite_master.name) AS foreign_key_list WHERE sqlite_master.type = 'table' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321115,https://api.github.com/repos/simonw/datasette/issues/1555,997321115,IC_kwDOBm6k_c47cemb,9599,simonw,2021-12-19T03:03:12Z,2021-12-19T03:03:12Z,OWNER,"Table columns is a bit harder, because `table_xinfo` is only in SQLite 3.26.0 or higher: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/utils/__init__.py#L565-L581 So if that function is available: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_xinfo.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_xinfo%28sqlite_master.name%29+AS+table_xinfo%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, table_xinfo.* FROM sqlite_master, pragma_table_xinfo(sqlite_master.name) AS table_xinfo WHERE sqlite_master.type = 'table' ``` And otherwise, using `table_info`: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_info.*%2C%0D%0A++0+as+hidden%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_info%28sqlite_master.name%29+AS+table_info%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, table_info.*, 0 as hidden FROM sqlite_master, pragma_table_info(sqlite_master.name) AS table_info WHERE sqlite_master.type = 'table' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321217,https://api.github.com/repos/simonw/datasette/issues/1555,997321217,IC_kwDOBm6k_c47ceoB,9599,simonw,2021-12-19T03:04:16Z,2021-12-19T03:04:16Z,OWNER,"One thing to watch out for though, from https://sqlite.org/pragma.html#pragfunc > The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321327,https://api.github.com/repos/simonw/datasette/issues/1555,997321327,IC_kwDOBm6k_c47cepv,9599,simonw,2021-12-19T03:05:39Z,2021-12-19T03:05:44Z,OWNER,"This caught me out once before in: - https://github.com/simonw/datasette/issues/1276 Turns out Glitch was running SQLite 3.11.0 from 2016-02-15.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321477,https://api.github.com/repos/simonw/datasette/issues/1555,997321477,IC_kwDOBm6k_c47cesF,9599,simonw,2021-12-19T03:07:33Z,2021-12-19T03:07:33Z,OWNER,"If I want to continue supporting SQLite prior to 3.16.0 (2017-01-02) I'll need this optimization to only kick in with versions that support table-valued PRAGMA functions, while keeping the old `PRAGMA foreign_key_list(table)` stuff working for those older versions. That's feasible, but it's a bit more work - and I need to make sure I have robust testing in place for SQLite 3.15.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321653,https://api.github.com/repos/simonw/datasette/issues/1555,997321653,IC_kwDOBm6k_c47ceu1,9599,simonw,2021-12-19T03:09:43Z,2021-12-19T03:09:43Z,OWNER,"On that same documentation page I just spotted this: > This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported. This makes me nervous to rely on pragma function optimizations in Datasette itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321767,https://api.github.com/repos/simonw/datasette/issues/1555,997321767,IC_kwDOBm6k_c47cewn,9599,simonw,2021-12-19T03:10:58Z,2021-12-19T03:10:58Z,OWNER,"I wonder how much overhead there is switching between the `async` event loop main code and the thread that runs the SQL queries. Would there be a performance boost if I gathered all of the column/index information in a single function run on the thread using `db.execute_fn()` I wonder? It would eliminate a bunch of switching between threads. Would be great to understand how much of an impact that would have.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997324156,https://api.github.com/repos/simonw/datasette/issues/1555,997324156,IC_kwDOBm6k_c47cfV8,9599,simonw,2021-12-19T03:40:05Z,2021-12-19T03:40:05Z,OWNER,"Using the prototype of this: - https://github.com/simonw/datasette-pretty-traces/issues/5 I'm seeing about 180ms spent running all of these queries on startup! ![CleanShot 2021-12-18 at 19 38 37@2x](https://user-images.githubusercontent.com/9599/146663045-46bda669-90de-474f-8870-345182725dc1.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997324666,https://api.github.com/repos/simonw/datasette/issues/1555,997324666,IC_kwDOBm6k_c47cfd6,9599,simonw,2021-12-19T03:47:51Z,2021-12-19T03:48:09Z,OWNER,"Here's a hacked together prototype of running all of that stuff inside a single function passed to `.execute_fn()`: ```diff diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index 95055d8..58f9982 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -1,4 +1,5 @@ import textwrap +from datasette.utils import table_column_details async def init_internal_db(db): @@ -70,49 +71,70 @@ async def populate_schema_tables(internal_db, db): ""DELETE FROM tables WHERE database_name = ?"", [database_name], block=True ) tables = (await db.execute(""select * from sqlite_master WHERE type = 'table'"")).rows - tables_to_insert = [] - columns_to_delete = [] - columns_to_insert = [] - foreign_keys_to_delete = [] - foreign_keys_to_insert = [] - indexes_to_delete = [] - indexes_to_insert = [] - for table in tables: - table_name = table[""name""] - tables_to_insert.append( - (database_name, table_name, table[""rootpage""], table[""sql""]) - ) - columns_to_delete.append((database_name, table_name)) - columns = await db.table_column_details(table_name) - columns_to_insert.extend( - { - **{""database_name"": database_name, ""table_name"": table_name}, - **column._asdict(), - } - for column in columns - ) - foreign_keys_to_delete.append((database_name, table_name)) - foreign_keys = ( - await db.execute(f""PRAGMA foreign_key_list([{table_name}])"") - ).rows - foreign_keys_to_insert.extend( - { - **{""database_name"": database_name, ""table_name"": table_name}, - **dict(foreign_key), - } - for foreign_key in foreign_keys - ) - indexes_to_delete.append((database_name, table_name)) - indexes = (await db.execute(f""PRAGMA index_list([{table_name}])"")).rows - indexes_to_insert.extend( - { - **{""database_name"": database_name, ""table_name"": table_name}, - **dict(index), - } - for index in indexes + def collect_info(conn): + tables_to_insert = [] + columns_to_delete = [] + columns_to_insert = [] + foreign_keys_to_delete = [] + foreign_keys_to_insert = [] + indexes_to_delete = [] + indexes_to_insert = [] + + for table in tables: + table_name = table[""name""] + tables_to_insert.append( + (database_name, table_name, table[""rootpage""], table[""sql""]) + ) + columns_to_delete.append((database_name, table_name)) + columns = table_column_details(conn, table_name) + columns_to_insert.extend( + { + **{""database_name"": database_name, ""table_name"": table_name}, + **column._asdict(), + } + for column in columns + ) + foreign_keys_to_delete.append((database_name, table_name)) + foreign_keys = conn.execute( + f""PRAGMA foreign_key_list([{table_name}])"" + ).fetchall() + foreign_keys_to_insert.extend( + { + **{""database_name"": database_name, ""table_name"": table_name}, + **dict(foreign_key), + } + for foreign_key in foreign_keys + ) + indexes_to_delete.append((database_name, table_name)) + indexes = conn.execute(f""PRAGMA index_list([{table_name}])"").fetchall() + indexes_to_insert.extend( + { + **{""database_name"": database_name, ""table_name"": table_name}, + **dict(index), + } + for index in indexes + ) + return ( + tables_to_insert, + columns_to_delete, + columns_to_insert, + foreign_keys_to_delete, + foreign_keys_to_insert, + indexes_to_delete, + indexes_to_insert, ) + ( + tables_to_insert, + columns_to_delete, + columns_to_insert, + foreign_keys_to_delete, + foreign_keys_to_insert, + indexes_to_delete, + indexes_to_insert, + ) = await db.execute_fn(collect_info) + await internal_db.execute_write_many( """""" INSERT INTO tables (database_name, table_name, rootpage, sql) ``` First impressions: it looks like this helps **a lot** - as far as I can tell this is now taking around 21ms to get to the point at which all of those internal databases have been populated, where previously it took more than 180ms. ![CleanShot 2021-12-18 at 19 47 22@2x](https://user-images.githubusercontent.com/9599/146663192-bba098d5-e7bd-4e2e-b525-2270867888a0.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997325189,https://api.github.com/repos/simonw/datasette/issues/1555,997325189,IC_kwDOBm6k_c47cfmF,9599,simonw,2021-12-19T03:55:01Z,2021-12-19T20:54:51Z,OWNER,"It's a bit annoying that the queries no longer show up in the trace at all now, thanks to running in `.execute_fn()`. I wonder if there's something smart I can do about that - maybe have `trace()` record that function with a traceback even though it doesn't have the executed SQL string? 5fac26aa221a111d7633f2dd92014641f7c0ade9 has the same problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997342494,https://api.github.com/repos/simonw/datasette/issues/1555,997342494,IC_kwDOBm6k_c47cj0e,9599,simonw,2021-12-19T07:22:04Z,2021-12-19T07:22:04Z,OWNER,"Another option would be to provide an abstraction that makes it easier to run a group of SQL queries in the same thread at the same time, and have them traced correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997459637,https://api.github.com/repos/simonw/datasette/issues/1555,997459637,IC_kwDOBm6k_c47dAa1,9599,simonw,2021-12-19T20:53:46Z,2021-12-19T20:53:46Z,OWNER,Using #1571 showed me that the `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ? and table_name = ?` queries were running way more times than I expected. I came up with a new optimization that just does `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ?` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997459958,https://api.github.com/repos/simonw/datasette/issues/1555,997459958,IC_kwDOBm6k_c47dAf2,9599,simonw,2021-12-19T20:55:59Z,2021-12-19T20:55:59Z,OWNER,"Closing this issue because I've optimized this a whole bunch, and it's definitely good enough for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1570#issuecomment-997460061,https://api.github.com/repos/simonw/datasette/issues/1570,997460061,IC_kwDOBm6k_c47dAhd,9599,simonw,2021-12-19T20:56:54Z,2021-12-19T20:56:54Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083921371,Separate db.execute_write() into three methods, https://github.com/simonw/datasette/issues/1547#issuecomment-997460731,https://api.github.com/repos/simonw/datasette/issues/1547,997460731,IC_kwDOBm6k_c47dAr7,9599,simonw,2021-12-19T21:02:15Z,2021-12-19T21:02:15Z,OWNER,"Yes, this is a bug. It looks like the problem is with the `if write:` branch in this code here: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L252-L327 Is missing this bit of code: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L343-L347","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1573#issuecomment-997462117,https://api.github.com/repos/simonw/datasette/issues/1573,997462117,IC_kwDOBm6k_c47dBBl,9599,simonw,2021-12-19T21:13:13Z,2021-12-19T21:13:13Z,OWNER,This might also be the impetus I need to bring the https://datasette.io/plugins/datasette-pretty-traces plugin into Datasette core itself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084185188,Make trace() a documented internal API, https://github.com/simonw/datasette/issues/1545#issuecomment-997462604,https://api.github.com/repos/simonw/datasette/issues/1545,997462604,IC_kwDOBm6k_c47dBJM,9599,simonw,2021-12-19T21:17:08Z,2021-12-19T21:17:08Z,OWNER,"Here's the relevant code: https://github.com/simonw/datasette/blob/4094741c2881c2ada3f3f878b532fdaec7914953/datasette/app.py#L1204-L1219 It's using `route_path.split(""/"")` which should be OK because that's the incoming `request.path` path - which I would expect to use `/` even on Windows. Then it uses `os.path.join` which should do the right thing. I need to get myself a proper Windows development environment setup to investigate this one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249,Custom pages don't work on windows, https://github.com/simonw/datasette/issues/1565#issuecomment-997473856,https://api.github.com/repos/simonw/datasette/issues/1565,997473856,IC_kwDOBm6k_c47dD5A,9599,simonw,2021-12-19T22:35:20Z,2021-12-19T22:35:20Z,OWNER,"Quick prototype of that tagged template `query` function: ```javascript function query(pieces, ...parameters) { var qs = new URLSearchParams(); var sql = pieces[0]; parameters.forEach((param, i) => { sql += `:p${i}${pieces[i + 1]}`; qs.append(`p${i}`, param); }); qs.append(""sql"", sql); return qs.toString(); } var id = 4; console.log(query`select * from ids where id > ${id}`); ``` Outputs: ``` p0=4&sql=select+*+from+ids+where+id+%3E+%3Ap0 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997474022,https://api.github.com/repos/simonw/datasette/issues/1565,997474022,IC_kwDOBm6k_c47dD7m,9599,simonw,2021-12-19T22:36:49Z,2021-12-19T22:37:29Z,OWNER,"No way with a tagged template literal to pass an extra database name argument, so instead I need a method that returns a callable that can be used for the tagged template literal for a specific database - or the default database. This could work (bit weird looking though): ```javascript var rows = await datasette.query(""fixtures"")`select * from foo`; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1566#issuecomment-997470633,https://api.github.com/repos/simonw/datasette/issues/1566,997470633,IC_kwDOBm6k_c47dDGp,9599,simonw,2021-12-19T22:12:00Z,2021-12-19T22:12:00Z,OWNER,"Released another alpha, 0.60a1: https://github.com/simonw/datasette/releases/tag/0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1547#issuecomment-997471672,https://api.github.com/repos/simonw/datasette/issues/1547,997471672,IC_kwDOBm6k_c47dDW4,9599,simonw,2021-12-19T22:18:26Z,2021-12-19T22:18:26Z,OWNER,"I released this [in an alpha](https://github.com/simonw/datasette/releases/tag/0.60a1), so you can try out this fix using: pip install datasette==0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1518#issuecomment-997472214,https://api.github.com/repos/simonw/datasette/issues/1518,997472214,IC_kwDOBm6k_c47dDfW,9599,simonw,2021-12-19T22:22:08Z,2021-12-19T22:22:08Z,OWNER,"I sketched out a chained SQL builder pattern that might be useful for further tidying up this code - though with the new plugin hook I'm less excited about it than I was: ```python class TableQuery: def __init__(self, table, columns, pks, is_view=False, prev=None): self.table = table self.columns = columns self.pks = pks self.is_view = is_view self.prev = prev # These can be changed for different instances in the chain: self._where_clauses = None self._order_by = None self._page_size = None self._offset = None self._select_columns = None self.select_all_columns = '*' self.select_specified_columns = '*' @property def where_clauses(self): wheres = [] current = self while current: if current._where_clauses is not None: wheres.extend(current._where_clauses) current = current.prev return list(reversed(wheres)) def where(self, where): new_cls = TableQuery(self.table, self.columns, self.pks, self.is_view, self) new_cls._where_clauses = [where] return new_cls @classmethod async def introspect(cls, db, table): return cls( table, columns = await db.table_columns(table), pks = await db.primary_keys(table), is_view = bool(await db.get_view_definition(table)) ) @property def sql_from(self): return f""from {self.table}{self.sql_where}"" @property def sql_where(self): if not self.where_clauses: return """" else: return f"" where {' and '.join(self.where_clauses)}"" @property def sql_no_order_no_limit(self): return f""select {self.select_all_columns} from {self.table}{self.sql_where}"" @property def sql(self): return f""select {self.select_specified_columns} from {self.table} {self.sql_where}{self._order_by} limit {self._page_size}{self._offset}"" @property def sql_count(self): return f""select count(*) {self.sql_from}"" def __repr__(self): return f""<TableQuery sql={self.sql}>"" ``` Usage: ```python from datasette.app import Datasette ds = Datasette(memory=True, files=[""/Users/simon/Dropbox/Development/datasette/fixtures.db""]) db = ds.get_database(""fixtures"") query = await TableQuery.introspect(db, ""facetable"") print(query.where(""foo = bar"").where(""baz = 1"").sql_count) # 'select count(*) from facetable where foo = bar and baz = 1' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1565#issuecomment-997472370,https://api.github.com/repos/simonw/datasette/issues/1565,997472370,IC_kwDOBm6k_c47dDhy,9599,simonw,2021-12-19T22:23:36Z,2021-12-19T22:23:36Z,OWNER,This should also expose the JSON API endpoints used to execute SQL against this database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997472509,https://api.github.com/repos/simonw/datasette/issues/1565,997472509,IC_kwDOBm6k_c47dDj9,9599,simonw,2021-12-19T22:24:50Z,2021-12-19T22:24:50Z,OWNER,"... huh, it could even expose a JavaScript function that can be called to execute a SQL query. ```javascript datasette.query(""select * from blah"").then(...) ``` Maybe it takes an optional second argument that specifies the database - defaulting to the one for the current page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997472639,https://api.github.com/repos/simonw/datasette/issues/1565,997472639,IC_kwDOBm6k_c47dDl_,9599,simonw,2021-12-19T22:25:50Z,2021-12-19T22:25:50Z,OWNER,"Or... ```javascript rows = await datasette.query`select * from searchable where id > ${id}`; ``` And it knows how to turn that into a parameterized call using tagged template literals.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1575#issuecomment-997513177,https://api.github.com/repos/simonw/datasette/issues/1575,997513177,IC_kwDOBm6k_c47dNfZ,9599,simonw,2021-12-20T01:24:25Z,2021-12-20T01:24:25Z,OWNER,Looks like `specname` is new in Pluggy 1.0: https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst#pluggy-100-2021-08-25,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084257842,__call__() got an unexpected keyword argument 'specname', https://github.com/simonw/datasette/issues/1547#issuecomment-997513369,https://api.github.com/repos/simonw/datasette/issues/1547,997513369,IC_kwDOBm6k_c47dNiZ,9599,simonw,2021-12-20T01:24:43Z,2021-12-20T01:24:43Z,OWNER,"@wragge thanks, that's a bug! Working on that in #1575.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1547#issuecomment-997514220,https://api.github.com/repos/simonw/datasette/issues/1547,997514220,IC_kwDOBm6k_c47dNvs,9599,simonw,2021-12-20T01:26:25Z,2021-12-20T01:26:25Z,OWNER,"OK, this should hopefully fix that for you: pip install https://github.com/simonw/datasette/archive/f36e010b3b69ada104b79d83c7685caf9359049e.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/pull/1554#issuecomment-998354538,https://api.github.com/repos/simonw/datasette/issues/1554,998354538,IC_kwDOBm6k_c47ga5q,9599,simonw,2021-12-20T23:52:04Z,2021-12-20T23:52:04Z,OWNER,Abandoning this since it didn't work how I wanted.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079129258,TableView refactor, https://github.com/simonw/datasette/issues/1518#issuecomment-999831967,https://api.github.com/repos/simonw/datasette/issues/1518,999831967,IC_kwDOBm6k_c47mDmf,9599,simonw,2021-12-22T20:04:47Z,2021-12-22T20:10:11Z,OWNER,"I think I might be able to clean up a lot of the stuff in here using the `render_cell` plugin hook: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L87-L89 The catch with that hook - https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette - is that it gets called for every single cell. I don't want the overhead of looking up the foreign key relationships etc once for every value in a specific column. But maybe I could extend the hook to include a shared cache that gets used for all of the cells in a specific table? Something like this: ```python render_cell(value, column, table, database, datasette, cache) ``` `cache` is a dictionary - and the same dictionary is passed to every call to that hook while rendering a specific page. It's a bit of a gross hack though, and would it ever be useful for plugins outside of the default plugin in Datasette which does the foreign key stuff? If I can think of one other potential application for this `cache` then I might implement it. No, this optimization doesn't make sense: the most complex cell enrichment logic is the stuff that does a `select * from categories where id in (2, 5, 6)` query, using just the distinct set of IDs that are rendered on the current page. That's not going to fit in the `render_cell` hook no matter how hard I try to warp it into the right shape, because it needs full visibility of all of the results that are being rendered in order to collect those unique ID values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999837569,https://api.github.com/repos/simonw/datasette/issues/1518,999837569,IC_kwDOBm6k_c47mE-B,9599,simonw,2021-12-22T20:15:45Z,2021-12-22T20:15:45Z,OWNER,"Also the whole `special_args` v.s. `request.args` thing is pretty confusing, I think that might be an older code pattern back from when I was using Sanic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999837220,https://api.github.com/repos/simonw/datasette/issues/1518,999837220,IC_kwDOBm6k_c47mE4k,9599,simonw,2021-12-22T20:15:04Z,2021-12-22T20:15:04Z,OWNER,"I think I can move this much higher up in the method, it's a bit confusing having it half way through: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L414-L436","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999850191,https://api.github.com/repos/simonw/datasette/issues/1518,999850191,IC_kwDOBm6k_c47mIDP,9599,simonw,2021-12-22T20:29:38Z,2021-12-22T20:29:38Z,OWNER,New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999863269,https://api.github.com/repos/simonw/datasette/issues/1518,999863269,IC_kwDOBm6k_c47mLPl,9599,simonw,2021-12-22T20:35:41Z,2021-12-22T20:37:13Z,OWNER,"It looks like the count has to be executed before facets can be, because the facet_class constructor needs that total count figure: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L660-L671 It's used in facet suggestion logic here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/facets.py#L172-L178","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999870282,https://api.github.com/repos/simonw/datasette/issues/1518,999870282,IC_kwDOBm6k_c47mM9K,9599,simonw,2021-12-22T20:45:56Z,2021-12-22T20:46:08Z,OWNER,"> New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`. I wrote code to execute those in parallel using `asyncio.gather()` - which seems to work but causes the SQL run inside the parallel `async def` functions not to show up in the trace graph at all. ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 9808fd2..ec9db64 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import urllib import itertools import json @@ -615,44 +616,37 @@ class TableView(RowTableShared): if request.args.get(""_timelimit""): extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit"")) - # Execute the main query! - results = await db.execute(sql, params, truncate=True, **extra_args) - - # Calculate the total count for this query - filtered_table_rows_count = None - if ( - not db.is_mutable - and self.ds.inspect_data - and count_sql == f""select count(*) from {table} "" - ): - # We can use a previously cached table row count - try: - filtered_table_rows_count = self.ds.inspect_data[database][""tables""][ - table - ][""count""] - except KeyError: - pass - - # Otherwise run a select count(*) ... - if count_sql and filtered_table_rows_count is None and not nocount: - try: - count_rows = list(await db.execute(count_sql, from_sql_params)) - filtered_table_rows_count = count_rows[0][0] - except QueryInterrupted: - pass - - # Faceting - if not self.ds.setting(""allow_facet"") and any( - arg.startswith(""_facet"") for arg in request.args - ): - raise BadRequest(""_facet= is not allowed"") + async def execute_count(): + # Calculate the total count for this query + filtered_table_rows_count = None + if ( + not db.is_mutable + and self.ds.inspect_data + and count_sql == f""select count(*) from {table} "" + ): + # We can use a previously cached table row count + try: + filtered_table_rows_count = self.ds.inspect_data[database][ + ""tables"" + ][table][""count""] + except KeyError: + pass + + if count_sql and filtered_table_rows_count is None and not nocount: + try: + count_rows = list(await db.execute(count_sql, from_sql_params)) + filtered_table_rows_count = count_rows[0][0] + except QueryInterrupted: + pass + + return filtered_table_rows_count + + filtered_table_rows_count = await execute_count() # pylint: disable=no-member facet_classes = list( itertools.chain.from_iterable(pm.hook.register_facet_classes()) ) - facet_results = {} - facets_timed_out = [] facet_instances = [] for klass in facet_classes: facet_instances.append( @@ -668,33 +662,58 @@ class TableView(RowTableShared): ) ) - if not nofacet: - for facet in facet_instances: - ( - instance_facet_results, - instance_facets_timed_out, - ) = await facet.facet_results() - for facet_info in instance_facet_results: - base_key = facet_info[""name""] - key = base_key - i = 1 - while key in facet_results: - i += 1 - key = f""{base_key}_{i}"" - facet_results[key] = facet_info - facets_timed_out.extend(instance_facets_timed_out) - - # Calculate suggested facets - suggested_facets = [] - if ( - self.ds.setting(""suggest_facets"") - and self.ds.setting(""allow_facet"") - and not _next - and not nofacet - and not nosuggest - ): - for facet in facet_instances: - suggested_facets.extend(await facet.suggest()) + async def execute_suggested_facets(): + # Calculate suggested facets + suggested_facets = [] + if ( + self.ds.setting(""suggest_facets"") + and self.ds.setting(""allow_facet"") + and not _next + and not nofacet + and not nosuggest + ): + for facet in facet_instances: + suggested_facets.extend(await facet.suggest()) + return suggested_facets + + async def execute_facets(): + facet_results = {} + facets_timed_out = [] + if not self.ds.setting(""allow_facet"") and any( + arg.startswith(""_facet"") for arg in request.args + ): + raise BadRequest(""_facet= is not allowed"") + + if not nofacet: + for facet in facet_instances: + ( + instance_facet_results, + instance_facets_timed_out, + ) = await facet.facet_results() + for facet_info in instance_facet_results: + base_key = facet_info[""name""] + key = base_key + i = 1 + while key in facet_results: + i += 1 + key = f""{base_key}_{i}"" + facet_results[key] = facet_info + facets_timed_out.extend(instance_facets_timed_out) + + return facet_results, facets_timed_out + + # Execute the main query, facets and facet suggestions in parallel: + ( + results, + suggested_facets, + (facet_results, facets_timed_out), + ) = await asyncio.gather( + db.execute(sql, params, truncate=True, **extra_args), + execute_suggested_facets(), + execute_facets(), + ) + + results = await db.execute(sql, params, truncate=True, **extra_args) # Figure out columns and rows for the query columns = [r[0] for r in results.description] ``` Here's the trace for `http://127.0.0.1:4422/fixtures/compound_three_primary_keys?_trace=1&_facet=pk1&_facet=pk2` with the missing facet and facet suggestion queries: <img width=""1447"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/147153051-62cdb9a5-de5e-4cc3-9215-b779f92a81c8.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1576#issuecomment-999874484,https://api.github.com/repos/simonw/datasette/issues/1576,999874484,IC_kwDOBm6k_c47mN-0,9599,simonw,2021-12-22T20:54:52Z,2021-12-22T20:54:52Z,OWNER,"Here's the full current relevant code from `tracer.py`: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L8-L64 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1518#issuecomment-999870993,https://api.github.com/repos/simonw/datasette/issues/1518,999870993,IC_kwDOBm6k_c47mNIR,9599,simonw,2021-12-22T20:47:18Z,2021-12-22T20:50:24Z,OWNER,"The reason they aren't showing up in the traces is that traces are stored just for the currently executing `asyncio` task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25 This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively ""child tasks"" of the current request, and hence should be tracked the same. https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using `asyncio.gather()` to work around this kind of problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1576#issuecomment-999874886,https://api.github.com/repos/simonw/datasette/issues/1576,999874886,IC_kwDOBm6k_c47mOFG,9599,simonw,2021-12-22T20:55:42Z,2021-12-22T20:57:28Z,OWNER,"One way to solve this would be to introduce a `set_task_id()` method, which sets an ID which will be returned by `get_task_id()` instead of using `id(current_task(loop=loop))`. It would be really nice if I could solve this using `with` syntax somehow. Something like: ```python with trace_child_tasks(): ( suggested_facets, (facet_results, facets_timed_out), ) = await asyncio.gather( execute_suggested_facets(), execute_facets(), ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999876666,https://api.github.com/repos/simonw/datasette/issues/1576,999876666,IC_kwDOBm6k_c47mOg6,9599,simonw,2021-12-22T20:59:22Z,2021-12-22T21:18:09Z,OWNER,"This article is relevant: [Context information storage for asyncio](https://blog.sqreen.com/asyncio/) - in particular the section https://blog.sqreen.com/asyncio/#context-inheritance-between-tasks which describes exactly the problem I have and their solution, which involves this trickery: ```python def request_task_factory(loop, coro): child_task = asyncio.tasks.Task(coro, loop=loop) parent_task = asyncio.Task.current_task(loop=loop) current_request = getattr(parent_task, 'current_request', None) setattr(child_task, 'current_request', current_request) return child_task loop = asyncio.get_event_loop() loop.set_task_factory(request_task_factory) ``` They released their solution as a library: https://pypi.org/project/aiocontext/ and https://github.com/sqreen/AioContext - but that company was acquired by Datadog back in April and doesn't seem to be actively maintaining their open source stuff any more: https://twitter.com/SqreenIO/status/1384906075506364417","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999878907,https://api.github.com/repos/simonw/datasette/issues/1576,999878907,IC_kwDOBm6k_c47mPD7,9599,simonw,2021-12-22T21:03:49Z,2021-12-22T21:10:46Z,OWNER,"`context_vars` can solve this but they were introduced in Python 3.7: https://www.python.org/dev/peps/pep-0567/ Python 3.6 support ends in a few days time, and it looks like Glitch has updated to 3.7 now - so maybe I can get away with Datasette needing 3.7 these days? Tweeted about that here: https://twitter.com/simonw/status/1473761478155010048","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999987418,https://api.github.com/repos/simonw/datasette/issues/1576,999987418,IC_kwDOBm6k_c47mpja,9599,simonw,2021-12-23T01:59:58Z,2021-12-23T02:02:12Z,OWNER,"Another option: https://github.com/Skyscanner/aiotask-context - looks like it might be better as it's been updated for Python 3.7 in this commit https://github.com/Skyscanner/aiotask-context/commit/67108c91d2abb445655cc2af446fdb52ca7890c4 The Skyscanner one doesn't attempt to wrap any existing factories, but that's OK for my purposes since I don't need to handle arbitrary `asyncio` code written by other people.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999990414,https://api.github.com/repos/simonw/datasette/issues/1576,999990414,IC_kwDOBm6k_c47mqSO,9599,simonw,2021-12-23T02:08:39Z,2021-12-23T18:16:35Z,OWNER,"It's tiny: I'm tempted to vendor it. https://github.com/Skyscanner/aiotask-context/blob/master/aiotask_context/__init__.py No, I'll add it as a pinned dependency, which I can then drop when I drop 3.6 support.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1577#issuecomment-1000461275,https://api.github.com/repos/simonw/datasette/issues/1577,1000461275,IC_kwDOBm6k_c47odPb,9599,simonw,2021-12-23T18:18:11Z,2021-12-23T18:18:11Z,OWNER,"From the Twitter thread, there are still a decent amount of LTS Linux releases out there that are stuck on pre-3.7 Python. Though many of those are 3.5 and Datasette dropped support for 3.5 in November 2019: cf7776d36fbacefa874cbd6e5fcdc9fff7661203","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1577#issuecomment-1000461900,https://api.github.com/repos/simonw/datasette/issues/1577,1000461900,IC_kwDOBm6k_c47odZM,9599,simonw,2021-12-23T18:19:44Z,2021-12-23T18:19:44Z,OWNER,"The 3.7 feature I want to use today is [contextvars](https://docs.python.org/3/library/contextvars.html) - but I have a workaround for the moment, see https://github.com/simonw/datasette/issues/1576#issuecomment-999987418 So I'm going to hold off on dropping 3.6 for a little bit longer. I imagine I'll drop it before Datasette 1.0 though. Leaving this issue open to gather thoughts and feedback on this issue from Datasette users and potential users.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1577#issuecomment-1000462309,https://api.github.com/repos/simonw/datasette/issues/1577,1000462309,IC_kwDOBm6k_c47odfl,9599,simonw,2021-12-23T18:20:46Z,2021-12-23T18:20:46Z,OWNER,There are a lot of improvements to `asyncio` in 3.7: https://docs.python.org/3/whatsnew/3.7.html#whatsnew37-asyncio,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1578#issuecomment-1000469107,https://api.github.com/repos/simonw/datasette/issues/1578,1000469107,IC_kwDOBm6k_c47ofJz,9599,simonw,2021-12-23T18:36:38Z,2021-12-23T18:36:38Z,OWNER,"This problem doesn't occur on my `localhost` running Uvicorn directly - but I'm seeing it in my production environment that runs Datasette behind an nginx proxy: ``` location / { proxy_pass http://127.0.0.1:8000/; proxy_set_header Host $host; } ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1578#issuecomment-1000470652,https://api.github.com/repos/simonw/datasette/issues/1578,1000470652,IC_kwDOBm6k_c47ofh8,9599,simonw,2021-12-23T18:40:46Z,2021-12-23T18:40:46Z,OWNER,"[This StackOverflow answer](https://serverfault.com/a/463932) suggests that the fix is to change this: proxy_pass http://127.0.0.1:8000/; To this: proxy_pass http://127.0.0.1:8000; Quoting the nginx documentation: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass > A request URI is passed to the server as follows: > > - If the `proxy_pass` directive is specified with a URI, then when a request is passed to the server, the part of a [normalized](http://nginx.org/en/docs/http/ngx_http_core_module.html#location) request URI matching the location is replaced by a URI specified in the directive: > > location /name/ { > proxy_pass http://127.0.0.1/remote/; > } > > - If `proxy_pass` is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI: > > location /some/path/ { > proxy_pass http://127.0.0.1; > }","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1578#issuecomment-1000471371,https://api.github.com/repos/simonw/datasette/issues/1578,1000471371,IC_kwDOBm6k_c47oftL,9599,simonw,2021-12-23T18:42:50Z,2021-12-23T18:42:50Z,OWNER,"Confirmed, that fixed the bug for me on my server.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1578#issuecomment-1000471782,https://api.github.com/repos/simonw/datasette/issues/1578,1000471782,IC_kwDOBm6k_c47ofzm,9599,simonw,2021-12-23T18:44:01Z,2021-12-23T18:44:01Z,OWNER,"The example nginx config on https://docs.datasette.io/en/stable/deploying.html#nginx-proxy-configuration is currently: ``` daemon off; events { worker_connections 1024; } http { server { listen 80; location /my-datasette { proxy_pass http://127.0.0.1:8009/my-datasette; proxy_set_header Host $host; } } } ``` This looks to me like it might exhibit the bug. Need to confirm that and figure out an alternative.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1579#issuecomment-1000476413,https://api.github.com/repos/simonw/datasette/issues/1579,1000476413,IC_kwDOBm6k_c47og79,9599,simonw,2021-12-23T18:56:06Z,2021-12-23T18:56:06Z,OWNER,"This is technically a breaking change, but a GitHub code search at https://cs.github.com/?scopeName=All+repos&scope=&q=execute_write%20datasette%20-owner%3Asimonw shows only one repo not-owned-by-me using this, and they're using `block=True`: https://github.com/mfa/datasette-webhook-write/blob/e82440f372a2f2e3ed27d1bd34c9fa3a53b49b94/datasette_webhook_write/__init__.py#L88-L89","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000477621,https://api.github.com/repos/simonw/datasette/issues/1579,1000477621,IC_kwDOBm6k_c47ohO1,9599,simonw,2021-12-23T18:59:12Z,2021-12-23T18:59:12Z,OWNER,"The easiest way to change this would be to default to `block=True` such that you need to pass `block=False` to the APIs to have them do fire-and-forget. An alternative would be to add new, separately named methods which do the fire-and-forget thing. If I hadn't recently added `execute_write_script` and `execute_write_many` in #1570 I'd be more into this idea, but I don't want to end up with eight methods - `execute_write`, `execute_write_queue`, `execute_write_many`, `execute_write_many_queue`, `execute_write_script`, `execute_write_scrript_queue`, `execute_write_fn`, `execute_write_fn_queue`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000477813,https://api.github.com/repos/simonw/datasette/issues/1579,1000477813,IC_kwDOBm6k_c47ohR1,9599,simonw,2021-12-23T18:59:41Z,2021-12-23T18:59:41Z,OWNER,"I'm going to go with `execute_write(..., block=False)` as the mechanism for fire-and-forget write queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000479737,https://api.github.com/repos/simonw/datasette/issues/1579,1000479737,IC_kwDOBm6k_c47ohv5,9599,simonw,2021-12-23T19:04:23Z,2021-12-23T19:04:23Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/docs/internals.rst#await-dbexecute_writesql-paramsnone-blocktrue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000481686,https://api.github.com/repos/simonw/datasette/issues/1579,1000481686,IC_kwDOBm6k_c47oiOW,9599,simonw,2021-12-23T19:09:23Z,2021-12-23T19:09:23Z,OWNER,"Re-opening this because I missed updating some of the docs, and I also need to update Datasette's own code to not use `block=True` in a bunch of places.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000485505,https://api.github.com/repos/simonw/datasette/issues/1579,1000485505,IC_kwDOBm6k_c47ojKB,9599,simonw,2021-12-23T19:19:13Z,2021-12-23T19:19:13Z,OWNER,Updated docs for `execute_write_fn()`: https://github.com/simonw/datasette/blob/75153ea9b94d09ec3d61f7c6ebdf378e0c0c7a0b/docs/internals.rst#await-dbexecute_write_fnfn-blocktrue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000485719,https://api.github.com/repos/simonw/datasette/issues/1579,1000485719,IC_kwDOBm6k_c47ojNX,9599,simonw,2021-12-23T19:19:45Z,2021-12-23T19:19:45Z,OWNER,All of those removed `block=True` lines in 8c401ee0f054de2f568c3a8302c9223555146407 really help confirm to me that this was a good decision.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1534#issuecomment-1000535904,https://api.github.com/repos/simonw/datasette/issues/1534,1000535904,IC_kwDOBm6k_c47ovdg,9599,simonw,2021-12-23T21:44:31Z,2021-12-23T21:44:31Z,OWNER,A big downside to this is that I would need to use `Vary: Accept` for when Datasette is running behind a cache such as Cloudflare - would that greatly reduce overall cache efficiency due to subtle variations in the accept headers sent by common browsers?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/issues/1577#issuecomment-1000673444,https://api.github.com/repos/simonw/datasette/issues/1577,1000673444,IC_kwDOBm6k_c47pRCk,9599,simonw,2021-12-24T06:08:58Z,2021-12-24T06:08:58Z,OWNER,"https://pypistats.org/packages/datasette shows a breakdown of downloads by Python version: <img width=""986"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/147323253-1ee22d93-3be2-472b-8ead-495d925958e5.png""> It looks like on a recent day I had 4,071 downloads from Python 3.7... and just 2 downloads from Python 3.6!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1576#issuecomment-1000935523,https://api.github.com/repos/simonw/datasette/issues/1576,1000935523,IC_kwDOBm6k_c47qRBj,9599,simonw,2021-12-24T21:33:05Z,2021-12-24T21:33:05Z,OWNER,"Another option would be to attempt to import `contextvars` and, if the import fails (for Python 3.6) continue using the current mechanism - then let Python 3.6 users know in the documentation that under Python 3.6 they will miss out on nested traces.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/878#issuecomment-1001699559,https://api.github.com/repos/simonw/datasette/issues/878,1001699559,IC_kwDOBm6k_c47tLjn,9599,simonw,2021-12-27T18:53:04Z,2021-12-27T18:53:04Z,OWNER,"I'm going to see if I can come up with the simplest possible version of this pattern for the `/-/metadata` and `/-/metadata.json` page, then try it for the database query page, before tackling the much more complex table page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1152#issuecomment-1001791592,https://api.github.com/repos/simonw/datasette/issues/1152,1001791592,IC_kwDOBm6k_c47tiBo,9599,simonw,2021-12-27T23:04:31Z,2021-12-27T23:04:31Z,OWNER,Another option: rethink permissions to always work in terms of where clauses users as part of a SQL query that returns the overall allowed set of databases or tables. This would require rethinking existing permissions but it might be worthwhile prior to 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view, https://github.com/simonw/datasette/issues/1609#issuecomment-1020456608,https://api.github.com/repos/simonw/datasette/issues/1609,1020456608,IC_kwDOBm6k_c480u6g,9599,simonw,2022-01-24T19:20:09Z,2022-01-24T19:20:09Z,OWNER,Uvicorn have a release out now that would have fixed this issue if I hadn't shipped my own fix: https://github.com/encode/uvicorn/releases/tag/0.17.0.post1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1612#issuecomment-1021413700,https://api.github.com/repos/simonw/datasette/issues/1612,1021413700,IC_kwDOBm6k_c484YlE,9599,simonw,2022-01-25T17:07:29Z,2022-01-25T17:07:29Z,OWNER,"That's a much better place for them, I like this idea. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1612#issuecomment-1021472918,https://api.github.com/repos/simonw/datasette/issues/1612,1021472918,IC_kwDOBm6k_c484nCW,9599,simonw,2022-01-25T18:14:27Z,2022-01-25T18:15:54Z,OWNER,"They're currently shown at the very bottom of the page, under the list of tables and far away from the SQL query box: https://latest.datasette.io/fixtures <img width=""353"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/151034640-c3b79214-7fec-4137-b3ed-45541167feff.png""> I'm also questioning if ""Queries"" is the best header for this. Other options: - **Canned queries** (what the feature is called in the documentation, but I don't think it's a great user-facing term) - **Saved queries** - overlaps with a mechanism by which queries can be saved by the user using a plugin such as [datasette-saved-queries](https://github.com/simonw/datasette-saved-queries) - though that plugin does itself use the canned queries plugin hook so not completely unrelated - **Sample or Example queries** - I don't like these much because they're more than just examples - they are often the core functionality of the specific customized Datasette instance - **Prepared queries** - overlaps with terminology used in other databases, so not great either - **Pre-configured queries** - urgh, don't like that language, feels clumsy - **Query recipes** - bit out of left-field this one, only really makes sense for queries that include named parameters for specific use-cases Maybe ""Queries"" is right after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1612#issuecomment-1021477220,https://api.github.com/repos/simonw/datasette/issues/1612,1021477220,IC_kwDOBm6k_c484oFk,9599,simonw,2022-01-25T18:19:31Z,2022-01-25T18:19:31Z,OWNER,"Here's something I like: I also added a ""Tables"" `<h2>` heading and bumped the tables themselves down from a `<h2>` to a `<h3>`: <img width=""909"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/151035671-9920db99-a943-46a6-8cd8-7a0ad7524474.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1612#issuecomment-1021489826,https://api.github.com/repos/simonw/datasette/issues/1612,1021489826,IC_kwDOBm6k_c484rKi,9599,simonw,2022-01-25T18:34:21Z,2022-01-25T18:34:21Z,OWNER,"OK, that's live on https://latest.datasette.io/fixtures now - I really like it. Thanks for the suggestion!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1613#issuecomment-1021860694,https://api.github.com/repos/simonw/datasette/issues/1613,1021860694,IC_kwDOBm6k_c486FtW,9599,simonw,2022-01-26T04:57:53Z,2022-01-26T04:57:53Z,OWNER,"The existing flow where you can apply filters to a table and then click ""View and edit SQL"" to see the query is a good starting point. Group by queries are both crucially important and difficult to assemble for beginners. Providing a way to see the query that was used by a facet (since facets are really just group-by-counts) would be very useful, which could come out of this: - #1080","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/issues/1613#issuecomment-1022254258,https://api.github.com/repos/simonw/datasette/issues/1613,1022254258,IC_kwDOBm6k_c487lyy,9599,simonw,2022-01-26T14:33:46Z,2022-01-26T14:37:31Z,OWNER,"Tool for setting up foreign key relationships. It could even verify the relationship before you apply it - checking that every value in the column does indeed correspond to a value in the other table. Could also detect and suggest possible ones.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/issues/1613#issuecomment-1022255862,https://api.github.com/repos/simonw/datasette/issues/1613,1022255862,IC_kwDOBm6k_c487mL2,9599,simonw,2022-01-26T14:35:31Z,2022-01-26T14:37:44Z,OWNER,"Joins are really hard. A mechanism for constructing them in the table view would help a lot: - https://github.com/simonw/datasette/issues/613","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/issues/1613#issuecomment-1022257496,https://api.github.com/repos/simonw/datasette/issues/1613,1022257496,IC_kwDOBm6k_c487mlY,9599,simonw,2022-01-26T14:37:14Z,2022-01-26T14:37:14Z,OWNER,"Better contextual help on the SQL editor - like in Django SQL Dashboard which shows all available tables and columns. Fancy inline autocomplete would be great too, but that's pretty hard for SQL based on past research.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/issues/1613#issuecomment-1022381732,https://api.github.com/repos/simonw/datasette/issues/1613,1022381732,IC_kwDOBm6k_c488E6k,9599,simonw,2022-01-26T16:41:45Z,2022-01-26T16:41:45Z,OWNER,A better interface for modifying the columns used in the SELECT clause would be useful too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/issues/1587#issuecomment-1008157908,https://api.github.com/repos/simonw/datasette/issues/1587,1008157908,IC_kwDOBm6k_c48F0TU,9599,simonw,2022-01-08T21:29:06Z,2022-01-08T21:29:06Z,OWNER,"Depending on the SQLite version (and compile options) that ran `ANALYZE` these can be called: - `sqlite_stat1` - `sqlite_stat2` - `sqlite_stat3` - `sqlite_stat4`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097040427,Add `sqlite_stat1`(-4) tables to hidden table list, https://github.com/simonw/datasette/issues/1587#issuecomment-1008157998,https://api.github.com/repos/simonw/datasette/issues/1587,1008157998,IC_kwDOBm6k_c48F0Uu,9599,simonw,2022-01-08T21:29:54Z,2022-01-08T21:29:54Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/datasette/database.py#L339-L354,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097040427,Add `sqlite_stat1`(-4) tables to hidden table list, https://github.com/simonw/datasette/issues/1588#issuecomment-1008227436,https://api.github.com/repos/simonw/datasette/issues/1588,1008227436,IC_kwDOBm6k_c48GFRs,9599,simonw,2022-01-09T04:23:37Z,2022-01-09T04:25:04Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L163-L170 https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L195-L204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097101917,`explain query plan select` is too strict about whitespace, https://github.com/simonw/datasette/issues/1588#issuecomment-1008227491,https://api.github.com/repos/simonw/datasette/issues/1588,1008227491,IC_kwDOBm6k_c48GFSj,9599,simonw,2022-01-09T04:24:09Z,2022-01-09T04:24:09Z,OWNER,"I think this is the fix: ```python re.compile(r""^explain\s+query\s+plan\s+select\b""), ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097101917,`explain query plan select` is too strict about whitespace, https://github.com/simonw/datasette/issues/1590#issuecomment-1010533133,https://api.github.com/repos/simonw/datasette/issues/1590,1010533133,IC_kwDOBm6k_c48O4MN,9599,simonw,2022-01-12T01:19:19Z,2022-01-12T01:19:19Z,OWNER,"Thanks for the steps to reproduce - I have your bug running on my laptop now. I've been mostly testing this stuff using the hosted copy of Datasette here, which doesn't exhibit the bug: https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures?sql=select+sqlite_version%28%29 Something interesting definitely going on here!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010537058,https://api.github.com/repos/simonw/datasette/issues/1590,1010537058,IC_kwDOBm6k_c48O5Ji,9599,simonw,2022-01-12T01:26:34Z,2022-01-12T01:26:34Z,OWNER,"I'm using the https://datasette.io/plugins/datasette-debug-asgi plugin to investigate. On my laptop using Daphne I get this: http://127.0.0.1:8032/datasettes/-/asgi-scope ``` {'actor': None, 'asgi': {'version': '3.0'}, 'client': ['127.0.0.1', 53767], 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x1122aeef0>, 'headers': [(b'host', b'127.0.0.1:8032'), (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko' b'/20100101 Firefox/95.0'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/' b'avif,image/webp,*/*;q=0.8'), (b'accept-language', b'en-US,en;q=0.5'), (b'accept-encoding', b'gzip, deflate'), (b'dnt', b'1'), (b'connection', b'keep-alive'), (b'cookie', b'_ga=GA1.1.742283954.1628542653'), (b'upgrade-insecure-requests', b'1'), (b'sec-fetch-dest', b'document'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-site', b'none'), (b'sec-fetch-user', b'?1')], 'http_version': '1.1', 'method': 'GET', 'path': '/datasettes/-/asgi-scope', 'path_remaining': '', 'query_string': b'', 'raw_path': b'/datasettes/-/asgi-scope', 'root_path': '', 'route_path': '/-/asgi-scope', 'scheme': 'http', 'server': ['127.0.0.1', 8032], 'type': 'http', 'url_route': {'kwargs': {}}} ``` On the demo running on Fly (which I just redeployed with that plugin) I get this: https://datasette-apache-proxy-demo.fly.dev/prefix/-/asgi-scope ``` {'actor': None, 'asgi': {'spec_version': '2.1', 'version': '3.0'}, 'client': ('86.109.12.167', 0), 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x7f4c0413bca0>, 'headers': [(b'host', b'datasette-apache-proxy-demo.fly.dev'), (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko' b'/20100101 Firefox/95.0'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/' b'avif,image/webp,*/*;q=0.8'), (b'accept-language', b'en-US,en;q=0.5'), (b'accept-encoding', b'gzip, deflate, br'), (b'dnt', b'1'), (b'x-request-start', b't=1641950740651658'), (b'sec-fetch-dest', b'document'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-site', b'none'), (b'sec-fetch-user', b'?1'), (b'fly-client-ip', b'24.5.172.176'), (b'x-forwarded-for', b'24.5.172.176, 213.188.193.173, 86.109.12.167'), (b'fly-forwarded-proto', b'https'), (b'x-forwarded-proto', b'https'), (b'fly-forwarded-ssl', b'on'), (b'x-forwarded-ssl', b'on'), (b'fly-forwarded-port', b'443'), (b'x-forwarded-port', b'443'), (b'fly-region', b'sjc'), (b'fly-request-id', b'01FS5Y805BX43HM94T8XW610KG'), (b'via', b'2 fly.io'), (b'fly-dispatch-start', b't=1641950740683198;instance=87f188a2'), (b'x-forwarded-host', b'datasette-apache-proxy-demo.fly.dev'), (b'x-forwarded-server', b'localhost'), (b'connection', b'Keep-Alive')], 'http_version': '1.1', 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'', 'raw_path': b'/-/asgi-scope', 'root_path': '', 'scheme': 'https', 'server': ('127.0.0.1', 8001), 'type': 'http', 'url_route': {'kwargs': {}}} ``` The version that works as ` 'raw_path': b'/-/asgi-scope'` - the version that fails has `'raw_path': b'/datasettes/-/asgi-scope'`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010538016,https://api.github.com/repos/simonw/datasette/issues/1590,1010538016,IC_kwDOBm6k_c48O5Yg,9599,simonw,2022-01-12T01:28:19Z,2022-01-12T01:28:19Z,OWNER,"The Daphne one has this key: `'route_path': '/-/asgi-scope',` Maybe Datasette's routing code needs to look out for that, if it's available, and use it to reconstruct the requested path? The code in question is here: https://github.com/simonw/datasette/blob/8c401ee0f054de2f568c3a8302c9223555146407/datasette/app.py#L1143-L1149","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010538188,https://api.github.com/repos/simonw/datasette/issues/1590,1010538188,IC_kwDOBm6k_c48O5bM,9599,simonw,2022-01-12T01:28:41Z,2022-01-12T01:30:43Z,OWNER,"Oh wait! It looks like `route_path` is something I invented there. Yup, I added it in https://github.com/simonw/datasette/commit/a63412152518581c6a3d4e142b937e27dabdbfdb - commit message says: > - new `route_path` key in `request.scope` storing the path that was used for routing with the `base_url` prefix stripped So actually part of the mystery here is: why does the Fly hosted one NOT have that key?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010540923,https://api.github.com/repos/simonw/datasette/issues/1590,1010540923,IC_kwDOBm6k_c48O6F7,9599,simonw,2022-01-12T01:33:49Z,2022-01-12T01:33:49Z,OWNER,"Looking closer at the code quoted above, it doesn't modify `path` or `raw_path` at all - ALL it does is add the `route_path` to the scope.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1591#issuecomment-1010764036,https://api.github.com/repos/simonw/datasette/issues/1591,1010764036,IC_kwDOBm6k_c48PwkE,9599,simonw,2022-01-12T08:22:16Z,2022-01-12T08:22:32Z,OWNER,"The challenge here is avoiding clashes. What if a plugin adds an option that I later want to use for a new Datasette core feature? Or what if two plugins define the same option? Maybe the solution is to make them use namespaces defined by the plugin name. How about this: datasette t.db --plugin-setting datasette-tiddlywiki db t.db It's a bit verbose having an option that itself then takes THREE strings: plugin name, setting name, setting value - but it would work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1592#issuecomment-1011185061,https://api.github.com/repos/simonw/datasette/issues/1592,1011185061,IC_kwDOBm6k_c48RXWl,9599,simonw,2022-01-12T15:50:41Z,2022-01-12T15:50:41Z,OWNER,Twitter: https://twitter.com/dracos/status/1481290103519592459,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100499619,Row pages should show links to foreign keys, https://github.com/simonw/datasette/issues/1590#issuecomment-1012656790,https://api.github.com/repos/simonw/datasette/issues/1590,1012656790,IC_kwDOBm6k_c48W-qW,9599,simonw,2022-01-14T01:05:34Z,2022-01-14T01:05:34Z,OWNER,"I think this prefixed string mechanism is supposed to prevent the `base_url` prefix from being applied twice: https://github.com/simonw/datasette/blob/3664ddd400062123e99500d28b160c7944408c1a/datasette/url_builder.py#L9-L16 But with a bit of extra logging all of the inputs to that are NOT prefixed strings: ``` Urls.path called with: /datasettes/fixtures/compound_three_primary_keys?_sort=content (PrefixedUrlString = False) returning /datasettes/datasettes/fixtures/compound_three_primary_keys?_sort=content ``` So it looks like `urls.path(...)` is indeed the code responsible for doubling up that `/datasettes/` prefix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1595#issuecomment-1012626243,https://api.github.com/repos/simonw/datasette/issues/1595,1012626243,IC_kwDOBm6k_c48W3ND,9599,simonw,2022-01-14T00:00:33Z,2022-01-14T00:00:33Z,OWNER,"Copying those in here: - New plugin hook: [filters_from_request(request, database, table, datasette)](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-filters-from-request), which runs on the table page and can be used to support new custom query string parameters that modify the SQL query. ([#473](https://github.com/simonw/datasette/issues/473)) - The number of unique values in a facet is now always displayed. Previously it was only displayed if the user specified `?_facet_size=max`. ([#1556](https://github.com/simonw/datasette/issues/1556)) - Fixed bug where `?_facet_array=tags&_facet=tags` would only display one of the two selected facets. ([#625](https://github.com/simonw/datasette/issues/625)) - Facets of type `date` or `array` can now be configured in `metadata.json`, see [Facets in metadata.json](https://docs.datasette.io/en/latest/facets.html#facets-metadata). Thanks, David Larlet. ([#1552](https://github.com/simonw/datasette/issues/1552)) - New `?_nosuggest=1` parameter for table views, which disables facet suggestion. ([#1557](https://github.com/simonw/datasette/issues/1557)) - Label columns detected for foreign keys are now case-insensitive, so `Name` or `TITLE` will be detected in the same way as `name` or `title`. ([#1544](https://github.com/simonw/datasette/issues/1544)) - The query string variables exposed by `request.args` will now include blank strings for arguments such as `foo` in `?foo=&bar=1` rather than ignoring those parameters entirely. ([#1551](https://github.com/simonw/datasette/issues/1551)) - Database write connections now execute the [prepare_connection(conn, database, datasette)](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-prepare-connection) plugin hook. ([#1564](https://github.com/simonw/datasette/issues/1564)) - The `Datasette()` constructor no longer requires the `files=` argument, and is now documented at [Datasette class](https://docs.datasette.io/en/latest/internals.html#internals-datasette). ([#1563](https://github.com/simonw/datasette/issues/1563)) - The tracing feature now traces write queries, not just read queries. ([#1568](https://github.com/simonw/datasette/issues/1568)) - Added two methods for writing to the database: [await db.execute_write_script(sql, block=False)](https://docs.datasette.io/en/latest/internals.html#database-execute-write-script) and [await db.execute_write_many(sql, params_seq, block=False)](https://docs.datasette.io/en/latest/internals.html#database-execute-write-many). ([#1570](https://github.com/simonw/datasette/issues/1570)) - Made several performance improvements to the database schema introspection code that runs when Datasette first starts up. ([#1555](https://github.com/simonw/datasette/issues/1555)) - Fixed bug where writable canned queries could not be used with custom templates. ([#1547](https://github.com/simonw/datasette/issues/1547))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102484126,Release notes for 0.60, https://github.com/simonw/datasette/issues/1595#issuecomment-1012626410,https://api.github.com/repos/simonw/datasette/issues/1595,1012626410,IC_kwDOBm6k_c48W3Pq,9599,simonw,2022-01-14T00:00:56Z,2022-01-14T01:17:47Z,OWNER,Commits since 0.60a1: https://github.com/simonw/datasette/compare/0.60a1...3664ddd40,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102484126,Release notes for 0.60, https://github.com/simonw/datasette/issues/1597#issuecomment-1012629825,https://api.github.com/repos/simonw/datasette/issues/1597,1012629825,IC_kwDOBm6k_c48W4FB,9599,simonw,2022-01-14T00:07:36Z,2022-01-14T00:07:36Z,OWNER,"Fixed: ``` % datasette --help Usage: datasette [OPTIONS] COMMAND [ARGS]... Datasette is an open source multi-tool for exploring and publishing data About Datasette: https://datasette.io/ Full documentation: https://docs.datasette.io/ Options: --version Show the version and exit. --help Show this message and exit. Commands: serve* Serve up specified SQLite database files with a web UI inspect Generate JSON summary of provided database files ... % datasette inspect --help Usage: datasette inspect [OPTIONS] [FILES]... Generate JSON summary of provided database files This can then be passed to ""datasette --inspect-file"" to speed up count operations against immutable database files. Options: --inspect-file TEXT --load-extension TEXT Path to a SQLite extension to load --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102612922,"""datasette inspect"" has no help summary", https://github.com/simonw/datasette/issues/1527#issuecomment-1012634659,https://api.github.com/repos/simonw/datasette/issues/1527,1012634659,IC_kwDOBm6k_c48W5Qj,9599,simonw,2022-01-14T00:17:00Z,2022-01-14T00:18:11Z,OWNER,"That's because that page has this unnecessary hidden form field: ```html <input type=""hidden"" name=""_city_id__gt"" value=""1""> ``` That field is added by this bit in the template: https://github.com/simonw/datasette/blob/515f8d38ebae203efc15ca79a8b42848276b35e5/datasette/templates/table.html#L119-L122 Which is populated here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/views/table.py#L813-L821 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1527#issuecomment-1012635696,https://api.github.com/repos/simonw/datasette/issues/1527,1012635696,IC_kwDOBm6k_c48W5gw,9599,simonw,2022-01-14T00:19:10Z,2022-01-14T00:20:36Z,OWNER,"Oh! This is because `_city_id` has a leading underscore (for testing purposes). I think I need to filter out any keys that contain `__` in that case. What happens to columns that contain a `__`? They shouldn't be reflected in the hidden arguments either - this code is really only supposed to catch things like `_where` and `_m2m_through=` and `_col` - basically most of the list on https://docs.datasette.io/en/stable/json_api.html#special-table-arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1598#issuecomment-1012643882,https://api.github.com/repos/simonw/datasette/issues/1598,1012643882,IC_kwDOBm6k_c48W7gq,9599,simonw,2022-01-14T00:34:49Z,2022-01-14T00:34:49Z,OWNER,There are four places in the documentation that use `.. literalinclude::` at the moment which I can replace - I can actually just link directly to the new https://docs.datasette.io/en/latest/cli-reference.html page instead of embedding the help directly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102637351,Replace update-docs-help.py script with cog, https://github.com/simonw/datasette/issues/1590#issuecomment-1012653966,https://api.github.com/repos/simonw/datasette/issues/1590,1012653966,IC_kwDOBm6k_c48W9-O,9599,simonw,2022-01-14T00:59:07Z,2022-01-14T00:59:07Z,OWNER,"Since this is a special case bug for when using Datasette as a library I wonder if a good fix here would be to support something like this: ```python application = URLRouter([ re_path(r""^datasettes/.*"", asgi_cors(datasette_.app(remove_path_prefix=""datasettes/""), allow_all=True)), ]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1527#issuecomment-1012653109,https://api.github.com/repos/simonw/datasette/issues/1527,1012653109,IC_kwDOBm6k_c48W9w1,9599,simonw,2022-01-14T00:57:08Z,2022-01-14T00:57:08Z,OWNER,Bug is fixed on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1591#issuecomment-1012506595,https://api.github.com/repos/simonw/datasette/issues/1591,1012506595,IC_kwDOBm6k_c48WZ_j,9599,simonw,2022-01-13T20:52:56Z,2022-01-13T20:52:56Z,OWNER,"You can already run `datasette --help-settings` to see detailed help on available settings. Maybe `datasette --help-plugin-settings` could do the same thing for plugin settings? Or I could even have available plugin settings show up as a list at the bottom of the `datasette --help-settings` output - which currently looks like this: ``` % datasette --help-settings Settings: default_page_size Default page size for the table view (default=100) max_returned_rows Maximum rows that can be returned from a table or custom query (default=1000) num_sql_threads Number of threads in the thread pool for executing SQLite queries (default=3) sql_time_limit_ms Time limit for a SQL query in milliseconds (default=1000) default_facet_size Number of values to return for requested facets (default=30) facet_time_limit_ms Time limit for calculating a requested facet (default=200) facet_suggest_time_limit_ms Time limit for calculating a suggested facet (default=50) hash_urls Include DB file contents hash in URLs, for far- future caching (default=False) allow_facet Allow users to specify columns to facet using ?_facet= parameter (default=True) allow_download Allow users to download the original SQLite database files (default=True) suggest_facets Calculate and display suggested facets (default=True) default_cache_ttl Default HTTP cache TTL (used in Cache-Control: max-age= header) (default=5) default_cache_ttl_hashed Default HTTP cache TTL for hashed URL pages (default=31536000) cache_size_kb SQLite cache size in KB (0 == use SQLite default) (default=0) allow_csv_stream Allow .csv?_stream=1 to download all rows (ignoring max_returned_rows) (default=True) max_csv_mb Maximum size allowed for CSV export in MB - set 0 to disable this limit (default=100) truncate_cells_html Truncate cells longer than this in HTML table view - set 0 to disable (default=2048) force_https_urls Force URLs in API output to always use https:// protocol (default=False) template_debug Allow display of template debug information with ?_context=1 (default=False) trace_debug Allow display of SQL trace debug information with ?_trace=1 (default=False) base_url Datasette URLs should use this base path (default=/) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1591#issuecomment-1012504251,https://api.github.com/repos/simonw/datasette/issues/1591,1012504251,IC_kwDOBm6k_c48WZa7,9599,simonw,2022-01-13T20:49:19Z,2022-01-13T20:49:59Z,OWNER,"I try to stick pretty closely to what [Click](https://click.palletsprojects.com/en/8.0.x/) supports, and Click likes you to define options explicitly so that it can display them in the output of `--help`. But... that makes me think that actually showing these options in `--help` is likely a better idea. My `--plugin-setting` concept would have help that looks something like this: ``` % datasette serve --help ... --plugin-setting <TEXT TEXT TEXT>... Setting for a specified plugin. ``` That's not great help! The alternative would be to allow plugins to register their extra options with the command - which would mean the help output could look like this instead: ``` % datasette serve --help ... --tiddlywiki-db <TEXT> Name of database to use for datasette-tiddlywiki ``` This feels like a good argument to me for plugins to explicitly register their settings. I'm not sure if I should enforce the `name-of-plugin-` prefix, or if I should allow plugins to define any setting they like. I'm still nervous about plugins over-riding existing or future default options to that command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1591#issuecomment-1012505706,https://api.github.com/repos/simonw/datasette/issues/1591,1012505706,IC_kwDOBm6k_c48WZxq,9599,simonw,2022-01-13T20:51:30Z,2022-01-13T20:51:30Z,OWNER,"Another option: if I make plugin settings a higher level concept in Datasette than they are at the moment, I could allow them to be set either using `--options` OR using the existing `metadata.yml/json` mechanism. https://docs.datasette.io/en/stable/plugins.html#plugin-configuration I want to make changes to that anyway, because I'm increasingly uncomfortable with plugin settings ending up in the ""metadata"" mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1590#issuecomment-1012661522,https://api.github.com/repos/simonw/datasette/issues/1590,1012661522,IC_kwDOBm6k_c48W_0S,9599,simonw,2022-01-14T01:16:08Z,2022-01-14T01:16:34Z,OWNER,"OK, I'm going to recommend a workaround for this instead. Here's `asgi.py` updated to strip the prefix before passing the request on to Datasette: ```python import pathlib from asgi_cors import asgi_cors from channels.routing import URLRouter from django.urls import re_path from datasette.app import Datasette def rewrite_path(app, prefix_to_strip): async def rewrite_path_app(scope, receive, send): if ( scope[""type""] == ""http"" and ""path"" in scope and scope[""path""].startswith(prefix_to_strip) ): scope[""path""] = scope[""path""][len(prefix_to_strip) :] if ""raw_path"" in scope: scope[""raw_path""] = scope[""raw_path""][len(prefix_to_strip) :] await app(scope, receive, send) return rewrite_path_app datasette_ = Datasette( files=[""fixtures.db""], settings={""base_url"": ""/datasettes/"", ""plugins"": {}}, ) application = URLRouter( [ re_path( r""^datasettes/.*"", asgi_cors(rewrite_path(datasette_.app(), ""/datasettes""), allow_all=True), ), ] ) ``` This works on my laptop - please re-open the ticket if it doesn't work for you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1594#issuecomment-1012508787,https://api.github.com/repos/simonw/datasette/issues/1594,1012508787,IC_kwDOBm6k_c48Wahz,9599,simonw,2022-01-13T20:56:14Z,2022-01-13T20:56:34Z,OWNER,"The implementation can be _almost_ exactly the same as this: https://github.com/simonw/sqlite-utils/blame/74586d3cb26fa3cc3412721985ecdc1864c2a31d/docs/cli-reference.rst#L11-L76 I need to do something extra to ensure the output of `datasette --help-settings` is shown too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102359726,"Add a CLI reference page to the docs, inspired by sqlite-utils", https://github.com/simonw/datasette/issues/1594#issuecomment-1012535024,https://api.github.com/repos/simonw/datasette/issues/1594,1012535024,IC_kwDOBm6k_c48Wg7w,9599,simonw,2022-01-13T21:36:53Z,2022-01-13T21:36:53Z,OWNER,I went with a simpler pattern that `sqlite-utils` because Datasette has a lot less commands: https://github.com/simonw/datasette/blob/4b23f01f3e668c8f2a2f1a294be49f49b4073969/docs/cli-reference.rst#L9-L35,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102359726,"Add a CLI reference page to the docs, inspired by sqlite-utils", https://github.com/simonw/datasette/issues/1594#issuecomment-1012536257,https://api.github.com/repos/simonw/datasette/issues/1594,1012536257,IC_kwDOBm6k_c48WhPB,9599,simonw,2022-01-13T21:38:48Z,2022-01-13T21:38:48Z,OWNER,https://docs.datasette.io/en/latest/cli-reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102359726,"Add a CLI reference page to the docs, inspired by sqlite-utils", https://github.com/simonw/datasette/issues/1336#issuecomment-1012546924,https://api.github.com/repos/simonw/datasette/issues/1336,1012546924,IC_kwDOBm6k_c48Wj1s,9599,simonw,2022-01-13T21:55:58Z,2022-01-13T21:55:58Z,OWNER,"See also: - #1412","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",895686039,Document turning on WAL for live served SQLite databases, https://github.com/simonw/datasette/issues/1466#issuecomment-1012546223,https://api.github.com/repos/simonw/datasette/issues/1466,1012546223,IC_kwDOBm6k_c48Wjqv,9599,simonw,2022-01-13T21:54:51Z,2022-01-13T21:54:51Z,OWNER,"Going with this for the copy: > [Datasette Desktop](https://datasette.io/desktop) is a packaged Mac application which bundles Datasette together with Python and allows you to install and run Datasette directly on your laptop. This is the best option for local installation if you are not comfortable using the command line.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991467558,Add Datasette Desktop to installation documentation, https://github.com/simonw/datasette/issues/650#issuecomment-1012552760,https://api.github.com/repos/simonw/datasette/issues/650,1012552760,IC_kwDOBm6k_c48WlQ4,9599,simonw,2022-01-13T22:04:56Z,2022-01-13T22:04:56Z,OWNER,"Challenge: explain the difference between view as in SQL view, and view as in the code that serves `TableView` / `DatabaseView` etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534629631,Add a glossary to the documentation, https://github.com/simonw/datasette/issues/1588#issuecomment-1012554208,https://api.github.com/repos/simonw/datasette/issues/1588,1012554208,IC_kwDOBm6k_c48Wlng,9599,simonw,2022-01-13T22:07:15Z,2022-01-13T22:07:15Z,OWNER,This works now: https://latest.datasette.io/fixtures?sql=explain+query+plan++select+*+from+facetable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097101917,`explain query plan select` is too strict about whitespace, https://github.com/simonw/datasette/issues/1595#issuecomment-1012575013,https://api.github.com/repos/simonw/datasette/issues/1595,1012575013,IC_kwDOBm6k_c48Wqsl,9599,simonw,2022-01-13T22:29:22Z,2022-01-13T22:29:22Z,OWNER,"Most of these are already written for these two alpha releases: - https://github.com/simonw/datasette/releases/tag/0.60a0 - https://github.com/simonw/datasette/releases/tag/0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102484126,Release notes for 0.60, https://github.com/simonw/datasette/issues/1590#issuecomment-1012583091,https://api.github.com/repos/simonw/datasette/issues/1590,1012583091,IC_kwDOBm6k_c48Wsqz,9599,simonw,2022-01-13T22:41:15Z,2022-01-13T22:41:15Z,OWNER,"Seeing as this area of the code has produced so many bugs in the past, I think part of the fix may be to write comprehensive documentation about how routing works for the internals documentation. Doing so might help me figure this bug out!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1595#issuecomment-1012664607,https://api.github.com/repos/simonw/datasette/issues/1595,1012664607,IC_kwDOBm6k_c48XAkf,9599,simonw,2022-01-14T01:22:58Z,2022-01-14T01:22:58Z,OWNER,"- Upgraded Pluggy dependency to 1.0. #1575 - Now using [Plausible](https://plausible.io/) analytics for the Datasette documentation. - The `db.execute_write()` internals method now defaults to blocking until the write operation has completed. Previously it defaulted to queuing the write and then continuing to run code while the write was in the queue. #1579 - `explain query plan` is now allowed with varying amounts of white space in the query. #1588 - New CLI reference page showing the output of `--help` for each of the `datasette` sub-commands. This lead to several small improvements to the help copy. #1594 - Fixed bug where columns with a underscore prefix could result in unnecessary hidden form fields. #1527","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102484126,Release notes for 0.60, https://github.com/simonw/datasette/issues/1566#issuecomment-1012680228,https://api.github.com/repos/simonw/datasette/issues/1566,1012680228,IC_kwDOBm6k_c48XEYk,9599,simonw,2022-01-14T01:59:54Z,2022-01-14T01:59:54Z,OWNER,This is now shipped!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1591#issuecomment-1013669543,https://api.github.com/repos/simonw/datasette/issues/1591,1013669543,IC_kwDOBm6k_c48a16n,9599,simonw,2022-01-15T11:56:59Z,2022-01-15T11:56:59Z,OWNER,"There's actually already a way to move regular Datasette `--setting` options to a `settings.json` file thanks to configuration directory mode: https://docs.datasette.io/en/stable/settings.html Maybe extending that mechanism to handle plugins would be a neat path forward here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1591#issuecomment-1013668967,https://api.github.com/repos/simonw/datasette/issues/1591,1013668967,IC_kwDOBm6k_c48a1xn,9599,simonw,2022-01-15T11:53:21Z,2022-01-15T11:53:21Z,OWNER,"The `datasette publish --plugin-secret name setting value` option already implements something a bit like this. https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values It's a bit of a messy hack to compensate for metadata being visible. Maybe I could replace that mechanism with the proposed plugin configuration rethink from this issue. I still like the debug benefits of making plugin settings public - perhaps add a rule that if a plugin setting has a `secret:` prefix it gets redacted on a new `/-/plugin-settings` page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1603#issuecomment-1016588326,https://api.github.com/repos/simonw/datasette/issues/1603,1016588326,IC_kwDOBm6k_c48l-gm,9599,simonw,2022-01-19T15:35:33Z,2022-01-19T15:35:33Z,OWNER,"I don't think abusing the template loader mechanism for this will work: Jinja provides an API for loading text templates, but I don't think it can be sensibly abused to open binary image files instead. Loaded code is here: https://github.com/pallets/jinja/blob/main/src/jinja2/loaders.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1016589140,https://api.github.com/repos/simonw/datasette/issues/1603,1016589140,IC_kwDOBm6k_c48l-tU,9599,simonw,2022-01-19T15:36:16Z,2022-01-19T15:36:16Z,OWNER,For the moment then I will hard-code a new favicon and leave it to ASGI plugins if people want to define their own.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1016589519,https://api.github.com/repos/simonw/datasette/issues/1603,1016589519,IC_kwDOBm6k_c48l-zP,9599,simonw,2022-01-19T15:36:38Z,2022-01-19T15:36:38Z,OWNER,Also people can use a custom base template and link to a custom favicon if they want to.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1016579661,https://api.github.com/repos/simonw/datasette/issues/1603,1016579661,IC_kwDOBm6k_c48l8ZN,9599,simonw,2022-01-19T15:27:05Z,2022-01-19T15:27:05Z,OWNER,I'd like this to be customizable. I'm going to load it from the template loading system such that a custom favicon for a specific instance can be dropped in using a file in `templates/`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1601#issuecomment-1016639294,https://api.github.com/repos/simonw/datasette/issues/1601,1016639294,IC_kwDOBm6k_c48mK8-,9599,simonw,2022-01-19T16:26:44Z,2022-01-20T03:57:17Z,OWNER,"I need to add `sqlite_stat1` to the hidden tables too, see: - https://github.com/simonw/datasette/issues/1587 - https://github.com/simonw/sqlite-utils/issues/366","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1604#issuecomment-1016636561,https://api.github.com/repos/simonw/datasette/issues/1604,1016636561,IC_kwDOBm6k_c48mKSR,9599,simonw,2022-01-19T16:23:54Z,2022-01-19T16:23:54Z,OWNER,"Potential design: datasette publish cloudrun data.db \ --service my-service \ --domain demo.datasette.io I think I'm OK with calling this `--domain` even when it is being used with a subdomain. This will require `datasette.io` to already have been verified with Google. Not sure how best to handle the DNS part - maybe print out instructions for the necessary CNAME?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108300685,Option to assign a domain/subdomain using `datasette publish cloudrun`, https://github.com/simonw/datasette/issues/1601#issuecomment-1016637722,https://api.github.com/repos/simonw/datasette/issues/1601,1016637722,IC_kwDOBm6k_c48mKka,9599,simonw,2022-01-19T16:25:07Z,2022-01-19T16:25:07Z,OWNER,Good idea - though I'm nervous about accidentally hiding a `data_licenses` table outside of the SpatiaLite case. I'll only hide that one if SpatiaLite is loaded.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1577#issuecomment-1017112130,https://api.github.com/repos/simonw/datasette/issues/1577,1017112130,IC_kwDOBm6k_c48n-ZC,9599,simonw,2022-01-20T04:33:57Z,2022-01-20T04:33:57Z,OWNER,"Dropped support for Python 3.6: updated `setup.py`, changed all references in the documentation to say 3.7 is the minimum required version, got rid of the GitHub Actions tests against 3.6.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1576#issuecomment-1017112543,https://api.github.com/repos/simonw/datasette/issues/1576,1017112543,IC_kwDOBm6k_c48n-ff,9599,simonw,2022-01-20T04:35:00Z,2022-02-05T04:33:46Z,OWNER,I dropped support for Python 3.6 in fae3983c51f4a3aca8335f3e01ff85ef27076fbf so now free to use `contextvars` for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1601#issuecomment-1017113831,https://api.github.com/repos/simonw/datasette/issues/1601,1017113831,IC_kwDOBm6k_c48n-zn,9599,simonw,2022-01-20T04:38:14Z,2022-01-20T04:38:14Z,OWNER,"I don't have solid tests in place for exercising SpatiaLite, but this change feels safe enough that I'm not going to add tests for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1601#issuecomment-1017095821,https://api.github.com/repos/simonw/datasette/issues/1601,1017095821,IC_kwDOBm6k_c48n6aN,9599,simonw,2022-01-20T03:56:14Z,2022-01-20T03:56:28Z,OWNER,"Oh interesting, I hadn't heard about KNN2. It looks like it was added to SpatiaLite on 21st June 2021 in https://www.gaia-gis.it/fossil/libspatialite/info/03786a62cdb4ab17 but the most recent release of SpatiaLite is 5.0.1 from 7th February 2021 - so it's not yet in a release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1606#issuecomment-1017108291,https://api.github.com/repos/simonw/datasette/issues/1606,1017108291,IC_kwDOBm6k_c48n9dD,9599,simonw,2022-01-20T04:24:54Z,2022-01-20T04:24:54Z,OWNER,"https://github.com/simonw/latest-datasette-with-all-plugins/commit/1e12ffe70be791e3281b41810e837515314c1317 shows that 5 days ago Datasette upgraded from Uvicorn 0.16 to 0.17 Sure enough, in the changelog for 0.17: https://github.com/encode/uvicorn/blob/0.17.0/CHANGELOG.md > - Drop Python 3.6 support ([#1261](https://github.com/encode/uvicorn/pull/1261)) 06/01/22","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108846067,Tests failing against Python 3.6, https://github.com/simonw/datasette/issues/1606#issuecomment-1017108960,https://api.github.com/repos/simonw/datasette/issues/1606,1017108960,IC_kwDOBm6k_c48n9ng,9599,simonw,2022-01-20T04:26:36Z,2022-01-20T04:26:36Z,OWNER,"https://pypistats.org/packages/datasette shows Python 3.6 is 0.24% of Datasette downloads. <img width=""941"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150273301-18ec2ec8-f9ce-49a3-b57d-e74826a5c7ae.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108846067,Tests failing against Python 3.6, https://github.com/simonw/datasette/issues/1606#issuecomment-1017109194,https://api.github.com/repos/simonw/datasette/issues/1606,1017109194,IC_kwDOBm6k_c48n9rK,9599,simonw,2022-01-20T04:27:07Z,2022-01-20T04:27:07Z,OWNER,"Relevant: - https://github.com/simonw/datasette/issues/1577","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108846067,Tests failing against Python 3.6, https://github.com/simonw/datasette/issues/1577#issuecomment-1017109401,https://api.github.com/repos/simonw/datasette/issues/1577,1017109401,IC_kwDOBm6k_c48n9uZ,9599,simonw,2022-01-20T04:27:34Z,2022-01-20T04:27:34Z,OWNER,"OK, now that Uvicorn has dropped 3.6 support - see #1606 - I think this decision is easy to make. I'm dropping 3.6.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1605#issuecomment-1016977725,https://api.github.com/repos/simonw/datasette/issues/1605,1016977725,IC_kwDOBm6k_c48ndk9,9599,simonw,2022-01-19T23:55:08Z,2022-01-19T23:55:08Z,OWNER,"Oh that's interesting. I was thinking about this from a slightly different angle recently - pondering what a static site generator built on top of Datasette might look like. Just a sketch at the moment, but I was imagining a YAML configuration file with a SQL query that returns a list of paths - then a tool that runs that query and uses the equivalent of `datasette --get` to create a static copy of each of those paths. I think these two ideas can probably be merged. I'd love to know more about how you are solving this right now!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1356#issuecomment-1017016553,https://api.github.com/repos/simonw/datasette/issues/1356,1017016553,IC_kwDOBm6k_c48nnDp,9599,simonw,2022-01-20T01:06:37Z,2022-01-20T01:06:37Z,OWNER,"> A problem with this is that if you're using `--query` you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is `.csv` and plugin formats can't handle full streams, see #1062 and #1177. I figured out a neat pattern for streaming JSON arrays in this TIL: https://til.simonwillison.net/python/output-json-array-streaming","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""", https://github.com/simonw/datasette/issues/1600#issuecomment-1017124310,https://api.github.com/repos/simonw/datasette/issues/1600,1017124310,IC_kwDOBm6k_c48oBXW,9599,simonw,2022-01-20T05:06:09Z,2022-01-20T05:06:09Z,OWNER,Fixed: https://docs.datasette.io/en/latest/plugins.html#seeing-what-plugins-are-installed,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1104691662,plugins --all example should use cog, https://github.com/simonw/datasette/issues/1603#issuecomment-1017131209,https://api.github.com/repos/simonw/datasette/issues/1603,1017131209,IC_kwDOBm6k_c48oDDJ,9599,simonw,2022-01-20T05:22:40Z,2022-01-20T05:22:40Z,OWNER,This one is 101KB and 1536*1536 which is a bit big! https://github.com/simonw/datasette-app/blob/main/build/icon.png,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017131801,https://api.github.com/repos/simonw/datasette/issues/1603,1017131801,IC_kwDOBm6k_c48oDMZ,9599,simonw,2022-01-20T05:23:57Z,2022-01-20T05:23:57Z,OWNER,"https://adamj.eu/tech/2022/01/18/how-to-add-a-favicon-to-your-django-site/ suggests 64x64, I'm going with 128x128 just in case anyone invents a retina-retina screen.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017132927,https://api.github.com/repos/simonw/datasette/issues/1603,1017132927,IC_kwDOBm6k_c48oDd_,9599,simonw,2022-01-20T05:26:29Z,2022-01-20T05:26:29Z,OWNER,"Here's the 128x128 one - 11kb, I resized it using Preview: ![icon](https://user-images.githubusercontent.com/9599/150278798-48cc2da2-3640-414d-a440-20c9d93c09f4.png) Now running it through [Squoosh](https://squoosh.app/editor) using OxiPNG effort=3 colours=24 - brought it down to 1.36KB. ![favicon](https://user-images.githubusercontent.com/9599/150278906-b312940e-0cdb-43ef-a325-45d2b1c240de.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017133357,https://api.github.com/repos/simonw/datasette/issues/1603,1017133357,IC_kwDOBm6k_c48oDkt,9599,simonw,2022-01-20T05:27:34Z,2022-01-20T05:27:34Z,OWNER,"I'm going to drop it in `datasette/static/favicon.png` and rewrite the `async def favicon()` function to return it, with caching headers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017136897,https://api.github.com/repos/simonw/datasette/issues/1603,1017136897,IC_kwDOBm6k_c48oEcB,9599,simonw,2022-01-20T05:36:29Z,2022-01-20T05:36:29Z,OWNER,"Here's what it looks like in Firefox, Chrome and Safari: ![CleanShot 2022-01-19 at 21 35 06@2x](https://user-images.githubusercontent.com/9599/150279832-6c233115-6540-4746-8cd1-dce25321ebbf.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017137117,https://api.github.com/repos/simonw/datasette/issues/1603,1017137117,IC_kwDOBm6k_c48oEfd,9599,simonw,2022-01-20T05:37:05Z,2022-01-20T05:37:34Z,OWNER,"I'm not crazy about the look - I think this version of the grid may have too many lines for this particular display size. I'm going to try reducing the number of lines in Figma to see if I like that better. https://www.figma.com/file/LKjceTFNtKm6wCbScDqm1Y/Datasette-Logo","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017139321,https://api.github.com/repos/simonw/datasette/issues/1603,1017139321,IC_kwDOBm6k_c48oFB5,9599,simonw,2022-01-20T05:43:07Z,2022-01-20T05:46:42Z,OWNER,"My attempt with one less grid line looked awful when shrunk down to the browser size: <img width=""350"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150280445-a482de8e-8b5f-4145-8567-36ac7105c303.png""> File here: ![favicon](https://user-images.githubusercontent.com/9599/150280463-817a8065-8393-4c10-aec4-5a627cf46fae.png) I'm going with the first attempt for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017142395,https://api.github.com/repos/simonw/datasette/issues/1603,1017142395,IC_kwDOBm6k_c48oFx7,9599,simonw,2022-01-20T05:50:55Z,2022-01-20T05:50:55Z,OWNER,The new `/favicon.ico` view: https://github.com/simonw/datasette/blob/0467723ee57c2cbc0f02daa47cef632dd4651df0/datasette/app.py#L182-L192,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017143437,https://api.github.com/repos/simonw/datasette/issues/1603,1017143437,IC_kwDOBm6k_c48oGCN,9599,simonw,2022-01-20T05:53:17Z,2022-01-20T05:53:17Z,OWNER,New favicon now live on https://latest.datasette.io/ - see also https://latest.datasette.io/favicon.ico,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017143817,https://api.github.com/repos/simonw/datasette/issues/1603,1017143817,IC_kwDOBm6k_c48oGIJ,9599,simonw,2022-01-20T05:54:09Z,2022-01-20T05:54:09Z,OWNER,"Oops, I pushed the one I liked least out of the two options! Fixing now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017149828,https://api.github.com/repos/simonw/datasette/issues/1603,1017149828,IC_kwDOBm6k_c48oHmE,9599,simonw,2022-01-20T06:07:31Z,2022-01-20T06:07:31Z,OWNER,"Now live on https://latest.datastte.io/ - I'm happy enough with this for the moment: <img width=""364"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150283069-dd23e688-ec77-4200-a892-38e0f8892f30.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017800506,https://api.github.com/repos/simonw/datasette/issues/1603,1017800506,IC_kwDOBm6k_c48qmc6,9599,simonw,2022-01-20T18:31:18Z,2022-01-20T18:31:18Z,OWNER,"One last go at tidying this up. I decided to do a 32x32 pixel version in Pixelmator, using this trick to access a pixel brush: https://osxdaily.com/2016/11/17/enable-pixel-brush-pixelmator-mac/ <img width=""916"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150399724-10b17a97-3ecf-4d82-868c-f8a642f16838.png""> Frustrating how the white boxes are all exactly four pixels high and ALMOST all four pixels wide, but one of them has to be three pixels wide to fit the space.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017806497,https://api.github.com/repos/simonw/datasette/issues/1603,1017806497,IC_kwDOBm6k_c48qn6h,9599,simonw,2022-01-20T18:39:27Z,2022-01-20T18:39:57Z,OWNER,"Here's a comparison between my hand-edited version and the one I have now: ![CleanShot 2022-01-20 at 10 38 00@2x](https://user-images.githubusercontent.com/9599/150401244-7e78ee93-1973-4c95-8f91-01e51e0d5366.png) The new 32x32 image: <img src=""https://user-images.githubusercontent.com/9599/150401295-8873fa40-8859-40e3-99d7-e53276319a15.png"" width=""32"" height=""32"">","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017808898,https://api.github.com/repos/simonw/datasette/issues/1603,1017808898,IC_kwDOBm6k_c48qogC,9599,simonw,2022-01-20T18:42:35Z,2022-01-20T18:42:35Z,OWNER,"Resized it down to 208 bytes with https://squoosh.app <img width=""883"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150401693-d19c7fdf-7f5d-49b4-8f1e-5c9e62cbdbed.png""> ![favicon](https://user-images.githubusercontent.com/9599/150401709-11a50492-a8c8-4eee-848c-c813f8cd3e4e.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017842366,https://api.github.com/repos/simonw/datasette/issues/1603,1017842366,IC_kwDOBm6k_c48qwq-,9599,simonw,2022-01-20T19:19:54Z,2022-01-20T19:19:54Z,OWNER,Wrote up a TIL: https://til.simonwillison.net/pixelmator/pixel-editing-favicon,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1608#issuecomment-1017970132,https://api.github.com/repos/simonw/datasette/issues/1608,1017970132,IC_kwDOBm6k_c48rP3U,9599,simonw,2022-01-20T22:08:55Z,2022-01-20T22:08:55Z,OWNER,"Might want to consider SEO here too - I want people from search engines to land on `/stable/`, I wonder if I should noindex or `rel=canonical` the other documentation versions? Not sure what best practices for that is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017969452,https://api.github.com/repos/simonw/datasette/issues/1608,1017969452,IC_kwDOBm6k_c48rPss,9599,simonw,2022-01-20T22:07:55Z,2022-01-20T22:07:55Z,OWNER,"I think I want a banner at the top of the page making it obvious which version the documentation is talking about. This can be pretty low key for the current stable release, but should be visually more prominent for the `/latest/` branch and for older releases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017971905,https://api.github.com/repos/simonw/datasette/issues/1608,1017971905,IC_kwDOBm6k_c48rQTB,9599,simonw,2022-01-20T22:11:40Z,2022-01-20T22:11:40Z,OWNER,"Huh, I had forgotten I already have a banner on older versions: ![D1A65C68-9A37-4FA2-80C4-534739A9D292](https://user-images.githubusercontent.com/9599/150430410-1e22e23f-ed27-4271-89ff-63467eb5f466.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017975322,https://api.github.com/repos/simonw/datasette/issues/1608,1017975322,IC_kwDOBm6k_c48rRIa,9599,simonw,2022-01-20T22:17:01Z,2022-01-20T22:27:07Z,OWNER,"Turns out that banner is something that ReadTheDocs implemented - I found it using GitHub code search, it's produced by this piece of JavaScript: https://github.com/readthedocs/readthedocs.org/blob/0852d7c10d725d954d3e9a93513171baa1116d9f/readthedocs/core/static-src/core/js/doc-embed/version-compare.js#L13-L21 ```javascript function init(data) { var rtd = rtddata.get(); /// Out of date message if (data.is_highest) { return; } var currentURL = window.location.pathname.replace(rtd['version'], data.slug); var warning = $( '<div class=""admonition warning""> ' + '<p class=""first admonition-title"">Note</p> ' + '<p class=""last""> ' + 'You are not reading the most recent version of this documentation. ' + '<a href=""#""></a> is the latest version available.' + '</p>' + '</div>'); warning .find('a') .attr('href', currentURL) .text(data.slug); var body = $(""div.body""); if (!body.length) { body = $(""div.document""); } body.prepend(warning); } ``` And here's where that module is called from the rest of their code: https://github.com/readthedocs/readthedocs.org/blob/bc3e147770e5740314a8e8c33fec5d111c850498/readthedocs/core/static-src/core/js/doc-embed/footer.js#L66-L86","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017981599,https://api.github.com/repos/simonw/datasette/issues/1608,1017981599,IC_kwDOBm6k_c48rSqf,9599,simonw,2022-01-20T22:26:32Z,2022-01-20T22:26:32Z,OWNER,I'm tempted to imitate their JavaScript but check for `/latest/` in the URL and use that to append a similar message warning about this being the documentation for the in-development version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017988556,https://api.github.com/repos/simonw/datasette/issues/1608,1017988556,IC_kwDOBm6k_c48rUXM,9599,simonw,2022-01-20T22:37:51Z,2022-01-20T22:37:51Z,OWNER,"Here's a jQuery recipe that seems to do the right thing: ```javascript jQuery(function ($) { // If this is a /latest/ doc page, show banner linking to /stable/ if (!/\/latest\//.test(location.pathname)) { return; } var stableUrl = location.pathname.replace(""/latest/"", ""/stable/""); // Check it's not a 404 fetch(stableUrl, { method: ""HEAD"" }).then((response) => { if (response.status == 200) { var warning = $( `<div class=""admonition warning""> <p class=""first admonition-title"">Note</p> <p class=""last""> This documentation covers the <strong>development version</strong> of Datasette.</p> <p>See <a href=""${stableUrl}"">this page</a> for the current stable release. </p> </div>` ); warning.find(""a"").attr(""href"", stableUrl); var body = $(""div.body""); if (!body.length) { body = $(""div.document""); } body.prepend(warning); } }); }); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017994925,https://api.github.com/repos/simonw/datasette/issues/1608,1017994925,IC_kwDOBm6k_c48rV6t,9599,simonw,2022-01-20T22:48:43Z,2022-01-20T22:54:02Z,OWNER,"https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for ""You are looking at v0.36 but you should be looking at 0.40"" but doesn't hand the case I need here which is ""you are looking at /latest/ but you should be looking at /stable/"". Just shipped my fix here: https://docs.datasette.io/en/latest/ <img width=""994"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150434433-e2f0d472-77a8-430b-88e5-f22e2c6e8ef0.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017998993,https://api.github.com/repos/simonw/datasette/issues/1608,1017998993,IC_kwDOBm6k_c48rW6R,9599,simonw,2022-01-20T22:56:00Z,2022-01-20T22:56:00Z,OWNER,"> https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for ""You are looking at v0.36 but you should be looking at 0.40"" but doesn't hand the case I need here which is ""you are looking at /latest/ but you should be looking at /stable/"". Correction! That tool DOES support that, as can be seen in their example configuration for their own documentation: https://github.com/humitos/sphinx-version-warning/blob/a82156c2ea08e5feab406514d0ccd9d48a345f48/docs/conf.py#L32-L38 ```python versionwarning_messages = { 'latest': 'This is a custom message only for version ""latest"" of this documentation.', } versionwarning_admonition_type = 'tip' versionwarning_banner_title = 'Tip' versionwarning_body_selector = 'div[itemprop=""articleBody""]' ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1018017637,https://api.github.com/repos/simonw/datasette/issues/1608,1018017637,IC_kwDOBm6k_c48rbdl,9599,simonw,2022-01-20T23:27:59Z,2022-01-20T23:27:59Z,OWNER,"Got a couple of TILs out of this: - [Promoting the stable version of the documentation using rel=canonical](https://til.simonwillison.net/readthedocs/documentation-seo-canonical) - [Linking from /latest/ to /stable/ on Read The Docs](https://til.simonwillison.net/readthedocs/link-from-latest-to-stable)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1609#issuecomment-1018064620,https://api.github.com/repos/simonw/datasette/issues/1609,1018064620,IC_kwDOBm6k_c48rm7s,9599,simonw,2022-01-21T01:00:12Z,2022-01-21T01:00:12Z,OWNER,"I think there are two possible solutions then: 1. Convince Uvicorn to publish one last `0.16.1` version which includes that `python_requires=` line, such that there's a version of Uvicorn on PyPI that Python 3.6 can still install. 2. Release a `0.60.1` version of Datasette which pins that Uvicorn version, and hence can be installed. I've made the request for 1) in Uvicorn Gitter here: https://gitter.im/encode/community?at=61ea044a6d9ba23328d0fa28 I'm going to investigate option 2) myself now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018063681,https://api.github.com/repos/simonw/datasette/issues/1609,1018063681,IC_kwDOBm6k_c48rmtB,9599,simonw,2022-01-21T00:58:25Z,2022-01-21T00:58:32Z,OWNER,"On Twitter: https://twitter.com/simonw/status/1484317711672877065 Here's the problem: Uvicorn only added `python_requires` to their `setup.py` a few days ago, which means the releases they have out on PyPI at the moment don't specify the Python version they need, which is why this mechanism doesn't work as expected: - https://github.com/encode/uvicorn/pull/1328","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018077009,https://api.github.com/repos/simonw/datasette/issues/1609,1018077009,IC_kwDOBm6k_c48rp9R,9599,simonw,2022-01-21T01:24:15Z,2022-01-21T01:24:43Z,OWNER,"Problem: if I ship this, it will be the most recent release of Datasette - but unlike other previous releases it has exactly pinned versions of all of the dependencies. Which is bad for people who run `pip install datasette` but want to not be stuck to those exact library versions. So maybe I ship this as 0.60.1, then ship a 0.60.2 release directly afterwards which unpins the dependencies again and requires Python 3.7?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018075357,https://api.github.com/repos/simonw/datasette/issues/1609,1018075357,IC_kwDOBm6k_c48rpjd,9599,simonw,2022-01-21T01:20:56Z,2022-01-21T01:20:56Z,OWNER,"I used the combo of `pyenv` and `pipenv` to run tests and figure out what the most recent versions of each dependency were that worked on Python 3.6. I also clicked around in the latest releases on pages such as https://pypi.org/project/aiofiles ``` cd /tmp git clone git@github.com:simonw/datasette cd /tmp/datasette pipenv shell --python 3.6.10 pip install -e '.[test]' pytest ``` I also used `pip freeze | grep black` to see which version was installed, since packages with `python_requires=` in them would automatically install the highest compatible version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018082792,https://api.github.com/repos/simonw/datasette/issues/1609,1018082792,IC_kwDOBm6k_c48rrXo,9599,simonw,2022-01-21T01:37:11Z,2022-01-21T01:37:11Z,OWNER,"Another option from https://twitter.com/samuel_hames/status/1484327636860293121 - environment markers, described in https://www.python.org/dev/peps/pep-0508/#environment-markers Found some examples of those in use using GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=%22%3Bpython_version%22+path%3Asetup.py - in particular https://github.com/xmendez/wfuzz/blob/1b695ee9a87d66a7d7bf6cae70d60a33fae51541/setup.py#L31-L38 ```python install_requires = [ 'pycurl', 'pyparsing<2.4.2;python_version<=""3.4""', 'pyparsing>=2.4*;python_version>=""3.5""', 'six', 'configparser;python_version<""3.5""', 'chardet', ] ``` So maybe I can ship 0.60.1 with loose dependencies _except_ for the `uvicorn` one on Python 3.6, using an environment marker. Here's my `setup.py` at the moment: https://github.com/simonw/datasette/blob/ffca55dfd7cc9b53522c2e5a2fa1ff67c9beadf2/setup.py#L44-L61 One other problem: there might be packages in that list right now which don't specify their 3.6 Python version but which will, at some point in the future, release a new version that doesn't work with 3.6 (like Uvicorn did) - in which case Python 3.6 installs would break in the future. I think what I'll do then is ship the `0.60.1` Python 3.6 version with strict upper limits on each version which are the current, tested-with-Datasette-on-Python3.6 versions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018086273,https://api.github.com/repos/simonw/datasette/issues/1609,1018086273,IC_kwDOBm6k_c48rsOB,9599,simonw,2022-01-21T01:45:46Z,2022-01-21T01:45:46Z,OWNER,"This whole thing reminds me of my ongoing internal debate about version pinning: should the Datasette package released to PyPI pin to the exact versions of the dependencies that are known to work, or should it allow a range of dependencies so users can pick other versions of the dependencies to use in their environment? As I understand it, the general rule is to use exact pinning for applications but use ranges for libraries. Datasette is almost entirely an application... but it can also be used as a library - and in fact I'm hoping to encourage that usage more in the future, see: - #1398 I'd also like to release a packaged version of Datasette that doesn't require Uvicorn, for running on AWS Lambda and other function-as-a-service platforms. Those platforms have their own HTTP layer and hence don't need the Uvicorn dependency. Maybe the answer is to have a `datasette-core` package which provides the core of Datasette with unpinned dependencies and no Uvicorn, and then have the existing `datasette` package provide the Datasette CLI tool with Uvicorn and pinned dependencies?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018086697,https://api.github.com/repos/simonw/datasette/issues/1609,1018086697,IC_kwDOBm6k_c48rsUp,9599,simonw,2022-01-21T01:46:43Z,2022-01-21T01:46:43Z,OWNER,https://github.com/simonw/datasette/runs/4890775227?check_suite_focus=true - the tests passed on Python 3.6 for this commit with the pinned dependencies: https://github.com/simonw/datasette/commit/41060e7e7cb838328c879de6a98ae794dc1886d0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018092984,https://api.github.com/repos/simonw/datasette/issues/1609,1018092984,IC_kwDOBm6k_c48rt24,9599,simonw,2022-01-21T02:00:38Z,2022-01-21T02:00:38Z,OWNER,"Out of curiosity, I installed this latest `setup.py` file using both Python 3.6 and Python 3.10, ran `pip freeze` on both of them and created a Gist to compare the difference. The result is here: https://gist.github.com/simonw/2e7d5b1beba675ef9a5bcd310cadc372/revisions From that, it looks like the Python packages in my dependencies which have released new versions that don't work with Python 3.6 are: - https://pypi.org/project/janus/#history - https://pypi.org/project/Pint/#history - https://pypi.org/project/platformdirs/#history - https://pypi.org/project/uvicorn/#history (already discussed) Sure enough, for the first three of those browsing through their recent versions on PyPI confirms that they switched from e.g. ""Requires: Python >=3.6"" on https://pypi.org/project/janus/0.7.0/ to ""Requires: Python >=3.7"" on https://pypi.org/project/janus/1.0.0/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018091322,https://api.github.com/repos/simonw/datasette/issues/1609,1018091322,IC_kwDOBm6k_c48rtc6,9599,simonw,2022-01-21T01:56:42Z,2022-01-21T01:56:42Z,OWNER,"OK, the environment markers approach seems to work!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018094767,https://api.github.com/repos/simonw/datasette/issues/1609,1018094767,IC_kwDOBm6k_c48ruSv,9599,simonw,2022-01-21T02:04:14Z,2022-01-21T02:04:14Z,OWNER,So I don't need to release 0.60.1 AND 0.60.2 after all - I can just release 0.60.1 with a bug fix that it no longer breaks installation for Python 3.6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018104868,https://api.github.com/repos/simonw/datasette/issues/1609,1018104868,IC_kwDOBm6k_c48rwwk,9599,simonw,2022-01-21T02:24:13Z,2022-01-21T02:24:13Z,OWNER,Just shipped 0.60.1 with the fix - and tested that `pip install datasette` does indeed work correctly on Python 3.6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1605#issuecomment-1018766727,https://api.github.com/repos/simonw/datasette/issues/1605,1018766727,IC_kwDOBm6k_c48uSWH,9599,simonw,2022-01-21T18:41:21Z,2022-01-21T18:42:03Z,OWNER,"Yeah I think this all hinges on: - #1101 Also this comment about streaming full JSON arrays (not just newline-delimited) using [this trick](https://til.simonwillison.net/python/output-json-array-streaming): - https://github.com/simonw/datasette/issues/1356#issuecomment-1017016553 I'm about ready to figure these out, as with so much it's still a little bit blocked on the refactor stuff from: - #1518 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1143#issuecomment-1038289584,https://api.github.com/repos/simonw/datasette/issues/1143,1038289584,IC_kwDOBm6k_c494wqw,9599,simonw,2022-02-13T17:40:50Z,2022-02-13T17:41:17Z,OWNER,"The way Drupal does this is interesting; https://www.drupal.org/node/2715637 - it supports the following YAML: ```yaml # Configure Cross-Site HTTP requests (CORS). # Read https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS # for more information about the topic in general. # Note: By default the configuration is disabled. cors.config: enabled: false # Specify allowed headers, like 'x-allowed-header'. allowedHeaders: [] # Specify allowed request methods, specify ['*'] to allow all possible ones. allowedMethods: [] # Configure requests allowed from specific origins. allowedOrigins: ['*'] # Sets the Access-Control-Expose-Headers header. exposedHeaders: false # Sets the Access-Control-Max-Age header. maxAge: false # Sets the Access-Control-Allow-Credentials header. supportsCredentials: false ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices", https://github.com/simonw/datasette/issues/1533#issuecomment-1027633686,https://api.github.com/repos/simonw/datasette/issues/1533,1027633686,IC_kwDOBm6k_c49QHIW,9599,simonw,2022-02-02T06:42:53Z,2022-02-02T06:42:53Z,OWNER,"I'm going to apply the hack, then fix it again in: - #1518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065431383,"Add `Link: rel=""alternate""` header pointing to JSON for a table/query", https://github.com/simonw/datasette/issues/1607#issuecomment-1027634490,https://api.github.com/repos/simonw/datasette/issues/1607,1027634490,IC_kwDOBm6k_c49QHU6,9599,simonw,2022-02-02T06:44:30Z,2022-02-02T06:44:30Z,OWNER,"Prototype: ```diff diff --git a/datasette/app.py b/datasette/app.py index 09d7d03..e2a5aea 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -724,6 +724,47 @@ class Datasette: sqlite_extensions[extension] = None except Exception: pass + # More details on SpatiaLite + if ""spatialite"" in sqlite_extensions: + spatialite_details = {} + fns = ( + ""spatialite_version"", + ""spatialite_target_cpu"", + ""rcheck_strict_sql_quoting"", + ""freexl_version"", + ""proj_version"", + ""geos_version"", + ""rttopo_version"", + ""libxml2_version"", + ""HasIconv"", + ""HasMathSQL"", + ""HasGeoCallbacks"", + ""HasProj"", + ""HasProj6"", + ""HasGeos"", + ""HasGeosAdvanced"", + ""HasGeosTrunk"", + ""HasGeosReentrant"", + ""HasGeosOnlyReentrant"", + ""HasMiniZip"", + ""HasRtTopo"", + ""HasLibXML2"", + ""HasEpsg"", + ""HasFreeXL"", + ""HasGeoPackage"", + ""HasGCP"", + ""HasTopology"", + ""HasKNN"", + ""HasRouting"", + ) + for fn in fns: + try: + result = conn.execute(""select {}()"".format(fn)) + spatialite_details[fn] = result.fetchone()[0] + except Exception: + pass + sqlite_extensions[""spatialite""] = spatialite_details + # Figure out supported FTS versions ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109783030,More detailed information about installed SpatiaLite version, https://github.com/simonw/datasette/issues/1611#issuecomment-1027635175,https://api.github.com/repos/simonw/datasette/issues/1611,1027635175,IC_kwDOBm6k_c49QHfn,9599,simonw,2022-02-02T06:45:47Z,2022-02-02T06:45:47Z,OWNER,"Prototype, not sure that this actually works yet: ```diff diff --git a/datasette/database.py b/datasette/database.py index 6ce8721..0c4aec7 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -256,18 +256,26 @@ class Database: # Try to get counts for each table, $limit timeout for each count counts = {} for table in await self.table_names(): - try: - table_count = ( - await self.execute( - f""select count(*) from [{table}]"", - custom_time_limit=limit, - ) - ).rows[0][0] - counts[table] = table_count - # In some cases I saw ""SQL Logic Error"" here in addition to - # QueryInterrupted - so we catch that too: - except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError): - counts[table] = None + print(table.lower()) + if table.lower() == ""knn"": + counts[table] = 0 + else: + try: + table_count = ( + await self.execute( + f""select count(*) from [{table}]"", + custom_time_limit=limit, + ) + ).rows[0][0] + counts[table] = table_count + # In some cases I saw ""SQL Logic Error"" here in addition to + # QueryInterrupted - so we catch that too: + except ( + QueryInterrupted, + sqlite3.OperationalError, + sqlite3.DatabaseError, + ): + counts[table] = None if not self.is_mutable: self._cached_table_counts = counts return counts ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1113384383,Avoid ever running count(*) against SpatiaLite KNN table, https://github.com/simonw/datasette/issues/1576#issuecomment-1027635925,https://api.github.com/repos/simonw/datasette/issues/1576,1027635925,IC_kwDOBm6k_c49QHrV,9599,simonw,2022-02-02T06:47:20Z,2022-02-02T06:47:20Z,OWNER,"Here's what I was hacking around with when I uncovered this problem: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 77fb285..8c57d08 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import urllib import itertools import json @@ -615,44 +616,37 @@ class TableView(RowTableShared): if request.args.get(""_timelimit""): extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit"")) - # Execute the main query! - results = await db.execute(sql, params, truncate=True, **extra_args) - - # Calculate the total count for this query - filtered_table_rows_count = None - if ( - not db.is_mutable - and self.ds.inspect_data - and count_sql == f""select count(*) from {table} "" - ): - # We can use a previously cached table row count - try: - filtered_table_rows_count = self.ds.inspect_data[database][""tables""][ - table - ][""count""] - except KeyError: - pass - - # Otherwise run a select count(*) ... - if count_sql and filtered_table_rows_count is None and not nocount: - try: - count_rows = list(await db.execute(count_sql, from_sql_params)) - filtered_table_rows_count = count_rows[0][0] - except QueryInterrupted: - pass - - # Faceting - if not self.ds.setting(""allow_facet"") and any( - arg.startswith(""_facet"") for arg in request.args - ): - raise BadRequest(""_facet= is not allowed"") + async def execute_count(): + # Calculate the total count for this query + filtered_table_rows_count = None + if ( + not db.is_mutable + and self.ds.inspect_data + and count_sql == f""select count(*) from {table} "" + ): + # We can use a previously cached table row count + try: + filtered_table_rows_count = self.ds.inspect_data[database][ + ""tables"" + ][table][""count""] + except KeyError: + pass + + if count_sql and filtered_table_rows_count is None and not nocount: + try: + count_rows = list(await db.execute(count_sql, from_sql_params)) + filtered_table_rows_count = count_rows[0][0] + except QueryInterrupted: + pass + + return filtered_table_rows_count + + filtered_table_rows_count = await execute_count() # pylint: disable=no-member facet_classes = list( itertools.chain.from_iterable(pm.hook.register_facet_classes()) ) - facet_results = {} - facets_timed_out = [] facet_instances = [] for klass in facet_classes: facet_instances.append( @@ -668,33 +662,58 @@ class TableView(RowTableShared): ) ) - if not nofacet: - for facet in facet_instances: - ( - instance_facet_results, - instance_facets_timed_out, - ) = await facet.facet_results() - for facet_info in instance_facet_results: - base_key = facet_info[""name""] - key = base_key - i = 1 - while key in facet_results: - i += 1 - key = f""{base_key}_{i}"" - facet_results[key] = facet_info - facets_timed_out.extend(instance_facets_timed_out) - - # Calculate suggested facets - suggested_facets = [] - if ( - self.ds.setting(""suggest_facets"") - and self.ds.setting(""allow_facet"") - and not _next - and not nofacet - and not nosuggest - ): - for facet in facet_instances: - suggested_facets.extend(await facet.suggest()) + async def execute_suggested_facets(): + # Calculate suggested facets + suggested_facets = [] + if ( + self.ds.setting(""suggest_facets"") + and self.ds.setting(""allow_facet"") + and not _next + and not nofacet + and not nosuggest + ): + for facet in facet_instances: + suggested_facets.extend(await facet.suggest()) + return suggested_facets + + async def execute_facets(): + facet_results = {} + facets_timed_out = [] + if not self.ds.setting(""allow_facet"") and any( + arg.startswith(""_facet"") for arg in request.args + ): + raise BadRequest(""_facet= is not allowed"") + + if not nofacet: + for facet in facet_instances: + ( + instance_facet_results, + instance_facets_timed_out, + ) = await facet.facet_results() + for facet_info in instance_facet_results: + base_key = facet_info[""name""] + key = base_key + i = 1 + while key in facet_results: + i += 1 + key = f""{base_key}_{i}"" + facet_results[key] = facet_info + facets_timed_out.extend(instance_facets_timed_out) + + return facet_results, facets_timed_out + + # Execute the main query, facets and facet suggestions in parallel: + ( + results, + suggested_facets, + (facet_results, facets_timed_out), + ) = await asyncio.gather( + db.execute(sql, params, truncate=True, **extra_args), + execute_suggested_facets(), + execute_facets(), + ) + + results = await db.execute(sql, params, truncate=True, **extra_args) # Figure out columns and rows for the query columns = [r[0] for r in results.description] ``` It's a hacky attempt at running some of the table page queries in parallel to see what happens.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1619#issuecomment-1027646659,https://api.github.com/repos/simonw/datasette/issues/1619,1027646659,IC_kwDOBm6k_c49QKTD,9599,simonw,2022-02-02T07:10:37Z,2022-02-02T07:10:37Z,OWNER,It's not just the table with slashes in the name. Same thing on http://127.0.0.1:3344/foo/bar/fixtures/attraction_characteristic/1 - the `json` link goes to a JSON-rendered 404 on http://127.0.0.1:3344/foo/bar/foo/bar/fixtures/attraction_characteristic/1.json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414,JSON link on row page is 404 if base_url setting is used, https://github.com/simonw/datasette/issues/1619#issuecomment-1027647257,https://api.github.com/repos/simonw/datasette/issues/1619,1027647257,IC_kwDOBm6k_c49QKcZ,9599,simonw,2022-02-02T07:11:43Z,2022-02-02T07:11:43Z,OWNER,Weirdly the bug does NOT exhibit itself on this demo: https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1 - which correctly links to https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1.json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414,JSON link on row page is 404 if base_url setting is used, https://github.com/simonw/datasette/issues/1586#issuecomment-1027648180,https://api.github.com/repos/simonw/datasette/issues/1586,1027648180,IC_kwDOBm6k_c49QKq0,9599,simonw,2022-02-02T07:13:31Z,2022-02-02T07:13:31Z,OWNER,"Running it as part of `datasette publish` is a smart idea - I'm slightly nervous about modifying the database file that has been published though, since part of the undocumented contract right now is that the bytes served are the exact same bytes as the ones you ran the publish against. But there's no reason for that expectation to exist, and I doubt anyone is relying on that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096536240,run analyze on all databases as part of start up or publishing, https://github.com/simonw/datasette/issues/1618#issuecomment-1027653005,https://api.github.com/repos/simonw/datasette/issues/1618,1027653005,IC_kwDOBm6k_c49QL2N,9599,simonw,2022-02-02T07:22:13Z,2022-02-02T07:22:13Z,OWNER,"There's a workaround for this at the moment, which is to use parameterized SQL queries. For example, this: https://fivethirtyeight.datasettes.com/polls?sql=select+*+from+books+where+title+%3D+%3Atitle&title=The+Pragmatic+Programmer So the SQL query is `select * from books where title = :title` and then `&title=...` is added to the URL. The reason behind the quite aggressive pragma filtering is that SQLite allows you to execute pragmas using function calls, like this one: ```sql SELECT * FROM pragma_index_info('idx52'); ``` These can be nested arbitrarily deeply in sub-queries, so it's difficult to write a regular expression that will definitely catch them. I'm open to relaxing the regex a bit, but I need to be very confident that it's safe to do so. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1618#issuecomment-1027654979,https://api.github.com/repos/simonw/datasette/issues/1618,1027654979,IC_kwDOBm6k_c49QMVD,9599,simonw,2022-02-02T07:25:22Z,2022-02-02T07:25:22Z,OWNER,"But... I just noticed something I had missed in the docs for https://www.sqlite.org/pragma.html#pragfunc > Table-valued functions exist only for PRAGMAs that return results and that have no side-effects. So it's possible I'm being overly paranoid here after all: what I want to block here is people running things like `PRAGMA case_sensitive_like = 1` which could affect the global state for that connection and cause unexpected behaviour later on. So maybe I should allow all pragma functions. I previously allowed an allow-list of them in: - #761","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1618#issuecomment-1027656000,https://api.github.com/repos/simonw/datasette/issues/1618,1027656000,IC_kwDOBm6k_c49QMlA,9599,simonw,2022-02-02T07:27:14Z,2022-02-02T07:27:14Z,OWNER,"I also just realized that `pragma pragma_list` can be used to generate a list of all known pragmas for the connection: sqlite-utils fixtures.db 'pragma pragma_list' --fmt github | name | |---------------------------| | analysis_limit | | application_id | | auto_vacuum | | automatic_index | | busy_timeout | | cache_size | | cache_spill | | case_sensitive_like | | cell_size_check | | checkpoint_fullfsync | | collation_list | | compile_options | | count_changes | | data_version | | database_list | | default_cache_size | | defer_foreign_keys | | empty_result_callbacks | | encoding | | foreign_key_check | | foreign_key_list | | foreign_keys | | freelist_count | | full_column_names | | fullfsync | | function_list | | hard_heap_limit | | ignore_check_constraints | | incremental_vacuum | | index_info | | index_list | | index_xinfo | | integrity_check | | journal_mode | | journal_size_limit | | legacy_alter_table | | lock_proxy_file | | locking_mode | | max_page_count | | mmap_size | | module_list | | optimize | | page_count | | page_size | | pragma_list | | query_only | | quick_check | | read_uncommitted | | recursive_triggers | | reverse_unordered_selects | | schema_version | | secure_delete | | short_column_names | | shrink_memory | | soft_heap_limit | | synchronous | | table_info | | table_list | | table_xinfo | | temp_store | | temp_store_directory | | threads | | trusted_schema | | user_version | | wal_autocheckpoint | | wal_checkpoint | | writable_schema | So I could use that list to create a much more specific regular expression, which would then allow the word ""pragma"" to be used more freely while still protecting against any known pragma function being called.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1618#issuecomment-1027656518,https://api.github.com/repos/simonw/datasette/issues/1618,1027656518,IC_kwDOBm6k_c49QMtG,9599,simonw,2022-02-02T07:28:14Z,2022-02-02T07:31:30Z,OWNER,"I also need to consider if supposedly harmless side-effect free pragma functions could be used to work around the Datasette permissions system. My hunch is that wouldn't be a problem, because if you're allowing arbitrary SQL queries you're already letting people ignore the permissions system. One example: ``` sqlite-utils fixtures.db 'pragma database_list' -t seq name file ----- ------ ------------------------------------------------------ 0 main /Users/simon/Dropbox/Development/datasette/fixtures.db ``` Though it looks like I already allow-listed that one in #761: https://latest.datasette.io/_memory?sql=select+*+from+pragma_database_list%28%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1618#issuecomment-1027659018,https://api.github.com/repos/simonw/datasette/issues/1618,1027659018,IC_kwDOBm6k_c49QNUK,9599,simonw,2022-02-02T07:32:47Z,2022-02-02T07:32:47Z,OWNER,"I was hoping that `explain select ...` might be able to easily spot when people are calling PRAGMA functions, but this output doesn't look very helpful: ``` % sqlite-utils fixtures.db 'explain select * from pragma_database_list()' -t addr opcode p1 p2 p3 p4 p5 comment ------ ----------- ---- ---- ---- ----------------- ---- --------- 0 Init 0 11 0 0 1 VOpen 0 0 0 vtab:7F9C90AC3070 0 2 Integer 0 1 0 0 3 Integer 0 2 0 0 4 VFilter 0 10 1 0 5 VColumn 0 0 3 0 6 VColumn 0 1 4 0 7 VColumn 0 2 5 0 8 ResultRow 3 3 0 0 9 VNext 0 5 0 0 10 Halt 0 0 0 0 11 Transaction 0 0 35 0 1 12 Goto 0 1 0 0 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1615#issuecomment-1027659890,https://api.github.com/repos/simonw/datasette/issues/1615,1027659890,IC_kwDOBm6k_c49QNhy,9599,simonw,2022-02-02T07:34:17Z,2022-02-02T07:34:17Z,OWNER,"I've been thinking about this a bunch too. If I build anything along these lines it will be as part of the Datasette Cloud hosted service I'm working on, maybe as a free tier.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1117132741,Potential simplified publishing mechanism, https://github.com/simonw/datasette/issues/1533#issuecomment-1027669851,https://api.github.com/repos/simonw/datasette/issues/1533,1027669851,IC_kwDOBm6k_c49QP9b,9599,simonw,2022-02-02T07:51:57Z,2022-02-02T07:51:57Z,OWNER,"Documentation: https://docs.datasette.io/en/latest/json_api.html#discovering-the-json-for-a-page https://docs.datasette.io/en/latest/json_api.html top `--cors` section mentions the new `Access-Control-Expose-Headers: Link` header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065431383,"Add `Link: rel=""alternate""` header pointing to JSON for a table/query", https://github.com/simonw/datasette/issues/1533#issuecomment-1027672617,https://api.github.com/repos/simonw/datasette/issues/1533,1027672617,IC_kwDOBm6k_c49QQop,9599,simonw,2022-02-02T07:56:51Z,2022-02-02T07:56:51Z,OWNER,"Demos - these pages both have `<link rel=...` if you view source on them: - https://latest.datasette.io/fixtures/sortable - https://latest.datasette.io/fixtures/sortable/a,a And you can hit them with `curl` like so: ``` % curl -I 'https://latest.datasette.io/fixtures/sortable' HTTP/1.1 200 OK link: https://latest.datasette.io/fixtures/sortable.json; rel=""alternate""; type=""application/json+datasette"" cache-control: max-age=5 referrer-policy: no-referrer access-control-allow-origin: * access-control-allow-headers: Authorization access-control-expose-headers: Link content-type: text/html; charset=utf-8 x-databases: _memory, _internal, fixtures, extra_database Date: Wed, 02 Feb 2022 07:56:17 GMT Server: Google Frontend Transfer-Encoding: chunked % curl -I 'https://latest.datasette.io/fixtures/sortable/a,a' HTTP/1.1 200 OK link: https://latest.datasette.io/fixtures/sortable/a,a.json; rel=""alternate""; type=""application/json+datasette"" cache-control: max-age=5 referrer-policy: no-referrer access-control-allow-origin: * access-control-allow-headers: Authorization access-control-expose-headers: Link content-type: text/html; charset=utf-8 x-databases: _memory, _internal, fixtures, extra_database Date: Wed, 02 Feb 2022 07:56:24 GMT Server: Google Frontend Transfer-Encoding: chunked ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065431383,"Add `Link: rel=""alternate""` header pointing to JSON for a table/query", https://github.com/simonw/datasette/issues/1620#issuecomment-1028385067,https://api.github.com/repos/simonw/datasette/issues/1620,1028385067,IC_kwDOBm6k_c49S-kr,9599,simonw,2022-02-02T21:42:23Z,2022-02-02T21:42:23Z,OWNER,"``` % curl -s -I 'https://latest.datasette.io/' | grep link link: https://latest.datasette.io/.json; rel=""alternate""; type=""application/json+datasette"" % curl -s -I 'https://latest.datasette.io/fixtures' | grep link link: https://latest.datasette.io/fixtures.json; rel=""alternate""; type=""application/json+datasette"" % curl -s -I 'https://latest.datasette.io/fixtures?sql=select+1' | grep link link: https://latest.datasette.io/fixtures.json?sql=select+1; rel=""alternate""; type=""application/json+datasette"" % curl -s -I 'https://latest.datasette.io/-/plugins' | grep link link: https://latest.datasette.io/-/plugins.json; rel=""alternate""; type=""application/json+datasette"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041,"Link: rel=""alternate"" to JSON for queries too", https://github.com/simonw/datasette/issues/1620#issuecomment-1028374330,https://api.github.com/repos/simonw/datasette/issues/1620,1028374330,IC_kwDOBm6k_c49S786,9599,simonw,2022-02-02T21:28:16Z,2022-02-02T21:28:16Z,OWNER,I just realized I can refactor this to make it much simpler.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041,"Link: rel=""alternate"" to JSON for queries too", https://github.com/simonw/datasette/issues/1623#issuecomment-1028389953,https://api.github.com/repos/simonw/datasette/issues/1623,1028389953,IC_kwDOBm6k_c49S_xB,9599,simonw,2022-02-02T21:48:34Z,2022-02-02T21:48:34Z,OWNER,"A few other pages do that too, including: - https://latest.datasette.io/-/messages - https://latest.datasette.io/-/allow-debug","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122416919,/-/patterns returns link: alternate JSON header to 404, https://github.com/simonw/datasette/issues/1620#issuecomment-1028393259,https://api.github.com/repos/simonw/datasette/issues/1620,1028393259,IC_kwDOBm6k_c49TAkr,9599,simonw,2022-02-02T21:53:02Z,2022-02-02T21:53:02Z,OWNER,"I ran the following on https://www.google.com/ in the console to demonstrate that these work as intended: ```javascript [ ""https://latest.datasette.io/fixtures"", ""https://latest.datasette.io/fixtures?sql=select+1"", ""https://latest.datasette.io/fixtures/facetable"" ].forEach(async (url) => { response = await fetch(url, {method: ""HEAD""}); console.log(response.headers.get(""Link"")); }); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041,"Link: rel=""alternate"" to JSON for queries too", https://github.com/simonw/datasette/issues/1624#issuecomment-1028396866,https://api.github.com/repos/simonw/datasette/issues/1624,1028396866,IC_kwDOBm6k_c49TBdC,9599,simonw,2022-02-02T21:58:06Z,2022-02-02T21:58:06Z,OWNER,"It looks like this is because `IndexView` extends `BaseView` rather than extending `DataView` which is where all that CORS stuff happens: https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/datasette/views/index.py#L18-L21 Another thing I should address with the refactor project in: - #878 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122427321,Index page `/` has no CORS headers, https://github.com/simonw/datasette/issues/1623#issuecomment-1028397935,https://api.github.com/repos/simonw/datasette/issues/1623,1028397935,IC_kwDOBm6k_c49TBtv,9599,simonw,2022-02-02T21:59:43Z,2022-02-02T21:59:43Z,OWNER,Here's the new test: https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/tests/test_html.py#L927-L936,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122416919,/-/patterns returns link: alternate JSON header to 404, https://github.com/simonw/datasette/pull/1616#issuecomment-1028414871,https://api.github.com/repos/simonw/datasette/issues/1616,1028414871,IC_kwDOBm6k_c49TF2X,9599,simonw,2022-02-02T22:23:45Z,2022-02-02T22:23:45Z,OWNER,First stable Black release!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1119413338,Bump black from 21.12b0 to 22.1.0, https://github.com/simonw/datasette/pull/1626#issuecomment-1028420821,https://api.github.com/repos/simonw/datasette/issues/1626,1028420821,IC_kwDOBm6k_c49THTV,9599,simonw,2022-02-02T22:32:26Z,2022-02-02T22:33:31Z,OWNER,"That broke on a macOS test: https://github.com/simonw/datasette/runs/5044036993?check_suite_focus=true I'm going to remove macOS and Ubuntu and just try Windows purely to see what happens there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122451096,Try test suite against macOS and Windows, https://github.com/simonw/datasette/issues/1534#issuecomment-1028461220,https://api.github.com/repos/simonw/datasette/issues/1534,1028461220,IC_kwDOBm6k_c49TRKk,9599,simonw,2022-02-02T23:39:33Z,2022-02-02T23:39:33Z,OWNER,"I've decided not to do this, because of the risk that Cloudflare could cache the JSON version for an HTML page or vice-versa.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/issues/1545#issuecomment-1028517268,https://api.github.com/repos/simonw/datasette/issues/1545,1028517268,IC_kwDOBm6k_c49Te2U,9599,simonw,2022-02-03T01:26:53Z,2022-02-03T01:26:53Z,OWNER,"I understand the problem now! https://github.com/pallets/jinja/issues/1378#issuecomment-812410922 > Jinja template names/paths are not always filesystem paths. So regardless of the OS Jinja always uses forward slashes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249,Custom pages don't work on windows, https://github.com/simonw/datasette/pull/1626#issuecomment-1028515161,https://api.github.com/repos/simonw/datasette/issues/1626,1028515161,IC_kwDOBm6k_c49TeVZ,9599,simonw,2022-02-03T01:22:43Z,2022-02-03T01:22:43Z,OWNER,"OK, the tests do NOT pass against Windows! https://github.com/simonw/datasette/runs/5044105941 <img width=""767"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/152265501-5b5af51b-0fae-4aa7-b86d-7b61aba24b3c.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122451096,Try test suite against macOS and Windows, https://github.com/simonw/datasette/pull/1617#issuecomment-1028517073,https://api.github.com/repos/simonw/datasette/issues/1617,1028517073,IC_kwDOBm6k_c49TezR,9599,simonw,2022-02-03T01:26:32Z,2022-02-03T01:26:32Z,OWNER,"Aha I understand the problem now! https://github.com/pallets/jinja/issues/1378#issuecomment-812410922 > Jinja template names/paths are not always filesystem paths. So regardless of the OS Jinja always uses forward slashes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1120990806,"Ensure template_path always uses ""/"" to match jinja", https://github.com/simonw/datasette/pull/1617#issuecomment-1028519382,https://api.github.com/repos/simonw/datasette/issues/1617,1028519382,IC_kwDOBm6k_c49TfXW,9599,simonw,2022-02-03T01:31:25Z,2022-02-03T01:31:25Z,OWNER,"I was hoping to get the test suite running on Windows before merging this PR but that looks like it will be a BIG job, see: - #1627 So I'm going to merge this one as-is for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1120990806,"Ensure template_path always uses ""/"" to match jinja", https://github.com/simonw/datasette/issues/1545#issuecomment-1028535868,https://api.github.com/repos/simonw/datasette/issues/1545,1028535868,IC_kwDOBm6k_c49TjY8,9599,simonw,2022-02-03T02:08:30Z,2022-02-03T02:08:30Z,OWNER,"Filed an issue with Jinja suggesting a documentation update: - https://github.com/pallets/jinja/issues/1578","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249,Custom pages don't work on windows, https://github.com/simonw/datasette/issues/1080#issuecomment-1029695083,https://api.github.com/repos/simonw/datasette/issues/1080,1029695083,IC_kwDOBm6k_c49X-Zr,9599,simonw,2022-02-04T06:24:40Z,2022-02-04T06:25:18Z,OWNER,"An initial prototype of that in my local `group-count` branch quickly started running into problems: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index be9e9c3..d30efe1 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -105,8 +105,12 @@ class RowTableShared(DataView): type_ = ""integer"" notnull = 0 else: - type_ = column_details[r[0]].type - notnull = column_details[r[0]].notnull + try: + type_ = column_details[r[0]].type + notnull = column_details[r[0]].notnull + except KeyError: # Probably count(*) + type_ = ""integer"" + notnull = False columns.append( { ""name"": r[0], @@ -613,6 +617,15 @@ class TableView(RowTableShared): offset=offset, ) + # If ?_group_count we convert the SQL query here + group_count = request.args.getlist(""_group_count"") + if group_count: + wrapped_sql = ""select {cols}, count(*) from ({sql}) group by {cols}"".format( + cols="", "".join(group_count), + sql=sql, + ) + sql = wrapped_sql + if request.args.get(""_timelimit""): extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit"")) ``` Resulted in errors like this one: ``` pk_path = path_from_row_pks(row, pks, not pks, False) File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 82, in path_from_row_pks bits = [ File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 83, in <listcomp> row[pk][""value""] if isinstance(row[pk], dict) else row[pk] for pk in pks IndexError: No item with that key ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them", https://github.com/simonw/datasette/issues/1080#issuecomment-1029691693,https://api.github.com/repos/simonw/datasette/issues/1080,1029691693,IC_kwDOBm6k_c49X9kt,9599,simonw,2022-02-04T06:16:45Z,2022-02-04T06:16:45Z,OWNER,"Had a new, different idea for how this could work: support a `?_group_count=colname` parameter to the table view, which turns the page into a `select colname, count(*) ... group by colname` query - but keeps things like the filter interface, facet selection, search box and so on.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them", https://github.com/simonw/datasette/issues/1576#issuecomment-1030528532,https://api.github.com/repos/simonw/datasette/issues/1576,1030528532,IC_kwDOBm6k_c49bJ4U,9599,simonw,2022-02-05T05:09:57Z,2022-02-05T05:09:57Z,OWNER,Needs documentation. I'll document `from datasette.tracer import trace` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-1030525218,https://api.github.com/repos/simonw/datasette/issues/1576,1030525218,IC_kwDOBm6k_c49bJEi,9599,simonw,2022-02-05T04:45:11Z,2022-02-05T04:45:11Z,OWNER,"Got a prototype working with `contextvars` - it identified two parallel executing queries using the patch from above: ![CleanShot 2022-02-04 at 20 41 50@2x](https://user-images.githubusercontent.com/9599/152628949-cf766b13-13cf-4831-b48d-2f23cadb6a05.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-1030530071,https://api.github.com/repos/simonw/datasette/issues/1576,1030530071,IC_kwDOBm6k_c49bKQX,9599,simonw,2022-02-05T05:21:35Z,2022-02-05T05:21:35Z,OWNER,New documentation section: https://docs.datasette.io/en/latest/internals.html#datasette-tracer,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/957#issuecomment-1030762140,https://api.github.com/repos/simonw/datasette/issues/957,1030762140,IC_kwDOBm6k_c49cC6c,9599,simonw,2022-02-06T06:36:41Z,2022-02-06T06:36:41Z,OWNER,Documented here: https://docs.datasette.io/en/latest/internals.html#import-shortcuts,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes, https://github.com/simonw/datasette/issues/1176#issuecomment-1030762279,https://api.github.com/repos/simonw/datasette/issues/1176,1030762279,IC_kwDOBm6k_c49cC8n,9599,simonw,2022-02-06T06:38:08Z,2022-02-06T06:41:37Z,OWNER,"Might do this using Sphinx auto-generated function and class documentation hooks, as seen here in `sqlite-utils`: https://sqlite-utils.datasette.io/en/stable/python-api.html#spatialite-helpers This would encourage me to add really good docstrings. ``` .. _python_api_gis_find_spatialite: Finding SpatiaLite ------------------ .. autofunction:: sqlite_utils.utils.find_spatialite ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/957#issuecomment-1030761625,https://api.github.com/repos/simonw/datasette/issues/957,1030761625,IC_kwDOBm6k_c49cCyZ,9599,simonw,2022-02-06T06:30:32Z,2022-02-06T06:31:44Z,OWNER,"I'm just going with: ```python from datasette import Response from datasette import Forbidden from datasette import NotFound from datasette import hookimpl from datasette import actor_matches_allow ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes, https://github.com/simonw/datasette/issues/932#issuecomment-1030940407,https://api.github.com/repos/simonw/datasette/issues/932,1030940407,IC_kwDOBm6k_c49cub3,9599,simonw,2022-02-06T23:31:22Z,2022-02-06T23:31:22Z,OWNER,"Great argument for doing this from a conversation on Twitter about documentation-driven development: > Long ago, when the majority of commercial programs were desktop apps, I've read a very wise advice: The user manual should be written first, before even a single line if code. https://twitter.com/b11c/status/1490466703175823362","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation, https://github.com/simonw/datasette/issues/1176#issuecomment-1031108559,https://api.github.com/repos/simonw/datasette/issues/1176,1031108559,IC_kwDOBm6k_c49dXfP,9599,simonw,2022-02-07T06:11:27Z,2022-02-07T06:11:27Z,OWNER,I'm going with `@documented` as the decorator for functions that should be documented.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031122800,https://api.github.com/repos/simonw/datasette/issues/1176,1031122800,IC_kwDOBm6k_c49da9w,9599,simonw,2022-02-07T06:34:21Z,2022-02-07T06:34:21Z,OWNER,"New section is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module But it's not correctly displaying the new autodoc stuff: <img width=""688"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/152736915-e6470e0a-05e2-4612-a2ff-ebd4f51b65cb.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031126547,https://api.github.com/repos/simonw/datasette/issues/1176,1031126547,IC_kwDOBm6k_c49db4T,9599,simonw,2022-02-07T06:42:58Z,2022-02-07T06:42:58Z,OWNER,"That fixed it: https://docs.datasette.io/en/latest/internals.html#parse-metadata-content <img width=""695"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/152737872-595e9a14-1a0d-47ff-9509-63ee765c0c5d.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031126801,https://api.github.com/repos/simonw/datasette/issues/1176,1031126801,IC_kwDOBm6k_c49db8R,9599,simonw,2022-02-07T06:43:31Z,2022-02-07T06:43:31Z,OWNER,Here's the new test: https://github.com/simonw/datasette/blob/03305ea183b1534bc4cef3a721fe5f3700273b84/tests/test_docs.py#L91-L104,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031123719,https://api.github.com/repos/simonw/datasette/issues/1176,1031123719,IC_kwDOBm6k_c49dbMH,9599,simonw,2022-02-07T06:36:32Z,2022-02-07T06:36:32Z,OWNER,"https://github.com/simonw/sqlite-utils/blob/main/.readthedocs.yaml looks like this (it works correctly): ```yaml version: 2 sphinx: configuration: docs/conf.py python: version: ""3.8"" install: - method: pip path: . extra_requirements: - docs ``` Compare to the current Datasette one here: https://github.com/simonw/datasette/blob/d9b508ffaa91f9f1840b366f5d282712d445f16b/.readthedocs.yaml#L1-L13 Looks like I need this bit: ```python python: version: ""3.8"" install: - method: pip path: . extra_requirements: - docs ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031125347,https://api.github.com/repos/simonw/datasette/issues/1176,1031125347,IC_kwDOBm6k_c49dblj,9599,simonw,2022-02-07T06:40:16Z,2022-02-07T06:40:16Z,OWNER,"Read The Docs error: > Problem in your project's configuration. Invalid ""python.version"": .readthedocs.yaml: Invalid configuration option: python.version. Make sure the key name is correct.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1439#issuecomment-1031141849,https://api.github.com/repos/simonw/datasette/issues/1439,1031141849,IC_kwDOBm6k_c49dfnZ,9599,simonw,2022-02-07T07:11:11Z,2022-02-07T07:11:11Z,OWNER,"I added a Link header to solve this problem for the JSON version in: - #1533 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1632#issuecomment-1032032686,https://api.github.com/repos/simonw/datasette/issues/1632,1032032686,IC_kwDOBm6k_c49g5Gu,9599,simonw,2022-02-07T23:16:10Z,2022-02-07T23:16:10Z,OWNER,"I found this bug while trying to get the following to work: datasette /data/one.db /data/two.db /data/*.db --create I want this to create any missing database files on startup out of that literal list of `one.db` and `two.db` and to also open any other `*.db` files in that folder - needed for `datasette-publish-fly` in https://github.com/simonw/datasette-publish-fly/pull/12#issuecomment-1032029874","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032034015,https://api.github.com/repos/simonw/datasette/issues/1632,1032034015,IC_kwDOBm6k_c49g5bf,9599,simonw,2022-02-07T23:17:57Z,2022-02-07T23:17:57Z,OWNER,I'm going to fix this in a 0.60.2 bug fix release.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032036525,https://api.github.com/repos/simonw/datasette/issues/1632,1032036525,IC_kwDOBm6k_c49g6Ct,9599,simonw,2022-02-07T23:19:59Z,2022-02-07T23:19:59Z,OWNER,"I'm going to fix this in the CLI code itself, rather than fixing it in the `Datasette` constructor. That way if someone has a truly weird reason to want this behaviour they can construct Datasette directly. https://github.com/simonw/datasette/blob/03305ea183b1534bc4cef3a721fe5f3700273b84/datasette/cli.py#L535-L550","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032037391,https://api.github.com/repos/simonw/datasette/issues/1632,1032037391,IC_kwDOBm6k_c49g6QP,9599,simonw,2022-02-07T23:21:07Z,2022-02-07T23:21:07Z,OWNER,"For the record, here's the code that picks the `one_2` name if that stem is already used as a database name: https://github.com/simonw/datasette/blob/03305ea183b1534bc4cef3a721fe5f3700273b84/datasette/app.py#L401-L417","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032050489,https://api.github.com/repos/simonw/datasette/issues/1632,1032050489,IC_kwDOBm6k_c49g9c5,9599,simonw,2022-02-07T23:39:11Z,2022-02-07T23:42:08Z,OWNER,"That implementation broke on Python 3.6 - which is still a supported Python version for the 0.60.x branch - `test_homepage` failed. ``` > assert ( ""2 rows in 1 table, 5 rows in 4 hidden tables, 1 view"" == counts_p.text.strip() ) E AssertionError: assert '2 rows in 1 ...ables, 1 view' == '1 table, 4 h...ables, 1 view' E - 1 table, 4 hidden tables, 1 view E + 2 rows in 1 table, 5 rows in 4 hidden tables, 1 view E ? ++++++++++ ++++++++++ ``` That's because this idiom isn't guaranteed to preserve order in versions earlier than Python 3.7: https://github.com/simonw/datasette/blob/fa5fc327adbbf70656ac533912f3fc0526a3873d/datasette/cli.py#L552-L553 I could say that `0.60.2` is the first version to require Python 3.7 - but that feels a little surprising. I'm going to use a different idiom for order-preserving de-duplication from [this StackOverflow](https://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-whilst-preserving-order) instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032057472,https://api.github.com/repos/simonw/datasette/issues/1632,1032057472,IC_kwDOBm6k_c49g_KA,9599,simonw,2022-02-07T23:50:01Z,2022-02-07T23:50:01Z,OWNER,Released in https://github.com/simonw/datasette/releases/tag/0.60.2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1607#issuecomment-1033403664,https://api.github.com/repos/simonw/datasette/issues/1607,1033403664,IC_kwDOBm6k_c49mH0Q,9599,simonw,2022-02-09T06:42:02Z,2022-02-09T06:42:02Z,OWNER,"Deployed a new build of https://github.com/simonw/calands-datasette/actions/workflows/build-and-deploy.yml for a live demo: https://calands.datasettes.com/-/versions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109783030,More detailed information about installed SpatiaLite version, https://github.com/simonw/datasette/issues/1634#issuecomment-1035664928,https://api.github.com/repos/simonw/datasette/issues/1634,1035664928,IC_kwDOBm6k_c49uv4g,9599,simonw,2022-02-11T00:10:07Z,2022-02-11T00:10:23Z,OWNER,Could also bump this up to Python 3.10: https://github.com/simonw/datasette/blob/5619069968ab39fd44c44a1888965e361c6f7fb9/Dockerfile#L1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1131295060,Update Dockerfile generated by `datasette publish`, https://github.com/simonw/datasette/issues/1634#issuecomment-1035664412,https://api.github.com/repos/simonw/datasette/issues/1634,1035664412,IC_kwDOBm6k_c49uvwc,9599,simonw,2022-02-11T00:09:18Z,2022-02-11T00:09:18Z,OWNER,Starting it with `FROM datasetteproject/datasette` might be a good idea. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1131295060,Update Dockerfile generated by `datasette publish`, https://github.com/simonw/datasette/issues/1634#issuecomment-1035667060,https://api.github.com/repos/simonw/datasette/issues/1634,1035667060,IC_kwDOBm6k_c49uwZ0,9599,simonw,2022-02-11T00:13:22Z,2022-02-11T00:13:22Z,OWNER,Looks like `3.10.2` is the latest: https://hub.docker.com/_/python?tab=tags&page=1&name=3.10.2-slim-bu,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1131295060,Update Dockerfile generated by `datasette publish`, https://github.com/simonw/datasette/issues/1660#issuecomment-1073355032,https://api.github.com/repos/simonw/datasette/issues/1660,1073355032,IC_kwDOBm6k_c4_-hkY,9599,simonw,2022-03-20T21:46:43Z,2022-03-20T21:46:43Z,OWNER,I think the way to get rid of most of the remaining complexity in `DataView` is to refactor how CSV stuff works - pulling it in line with other export factors and extracting the streaming mechanism. Opening a fresh issue for that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1672#issuecomment-1073355818,https://api.github.com/repos/simonw/datasette/issues/1672,1073355818,IC_kwDOBm6k_c4_-hwq,9599,simonw,2022-03-20T21:52:38Z,2022-03-20T21:52:38Z,OWNER,"That means taking on these issues: - https://github.com/simonw/datasette/issues/1101 - https://github.com/simonw/datasette/issues/1096 - https://github.com/simonw/datasette/issues/1062","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174697144,Refactor CSV handling code out of DataView, https://github.com/simonw/datasette/issues/1673#issuecomment-1073361986,https://api.github.com/repos/simonw/datasette/issues/1673,1073361986,IC_kwDOBm6k_c4_-jRC,9599,simonw,2022-03-20T22:31:41Z,2022-03-20T22:34:06Z,OWNER,"Maybe it's because `supports_table_xinfo()` creates a brand new in-memory SQLite connection every time you call it? https://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/sqlite.py#L22-L35 Actually no, I'm caching that already: https://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/sqlite.py#L12-L19","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174708375,Streaming CSV spends a lot of time in `table_column_details`, https://github.com/simonw/datasette/issues/1355#issuecomment-1073362979,https://api.github.com/repos/simonw/datasette/issues/1355,1073362979,IC_kwDOBm6k_c4_-jgj,9599,simonw,2022-03-20T22:38:53Z,2022-03-20T22:38:53Z,OWNER,"Built a research prototype: ```diff diff --git a/datasette/app.py b/datasette/app.py index 5c8101a..5cd3e63 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1,6 +1,7 @@ import asyncio import asgi_csrf import collections +import contextlib import datetime import functools import glob @@ -1490,3 +1491,11 @@ class DatasetteClient: return await client.request( method, self._fix(path, avoid_path_rewrites), **kwargs ) + + @contextlib.asynccontextmanager + async def stream(self, method, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + print(""async with as client"") + async with client.stream(method, self._fix(path), **kwargs) as response: + print(""async with client.stream about to yield response"") + yield response diff --git a/datasette/cli.py b/datasette/cli.py index 3c6e1b2..3025ead 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -585,11 +585,19 @@ def serve( asyncio.get_event_loop().run_until_complete(check_databases(ds)) if get: - client = TestClient(ds) - response = client.get(get) - click.echo(response.text) - exit_code = 0 if response.status == 200 else 1 - sys.exit(exit_code) + + async def _run_get(): + print(""_run_get"") + async with ds.client.stream(""GET"", get) as response: + print(""Got response:"", response) + async for chunk in response.aiter_bytes(chunk_size=1024): + print("" chunk"") + sys.stdout.buffer.write(chunk) + sys.stdout.buffer.flush() + exit_code = 0 if response.status_code == 200 else 1 + sys.exit(exit_code) + + asyncio.get_event_loop().run_until_complete(_run_get()) return # Start the server ``` But for some reason it didn't appear to stream out the response - it would print this out: ``` % datasette covid.db --get '/covid/ny_times_us_counties.csv?_size=10&_stream=on' _run_get async with as client ``` And then hang. I would expect it to start printing out chunks of CSV data here, but instead it looks like it waited for everything to be generated before returning anything to the console. No idea why. I dropped this for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910088936,datasette --get should efficiently handle streaming CSV, https://github.com/simonw/datasette/issues/1674#issuecomment-1073366436,https://api.github.com/repos/simonw/datasette/issues/1674,1073366436,IC_kwDOBm6k_c4_-kWk,9599,simonw,2022-03-20T22:58:40Z,2022-03-20T22:58:40Z,OWNER,"This will probably happen as part of turning this into an officially documented API that serves the template context for the homepage: - #1510","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174717287,Tweak design of /.json, https://github.com/simonw/datasette/issues/1510#issuecomment-1073366630,https://api.github.com/repos/simonw/datasette/issues/1510,1073366630,IC_kwDOBm6k_c4_-kZm,9599,simonw,2022-03-20T22:59:33Z,2022-03-20T22:59:33Z,OWNER,"I really like the idea of making this effectively the same thing as the fully documented, stable JSON API that comes as part of 1.0. If you want to know what will be available to your templates, consult the API documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054244712,Datasette 1.0 documented template context (maybe via API docs), https://github.com/simonw/datasette/issues/1663#issuecomment-1071519407,https://api.github.com/repos/simonw/datasette/issues/1663,1071519407,IC_kwDOBm6k_c4_3hav,9599,simonw,2022-03-17T21:32:35Z,2022-03-17T21:32:35Z,OWNER,"Updated docs: - https://docs.datasette.io/en/latest/internals.html#datasette-class - https://docs.datasette.io/en/latest/internals.html#db-hash","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170554975,Document the internals that were used in datasette-hashed-urls, https://github.com/simonw/datasette/issues/1661#issuecomment-1071706993,https://api.github.com/repos/simonw/datasette/issues/1661,1071706993,IC_kwDOBm6k_c4_4PNx,9599,simonw,2022-03-17T22:42:21Z,2022-03-17T22:42:21Z,OWNER,"As part of this I'm going to get rid of this mechanism: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/base.py#L170-L173 Unwrapping `request.scope[""url_route""][""kwargs""]` into keyword argument to view functions just made the code harder to follow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1071793307,https://api.github.com/repos/simonw/datasette/issues/1661,1071793307,IC_kwDOBm6k_c4_4kSb,9599,simonw,2022-03-17T23:17:32Z,2022-03-17T23:17:32Z,OWNER,"Surprisingly I managed to break https://latest.datasette.io/fixtures/custom_foreign_key_label while working on this change: ![CleanShot 2022-03-17 at 16 16 54@2x](https://user-images.githubusercontent.com/9599/158909271-717b65e8-cfcc-44c4-b1cc-f34478b0f803.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1071797707,https://api.github.com/repos/simonw/datasette/issues/1661,1071797707,IC_kwDOBm6k_c4_4lXL,9599,simonw,2022-03-17T23:19:24Z,2022-03-17T23:19:24Z,OWNER,"Moving this to PR so I can comment on individual lines: - #1664","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1071803114,https://api.github.com/repos/simonw/datasette/issues/1664,1071803114,IC_kwDOBm6k_c4_4mrq,9599,simonw,2022-03-17T23:22:00Z,2022-03-17T23:22:00Z,OWNER,"Surprisingly I managed to break https://latest.datasette.io/fixtures/custom_foreign_key_label while working on this change: ![CleanShot 2022-03-17 at 16 16 54@2x](https://user-images.githubusercontent.com/9599/158909271-717b65e8-cfcc-44c4-b1cc-f34478b0f803.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1071809988,https://api.github.com/repos/simonw/datasette/issues/1664,1071809988,IC_kwDOBm6k_c4_4oXE,9599,simonw,2022-03-17T23:24:57Z,2022-03-17T23:24:57Z,OWNER,"My hunch is that this is broken because of this: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/app.py#L1098-L1107 Note how the table uses `table_and_format` but the row uses just `table` - I think there's code that's getting confused by this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1071813296,https://api.github.com/repos/simonw/datasette/issues/1664,1071813296,IC_kwDOBm6k_c4_4pKw,9599,simonw,2022-03-17T23:26:22Z,2022-03-17T23:26:22Z,OWNER,Probably caused by the convoluted code is `get_format()`: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/base.py#L466-L481,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/issues/1605#issuecomment-1072907200,https://api.github.com/repos/simonw/datasette/issues/1605,1072907200,IC_kwDOBm6k_c4_80PA,9599,simonw,2022-03-19T00:52:54Z,2022-03-19T00:53:45Z,OWNER,"Had a thought about the implementation of this: it could make a really neat plugin. Something like `datasette-export` which adds a `export` command using https://docs.datasette.io/en/stable/plugin_hooks.html#register-commands-cli - then you could run: datasette export my-export-dir mydatabase.db -m metadata.json --template-dir templates/ And the command would then: - Create a `Datasette()` instance with those databases/metadata/etc - Execute`await datasette.client.get(""/"")` to get the homepage HTML - Parse the HTML using BeautifulSoup to find all `a[href]`, `link[href]`, `script[src]`, `img[src]` elements that reference a relative path as opposed to one that starts with `http://` - Write out the homepage to `my-export-dir/index.html` - Recursively fetch and dump all of the other pages and assets that it found too All of that HTML parsing may be over-complicating things. It could alternatively accept options for which pages you want to export: ``` datasette export my-export-dir \ mydatabase.db -m metadata.json --template-dir templates/ \ --path / \ --path /mydatabase ... ``` Or a really wild option: it could allow you to define the paths you want to export using a SQL query: ``` datasette export my-export-dir \ mydatabase.db -m metadata.json --template-dir templates/ \ --sql "" select '/' as path, 'index.html' as filename union all select '/mydatabase/articles/' || id as path, 'article-' || id || '.html' as filename from articles union all select '/mydatabase/tags/' || tag as path, 'tag-' || tag || '.html' as filename from tags "" ``` Which would save these files: - `index.html` as the content of `/` - `article-1.html` (and more) as the content of `/mydatabase/articles/1` - `tag-python.html` (and more) as the content of `/mydatabase/tags/python`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1228#issuecomment-1072907610,https://api.github.com/repos/simonw/datasette/issues/1228,1072907610,IC_kwDOBm6k_c4_80Va,9599,simonw,2022-03-19T00:55:29Z,2022-03-19T00:55:29Z,OWNER,"It looks to me like something is causing the faceting query here to return a string when it was expected to return a number: https://github.com/simonw/datasette/blob/32963018e7edfab1233de7c7076c428d0e5c7813/datasette/facets.py#L153-L170 I can't think of any way that a `count(*) as n` would turn into a string though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/issues/1228#issuecomment-1072907680,https://api.github.com/repos/simonw/datasette/issues/1228,1072907680,IC_kwDOBm6k_c4_80Wg,9599,simonw,2022-03-19T00:55:48Z,2022-03-19T00:55:48Z,OWNER,... unless your data had a column called `n`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/issues/1228#issuecomment-1072908029,https://api.github.com/repos/simonw/datasette/issues/1228,1072908029,IC_kwDOBm6k_c4_80b9,9599,simonw,2022-03-19T00:57:54Z,2022-03-19T00:57:54Z,OWNER,"Yes! That's the problem. I was able to replicate it like so: ``` echo '[{ ""n"": ""one"", ""abc"": 1 }, { ""n"": ""one"", ""abc"": 2 }, { ""n"": ""two"", ""abc"": 3 }]' | sqlite-utils insert column-called-n.db t - ``` <img width=""564"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/159100494-9a072e3c-7bad-4fc5-b90e-b5078c11fc44.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/issues/1228#issuecomment-1072915936,https://api.github.com/repos/simonw/datasette/issues/1228,1072915936,IC_kwDOBm6k_c4_82Xg,9599,simonw,2022-03-19T01:50:27Z,2022-03-19T01:50:27Z,OWNER,Demo: https://latest.datasette.io/fixtures/facetable - which now has a column called `n`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/issues/1666#issuecomment-1072933875,https://api.github.com/repos/simonw/datasette/issues/1666,1072933875,IC_kwDOBm6k_c4_86vz,9599,simonw,2022-03-19T04:03:42Z,2022-03-19T04:03:42Z,OWNER,Tests so far: https://github.com/simonw/datasette/blob/711767bcd3c1e76a0861fe7f24069ff1c8efc97a/tests/test_routes.py#L12-L34,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174162781,Refactor URL routing to enable testing, https://github.com/simonw/datasette/issues/1561#issuecomment-1072939780,https://api.github.com/repos/simonw/datasette/issues/1561,1072939780,IC_kwDOBm6k_c4_88ME,9599,simonw,2022-03-19T04:45:40Z,2022-03-19T04:45:40Z,OWNER,"I ended up moving hashed URL mode out to a plugin in: - #647 If you're still interested in using it with `_memory` please open an issue in that repo here: https://github.com/simonw/datasette-hashed-urls","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082765654,"add hash id to ""_memory"" url if hashed url mode is turned on and crossdb is also turned on", https://github.com/simonw/datasette/pull/1664#issuecomment-1072890205,https://api.github.com/repos/simonw/datasette/issues/1664,1072890205,IC_kwDOBm6k_c4_8wFd,9599,simonw,2022-03-18T23:43:15Z,2022-03-18T23:43:15Z,OWNER,"Now almost everything is working except for foreign key expansion: ![CleanShot 2022-03-18 at 16 41 39@2x](https://user-images.githubusercontent.com/9599/159097349-6f41dfdf-5bab-449b-a148-5cda3df6534c.png) Using the debugger I tracked it down to this code: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/table.py#L708-L715 Turns out `default_labels` there is `None` - and it's a parameter to that `data()` method: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/table.py#L325-L334 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1072890524,https://api.github.com/repos/simonw/datasette/issues/1664,1072890524,IC_kwDOBm6k_c4_8wKc,9599,simonw,2022-03-18T23:44:33Z,2022-03-19T00:06:51Z,OWNER,Looks like that was set here: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L490-L492,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1072898797,https://api.github.com/repos/simonw/datasette/issues/1664,1072898797,IC_kwDOBm6k_c4_8yLt,9599,simonw,2022-03-19T00:11:09Z,2022-03-19T00:11:09Z,OWNER,Still need to remove it from the documentation and do something about that `hash_urls` setting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1072898923,https://api.github.com/repos/simonw/datasette/issues/1664,1072898923,IC_kwDOBm6k_c4_8yNr,9599,simonw,2022-03-19T00:11:33Z,2022-03-19T00:11:33Z,OWNER,I'm going to land this and handle those in separate commits.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1072901159,https://api.github.com/repos/simonw/datasette/issues/1661,1072901159,IC_kwDOBm6k_c4_8ywn,9599,simonw,2022-03-19T00:20:27Z,2022-03-19T00:20:27Z,OWNER,I can remove the `default_cache_ttl_hashed` setting too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1072904703,https://api.github.com/repos/simonw/datasette/issues/1661,1072904703,IC_kwDOBm6k_c4_8zn_,9599,simonw,2022-03-19T00:37:36Z,2022-03-19T00:37:36Z,OWNER,Updated docs: https://docs.datasette.io/en/latest/performance.html#datasette-hashed-urls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1662#issuecomment-1072905467,https://api.github.com/repos/simonw/datasette/issues/1662,1072905467,IC_kwDOBm6k_c4_8zz7,9599,simonw,2022-03-19T00:42:23Z,2022-03-19T00:42:23Z,OWNER,"Those client-side SQLite tricks are _really_ neat. `datasette publish` defaults to configuring it so the raw SQLite database can be downloaded from `/fixtures.db` - and this issue updated it to be served with a CORS header that would allow client-side scripts to load the file: - #1057 If you're not going to run any server-side code at all you don't need Datasette for this - you can upload the SQLite database file to any static hosting with CORS headers and load it into the client that way. In terms of static publishing, I do think there's something interesting about using Datasette to generate static sites. There's an issue discussing options for that over here: - #1605","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170497629,[feature request] Publish to fully static website, https://github.com/simonw/datasette/issues/878#issuecomment-1073037939,https://api.github.com/repos/simonw/datasette/issues/878,1073037939,IC_kwDOBm6k_c4_9UJz,9599,simonw,2022-03-19T16:19:30Z,2022-03-19T16:19:30Z,OWNER,"On revisiting https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 a few months later I'm having second thoughts about using `@inject` on the `main()` method. But I still like the pattern as a way to resolve more complex cases like ""to generate GeoJSON of the expanded view with labels, the label expansion code needs to run once at some before the GeoJSON formatting code does"". So I'm going to stick with it a tiny bit longer, but maybe try to make it a lot more explicit when it's going to happen rather than having the main view methods themselves also use async DI.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1666#issuecomment-1073039241,https://api.github.com/repos/simonw/datasette/issues/1666,1073039241,IC_kwDOBm6k_c4_9UeJ,9599,simonw,2022-03-19T16:28:15Z,2022-03-19T16:28:15Z,OWNER,This is more interesting if it also asserts against the captured matches from the pattern.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174162781,Refactor URL routing to enable testing, https://github.com/simonw/datasette/issues/1666#issuecomment-1073039670,https://api.github.com/repos/simonw/datasette/issues/1666,1073039670,IC_kwDOBm6k_c4_9Uk2,9599,simonw,2022-03-19T16:31:08Z,2022-03-19T16:31:57Z,OWNER,"This does make it more interesting - it also highlights how inconsistent the way the capturing works is. Especially `as_format` which can be `None` or `""""` or `.json` or `json` or not used at all in the case of `TableView`. https://github.com/simonw/datasette/blob/764738dfcb16cd98b0987d443f59d5baa9d3c332/tests/test_routes.py#L12-L36","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174162781,Refactor URL routing to enable testing, https://github.com/simonw/datasette/issues/1667#issuecomment-1073040072,https://api.github.com/repos/simonw/datasette/issues/1667,1073040072,IC_kwDOBm6k_c4_9UrI,9599,simonw,2022-03-19T16:34:02Z,2022-03-19T16:34:02Z,OWNER,"I called it `as_format` to avoid clashing with the Python built-in `format()` function when these things were turned into keyword arguments, but now that they're not I can use `format` instead. I think I'm going to go with `database`, `table`, `format` and `pks`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174302994,Make route matched pattern groups more consistent, https://github.com/simonw/datasette/issues/1667#issuecomment-1073042554,https://api.github.com/repos/simonw/datasette/issues/1667,1073042554,IC_kwDOBm6k_c4_9VR6,9599,simonw,2022-03-19T16:50:01Z,2022-03-19T16:52:35Z,OWNER,"OK, I've made this more consistent - I still need to address the fact that `format` can be `.json` or `json` or not used at all before I close this issue. https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/tests/test_routes.py#L15-L35","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174302994,Make route matched pattern groups more consistent, https://github.com/simonw/datasette/issues/1668#issuecomment-1073043350,https://api.github.com/repos/simonw/datasette/issues/1668,1073043350,IC_kwDOBm6k_c4_9VeW,9599,simonw,2022-03-19T16:54:26Z,2022-03-19T16:54:26Z,OWNER,"The `Database` class already has a `path` property but it means something else - it's the path to the `.db` file on disk: https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/database.py#L29-L50 So need a different name for the path-that-is-used-in-the-URL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073043433,https://api.github.com/repos/simonw/datasette/issues/1668,1073043433,IC_kwDOBm6k_c4_9Vfp,9599,simonw,2022-03-19T16:54:55Z,2022-03-19T20:01:19Z,OWNER,"Options: - `route_path` - `url_path` - `route` I like `route_path`, or maybe `route`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073043713,https://api.github.com/repos/simonw/datasette/issues/1668,1073043713,IC_kwDOBm6k_c4_9VkB,9599,simonw,2022-03-19T16:56:19Z,2022-03-19T16:56:19Z,OWNER,"Worth noting that the `name` right now is picked automatically to avoid conflicts: https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/app.py#L397-L413","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073073547,https://api.github.com/repos/simonw/datasette/issues/1668,1073073547,IC_kwDOBm6k_c4_9c2L,9599,simonw,2022-03-19T20:06:07Z,2022-03-19T20:06:07Z,OWNER,"Implementing this is a little tricky because there's a whole lot of code that expects the `database` captured by the URL routing to be the name used to look up the database in `datasette.databases` - or via `.get_database()`. The `DataView.get()` method is a good example of the trickyness here. It even has code that dispatches out to plugin hooks that take `database` as a parameter. https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/views/base.py#L383-L555 All the more reason to get rid of that `BaseView -> DataView -> TableView` hierarchy entirely: - #1660","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1660#issuecomment-1073073599,https://api.github.com/repos/simonw/datasette/issues/1660,1073073599,IC_kwDOBm6k_c4_9c2_,9599,simonw,2022-03-19T20:06:40Z,2022-03-19T20:06:40Z,OWNER,"This blocks: - #1668","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1668#issuecomment-1073073579,https://api.github.com/repos/simonw/datasette/issues/1668,1073073579,IC_kwDOBm6k_c4_9c2r,9599,simonw,2022-03-19T20:06:27Z,2022-03-19T20:06:27Z,OWNER,Marking this as blocked until #1660 is done.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073075697,https://api.github.com/repos/simonw/datasette/issues/1668,1073075697,IC_kwDOBm6k_c4_9dXx,9599,simonw,2022-03-19T20:24:06Z,2022-03-19T20:24:06Z,OWNER,"Right now if a database has a `.` in its name e.g. `fixtures.dot` the URL to that database is: /fixtures~2Edot But the output on `/-/databases` doesn't reflect that, it still shows the name with the dot.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073075913,https://api.github.com/repos/simonw/datasette/issues/1668,1073075913,IC_kwDOBm6k_c4_9dbJ,9599,simonw,2022-03-19T20:25:46Z,2022-03-19T20:26:08Z,OWNER,"The output of `/.json` DOES use `path` to mean the URL path, not the path to the file on disk: ``` { ""fixtures.dot"": { ""name"": ""fixtures.dot"", ""hash"": null, ""color"": ""631f11"", ""path"": ""/fixtures~2Edot"", ``` So that's a problem already: having `db.path` refer to something different from that JSON is inconsistent.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073076015,https://api.github.com/repos/simonw/datasette/issues/1668,1073076015,IC_kwDOBm6k_c4_9dcv,9599,simonw,2022-03-19T20:26:32Z,2022-03-19T20:26:32Z,OWNER,I'm inclined to redefine `ds.path` to `ds.file_path` to fix this. Or `ds.filepath`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073076110,https://api.github.com/repos/simonw/datasette/issues/1668,1073076110,IC_kwDOBm6k_c4_9deO,9599,simonw,2022-03-19T20:27:22Z,2022-03-19T20:27:22Z,OWNER,"The docs do currently describe `path` as the filesystem path here: https://docs.datasette.io/en/stable/internals.html#database-class <img width=""720"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/159137373-5bbc7fea-7544-42b3-8423-be4e11eb4d52.png""> Good thing I'm not at 1.0 yet so I can change that!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073076136,https://api.github.com/repos/simonw/datasette/issues/1668,1073076136,IC_kwDOBm6k_c4_9deo,9599,simonw,2022-03-19T20:27:44Z,2022-03-19T20:27:44Z,OWNER,"Pretty sure changing it will break some existing plugins though, including likely Datasette Desktop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073076187,https://api.github.com/repos/simonw/datasette/issues/1668,1073076187,IC_kwDOBm6k_c4_9dfb,9599,simonw,2022-03-19T20:28:20Z,2022-03-19T20:28:20Z,OWNER,I'm going to keep `path` as the path to the file on disk. I'll pick a new name for what is currently `path` in that undocumented JSON API.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1667#issuecomment-1073076624,https://api.github.com/repos/simonw/datasette/issues/1667,1073076624,IC_kwDOBm6k_c4_9dmQ,9599,simonw,2022-03-19T20:31:44Z,2022-03-19T20:31:44Z,OWNER,I can now read `format` from `request.url_vars` and delete this code entirely: https://github.com/simonw/datasette/blob/b9c2b1cfc8692b9700416db98721fa3ec982f6be/datasette/views/base.py#L375-L381,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174302994,Make route matched pattern groups more consistent, https://github.com/simonw/datasette/issues/1668#issuecomment-1073097394,https://api.github.com/repos/simonw/datasette/issues/1668,1073097394,IC_kwDOBm6k_c4_9iqy,9599,simonw,2022-03-19T20:56:35Z,2022-03-19T20:56:35Z,OWNER,"I'm trying to think if there's any reason not to use `route` for this. Would I possibly want to use that noun for something else in the future? I like it more than `route_path` because it has no underscore. Decision made: I'm going with `route`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073112104,https://api.github.com/repos/simonw/datasette/issues/1668,1073112104,IC_kwDOBm6k_c4_9mQo,9599,simonw,2022-03-19T21:08:21Z,2022-03-19T21:08:21Z,OWNER,"I think I've got this working but I need to write a test for it that covers the rare case when the route is not the same thing as the database name. I'll do that with a new test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073125334,https://api.github.com/repos/simonw/datasette/issues/1668,1073125334,IC_kwDOBm6k_c4_9pfW,9599,simonw,2022-03-19T22:53:55Z,2022-03-19T22:53:55Z,OWNER,"Need to update documentation in a few places - e.g. https://docs.datasette.io/en/stable/internals.html#remove-database-name > This removes a database that has been previously added. `name=` is the unique name of that database, used in its URL path.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073126264,https://api.github.com/repos/simonw/datasette/issues/1668,1073126264,IC_kwDOBm6k_c4_9pt4,9599,simonw,2022-03-19T22:59:30Z,2022-03-19T22:59:30Z,OWNER,"Also need to update the `datasette.urls` methods that construct the URL to a database/table/row - they take the database name but they need to know to look for the route. Need to add tests that check the links in the HTML and can confirm this is working correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073135433,https://api.github.com/repos/simonw/datasette/issues/1668,1073135433,IC_kwDOBm6k_c4_9r9J,9599,simonw,2022-03-20T00:20:36Z,2022-03-20T00:20:36Z,OWNER,"Building this plugin instantly revealed that all of the links - on the homepage and the database page and so on - are incorrect: ```python from datasette import hookimpl @hookimpl def startup(datasette): db = datasette.get_database(""fixtures2"") db.route = ""alternative-route"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073134206,https://api.github.com/repos/simonw/datasette/issues/1668,1073134206,IC_kwDOBm6k_c4_9rp-,9599,simonw,2022-03-20T00:12:03Z,2022-03-20T00:12:03Z,OWNER,I'd like to have a live demo of this up on `latest.datasette.io` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073134816,https://api.github.com/repos/simonw/datasette/issues/1668,1073134816,IC_kwDOBm6k_c4_9rzg,9599,simonw,2022-03-20T00:16:22Z,2022-03-20T00:16:22Z,OWNER,I'm going to add a `fixtures2.db` database which has that as the name but `alternative-route` as the route. I'll set that up using a custom plugin in the `plugins/` folder that gets deployed by https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073136686,https://api.github.com/repos/simonw/datasette/issues/1668,1073136686,IC_kwDOBm6k_c4_9sQu,9599,simonw,2022-03-20T00:31:13Z,2022-03-20T00:31:13Z,OWNER,"That demo is now live: - https://latest.datasette.io/alternative-route - https://latest.datasette.io/alternative-route/attraction_characteristic","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073136896,https://api.github.com/repos/simonw/datasette/issues/1668,1073136896,IC_kwDOBm6k_c4_9sUA,9599,simonw,2022-03-20T00:33:23Z,2022-03-20T00:33:23Z,OWNER,I'm going to release this as a 0.61 alpha so I can more easily depend on it from `datasette-hashed-urls`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1669#issuecomment-1073137170,https://api.github.com/repos/simonw/datasette/issues/1669,1073137170,IC_kwDOBm6k_c4_9sYS,9599,simonw,2022-03-20T00:35:52Z,2022-03-20T00:35:52Z,OWNER,https://github.com/simonw/datasette/compare/0.60.2...main,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174404647,Release 0.61 alpha, https://github.com/simonw/datasette/issues/1669#issuecomment-1073143413,https://api.github.com/repos/simonw/datasette/issues/1669,1073143413,IC_kwDOBm6k_c4_9t51,9599,simonw,2022-03-20T01:24:36Z,2022-03-20T01:24:36Z,OWNER,https://github.com/simonw/datasette/releases/tag/0.61a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174404647,Release 0.61 alpha, https://github.com/simonw/datasette/issues/1439#issuecomment-1059802318,https://api.github.com/repos/simonw/datasette/issues/1439,1059802318,IC_kwDOBm6k_c4_K0zO,9599,simonw,2022-03-05T17:34:33Z,2022-03-05T17:34:33Z,OWNER,"Wrote documentation: <img width=""741"" alt=""Dash encoding. Datasette uses a custom encoding scheme in some places, called dash encoding. This is primarily used for table names and row primary keys, to avoid any confusion between / characters in those values and the Datasette URL that references them. Dash encoding applies the following rules, in order: 1. All single - characters are replaced by -- 2. . characters are replaced by -. 3. / characters are replaced by ./ These rules are applied in reverse order to decode a dash encoded string."" src=""https://user-images.githubusercontent.com/9599/156893903-5723f60e-e054-4365-84bc-f3084d11183d.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1647#issuecomment-1059804577,https://api.github.com/repos/simonw/datasette/issues/1647,1059804577,IC_kwDOBm6k_c4_K1Wh,9599,simonw,2022-03-05T17:49:46Z,2022-03-05T17:49:46Z,OWNER,My best guess is that this is an undocumented change in SQLite 3.38 - I get that test failure with that SQLite version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1647#issuecomment-1059807598,https://api.github.com/repos/simonw/datasette/issues/1647,1059807598,IC_kwDOBm6k_c4_K2Fu,9599,simonw,2022-03-05T18:06:56Z,2022-03-05T18:08:00Z,OWNER,"Had a look through the commits in https://github.com/sqlite/sqlite/compare/version-3.37.2...version-3.38.0 but couldn't see anything obvious that might have caused this. Really wish I had a good mechanism for running the test suite against different SQLite versions! May have to revisit this old trick: https://til.simonwillison.net/sqlite/ld-preload","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1647#issuecomment-1059823119,https://api.github.com/repos/simonw/datasette/issues/1647,1059823119,IC_kwDOBm6k_c4_K54P,9599,simonw,2022-03-05T19:56:27Z,2022-03-05T19:56:27Z,OWNER,Updated this TIL with extra patterns I figured out: https://til.simonwillison.net/sqlite/ld-preload,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1647#issuecomment-1059819628,https://api.github.com/repos/simonw/datasette/issues/1647,1059819628,IC_kwDOBm6k_c4_K5Bs,9599,simonw,2022-03-05T19:28:54Z,2022-03-05T19:28:54Z,OWNER,"OK, using that trick worked for testing this: docker run -it -p 8001:8001 ubuntu Then inside that container: apt-get install -y python3 build-essential tcl wget python3-pip git python3.8-venv For each version of SQLite I wanted to test I needed to figure out the tarball URL - for example, for `3.38.0` I navigated to https://www.sqlite.org/src/timeline?t=version-3.38.0 and clicked the ""checkin"" link and copied the tarball link: https://www.sqlite.org/src/tarball/40fa792d/SQLite-40fa792d.tar.gz Then to build it (the `CPPFLAGS` took some trial and error): ``` cd /tmp wget https://www.sqlite.org/src/tarball/40fa792d/SQLite-40fa792d.tar.gz tar -xzvf SQLite-40fa792d.tar.gz cd SQLite-40fa792d CPPFLAGS=""-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1"" ./configure make ``` Then to test with Datasette: ``` cd /tmp git clone https://github.com/simonw/datasette cd datasette python3 -m venv venv source venv/bin/activate pip install wheel # So bdist_wheel works in next step pip install -e '.[test]' LD_PRELOAD=/tmp/SQLite-40fa792d/.libs/libsqlite3.so pytest ``` After some trial and error I proved that those tests passed with 3.36.0: ``` cd /tmp wget https://www.sqlite.org/src/tarball/5c9a6c06/SQLite-5c9a6c06.tar.gz tar -xzvf SQLite-5c9a6c06.tar.gz cd SQLite-5c9a6c06 CPPFLAGS=""-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1"" ./configure make cd /tmp/datasette LD_PRELOAD=/tmp/SQLite-5c9a6c06/.libs/libsqlite3.so pytest tests/test_internals_database.py ``` BUT failed with 3.37.0: ``` # 3.37.0 cd /tmp wget https://www.sqlite.org/src/tarball/bd41822c/SQLite-bd41822c.tar.gz tar -xzvf SQLite-bd41822c.tar.gz cd SQLite-bd41822c CPPFLAGS=""-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1"" ./configure make cd /tmp/datasette LD_PRELOAD=/tmp/SQLite-bd41822c/.libs/libsqlite3.so pytest tests/test_internals_database.py ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1647#issuecomment-1059821674,https://api.github.com/repos/simonw/datasette/issues/1647,1059821674,IC_kwDOBm6k_c4_K5hq,9599,simonw,2022-03-05T19:44:32Z,2022-03-05T19:44:32Z,OWNER,"I thought I'd need to introduce https://dirty-equals.helpmanual.io/types/string/ to help write tests for this, but I think I've found a good alternative that doesn't need a new dependency.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1439#issuecomment-1059822151,https://api.github.com/repos/simonw/datasette/issues/1439,1059822151,IC_kwDOBm6k_c4_K5pH,9599,simonw,2022-03-05T19:48:35Z,2022-03-05T19:48:35Z,OWNER,Those new docs: https://github.com/simonw/datasette/blob/d1cb73180b4b5a07538380db76298618a5fc46b6/docs/internals.rst#dash-encoding,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059822391,https://api.github.com/repos/simonw/datasette/issues/1439,1059822391,IC_kwDOBm6k_c4_K5s3,9599,simonw,2022-03-05T19:50:12Z,2022-03-05T19:50:12Z,OWNER,I'm going to move this work to a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059836599,https://api.github.com/repos/simonw/datasette/issues/1439,1059836599,IC_kwDOBm6k_c4_K9K3,9599,simonw,2022-03-05T21:52:10Z,2022-03-05T21:52:10Z,OWNER,Blogged about this here: https://simonwillison.net/2022/Mar/5/dash-encoding/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1645#issuecomment-1059634688,https://api.github.com/repos/simonw/datasette/issues/1645,1059634688,IC_kwDOBm6k_c4_KL4A,9599,simonw,2022-03-05T01:06:08Z,2022-03-05T01:06:08Z,OWNER,"It sounds like you can workaround this with Varnish configuration for the moment, but I'm going to bump this up the list of things to fix - it's particularly relevant now as I'd like to get a solution in place before Datasette 1.0, since it's likely to be beneficial to plugins and hence should be part of the stable, documented plugin interface.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1154399841,"Sensible `cache-control` headers for static assets, including those served by plugins", https://github.com/simonw/datasette/issues/1645#issuecomment-1059633902,https://api.github.com/repos/simonw/datasette/issues/1645,1059633902,IC_kwDOBm6k_c4_KLru,9599,simonw,2022-03-05T01:03:06Z,2022-03-05T01:03:06Z,OWNER,"I agree: this is bad. Ideally, content served from `/static/` would apply best practices for static content serving - which to my mind means the following: - Where possible, serve with a far-future cache expiry header and use an asset URL that changes when the file itself changes - For assets without that, support conditional GET to avoid transferring the whole asset if it hasn't changed - Some kind of sensible mechanism for setting cache TTLs on assets that don't have a unique-file-per-version - in particular assets that might be served from plugins. Datasette half-implemented the first of these: if you view source on https://latest.datasette.io/ you'll see it links to `/-/static/app.css?cead5a` - which in the template looks like this: https://github.com/simonw/datasette/blob/dd94157f8958bdfe9f45575add934ccf1aba6d63/datasette/templates/base.html#L5 I had forgotten I had implemented this! Here is how it is calculated: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/app.py#L510-L516 So `app.css` right now could be safely served with a far-future cache header... only it isn't: ``` ~ % curl -i 'https://latest.datasette.io/-/static/app.css?cead5a' HTTP/2 200 content-type: text/css x-databases: _memory, _internal, fixtures, extra_database x-cloud-trace-context: 9ddc825620eb53d30fc127d1c750f342 date: Sat, 05 Mar 2022 01:01:53 GMT server: Google Frontend content-length: 16178 ``` The larger question though is what to do about other assets. I'm particularly interested in plugin assets, since visualization plugins like `datasette-vega` and `datasette-cluster-map` ship with large amounts of JavaScript and I'd really like that to be sensibly cached by default.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1154399841,"Sensible `cache-control` headers for static assets, including those served by plugins", https://github.com/simonw/datasette/issues/1645#issuecomment-1059634412,https://api.github.com/repos/simonw/datasette/issues/1645,1059634412,IC_kwDOBm6k_c4_KLzs,9599,simonw,2022-03-05T01:04:53Z,2022-03-05T01:04:53Z,OWNER,"The existing `app_css_hash` already isn't good enough, because I built that before `table.js` existed, and that file should obviously be smartly cached too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1154399841,"Sensible `cache-control` headers for static assets, including those served by plugins", https://github.com/simonw/datasette/issues/1640#issuecomment-1059638778,https://api.github.com/repos/simonw/datasette/issues/1640,1059638778,IC_kwDOBm6k_c4_KM36,9599,simonw,2022-03-05T01:19:00Z,2022-03-05T01:19:00Z,OWNER,"The reason I implemented it like this was to support things like the `curl` progress bar if users decide to serve up large files using the `--static` mechanism. Here's the code that hooks it up to the URL resolver: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/app.py#L1001-L1005 Which uses this function: https://github.com/simonw/datasette/blob/a6ff123de5464806441f6a6f95145c9a83b7f20b/datasette/utils/asgi.py#L285-L310 One option here would be to support a workaround that looks something like this: http://localhost:8001/my-static/log.txt?_unknown_size=1` The URL routing code could then look out for that `?_unknown_size=1` option and, if it's present, omit the `content-length` header entirely. It's a bit of a cludge, but it would be pretty straight-forward to implement. Would that work for you @broccolihighkicks?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1148725876,"Support static assets where file length may change, e.g. logs", https://github.com/simonw/datasette/issues/1642#issuecomment-1059635969,https://api.github.com/repos/simonw/datasette/issues/1642,1059635969,IC_kwDOBm6k_c4_KMMB,9599,simonw,2022-03-05T01:11:17Z,2022-03-05T01:11:17Z,OWNER,"`pip install datasette` in a fresh virtual environment doesn't show any warnings. Neither does `pip install -e '.'` in a fresh checkout. Or `pip install -e '.[test]'`. Closing this as can't reproduce.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1152072027,Dependency issue with asgiref and uvicorn, https://github.com/simonw/datasette/issues/1640#issuecomment-1059636420,https://api.github.com/repos/simonw/datasette/issues/1640,1059636420,IC_kwDOBm6k_c4_KMTE,9599,simonw,2022-03-05T01:13:26Z,2022-03-05T01:13:26Z,OWNER,"Hah, this is certainly unexpected. It looks like this is the code in question: https://github.com/simonw/datasette/blob/a6ff123de5464806441f6a6f95145c9a83b7f20b/datasette/utils/asgi.py#L259-L266 You're right: it assumes that the file it is serving won't change length while it is serving it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1148725876,"Support static assets where file length may change, e.g. logs", https://github.com/simonw/datasette/pull/1648#issuecomment-1060065736,https://api.github.com/repos/simonw/datasette/issues/1648,1060065736,IC_kwDOBm6k_c4_L1HI,9599,simonw,2022-03-06T23:43:00Z,2022-03-06T23:43:11Z,OWNER,"> * Maybe use dash encoding for database name too? Yes, I'm going to do this. At the moment if a DB file is called `fixx%tures.db` when you run it in Datasette the path is `/fix%2525tures` - which is liable to break.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/pull/1648#issuecomment-1060067031,https://api.github.com/repos/simonw/datasette/issues/1648,1060067031,IC_kwDOBm6k_c4_L1bX,9599,simonw,2022-03-06T23:50:40Z,2022-03-06T23:58:31Z,OWNER,"I may have to do extra work here ```python def database(self, database, format=None): db = self.ds.databases[database] if self.ds.setting(""hash_urls"") and db.hash: path = self.path( f""{dash_encode(database)}-{db.hash[:HASH_LENGTH]}"", format=format ) else: path = self.path(dash_encode(database), format=format) return path ``` The URLs that incorporate a hash have a `dbname-hash` format - will that `-` in the middle there mess up the dash decoding mechanism? I think it will. Might be able to solve that like so: ```python async def resolve_db_name(self, request, db_name, **kwargs): hash = None name = None decoded_name = dash_decode(db_name) if decoded_name not in self.ds.databases and ""-"" in db_name: # No matching DB found, maybe it's a name-hash? name_bit, hash_bit = db_name.rsplit(""-"", 1) if dash_decode(name_bit) not in self.ds.databases: raise NotFound(f""Database not found: {name}"") else: name = dash_decode(name_bit) hash = hash_bit else: name = decoded_name ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/issues/1439#issuecomment-1059850369,https://api.github.com/repos/simonw/datasette/issues/1439,1059850369,IC_kwDOBm6k_c4_LAiB,9599,simonw,2022-03-05T23:28:56Z,2022-03-05T23:28:56Z,OWNER,"Lots of great conversations about the dash encoding implementation on Twitter: https://twitter.com/simonw/status/1500228316309061633 @dracos helped me figure out a simpler regex: https://twitter.com/dracos/status/1500236433809973248 `^/(?P<database>[^/]+)/(?P<table>[^\/\-\.]*|\-/|\-\.|\-\-)*(?P<format>\.\w+)?$` ![image](https://user-images.githubusercontent.com/9599/156903088-c01933ae-4713-4e91-8d71-affebf70b945.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059851259,https://api.github.com/repos/simonw/datasette/issues/1439,1059851259,IC_kwDOBm6k_c4_LAv7,9599,simonw,2022-03-05T23:35:47Z,2022-03-05T23:35:59Z,OWNER,"This [comment from glyph](https://twitter.com/glyph/status/1500244937312329730) got me thinking: > Have you considered replacing % with some other character and then using percent-encoding? What happens if a table name includes a `%` character and that ends up getting mangled by a misbehaving proxy? I should consider `%` in the escaping system too. And maybe go with that suggestion of using percent-encoding directly but with a different character.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059853526,https://api.github.com/repos/simonw/datasette/issues/1439,1059853526,IC_kwDOBm6k_c4_LBTW,9599,simonw,2022-03-05T23:49:59Z,2022-03-05T23:49:59Z,OWNER,"I want to try regular percentage encoding, except that it also encodes both the `-` and the `.` characters, AND it uses `-` instead of `%` as the encoding character. Should check what it does with emoji too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059854864,https://api.github.com/repos/simonw/datasette/issues/1439,1059854864,IC_kwDOBm6k_c4_LBoQ,9599,simonw,2022-03-05T23:59:05Z,2022-03-05T23:59:05Z,OWNER,"OK, for that percentage thing: the Python core implementation of URL percentage escaping deliberately ignores two of the characters we want to escape: `.` and `-`: https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Lib/urllib/parse.py#L780-L783 ```python _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-~') ``` It also defaults to skipping `/` (passed as a `safe=` parameter to various things). I'm going to try borrowing and modifying the core of the Python implementation: https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Lib/urllib/parse.py#L795-L814 ```python class _Quoter(dict): """"""A mapping from bytes numbers (in range(0,256)) to strings. String values are percent-encoded byte values, unless the key < 128, and in either of the specified safe set, or the always safe set. """""" # Keeps a cache internally, via __missing__, for efficiency (lookups # of cached keys don't call Python code at all). def __init__(self, safe): """"""safe: bytes object."""""" self.safe = _ALWAYS_SAFE.union(safe) def __repr__(self): return f""<Quoter {dict(self)!r}>"" def __missing__(self, b): # Handle a cache miss. Store quoted string in cache and return. res = chr(b) if b in self.safe else '%{:02X}'.format(b) self[b] = res return res ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059855418,https://api.github.com/repos/simonw/datasette/issues/1439,1059855418,IC_kwDOBm6k_c4_LBw6,9599,simonw,2022-03-06T00:00:53Z,2022-03-06T00:04:18Z,OWNER,"```python _ESCAPE_SAFE = frozenset( b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789_' ) # I removed b'.-~') class Quoter(dict): # Keeps a cache internally, via __missing__ def __missing__(self, b): # Handle a cache miss. Store quoted string in cache and return. res = chr(b) if b in _ESCAPE_SAFE else '-{:02X}'.format(b) self[b] = res return res quoter = Quoter().__getitem__ ''.join([quoter(char) for char in b'foo/bar.csv']) # 'foo-2Fbar-2Ecsv' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059864154,https://api.github.com/repos/simonw/datasette/issues/1439,1059864154,IC_kwDOBm6k_c4_LD5a,9599,simonw,2022-03-06T00:59:04Z,2022-03-06T00:59:04Z,OWNER,"Needs more testing, but this seems to work for decoding the percent-escaped-with-dashes format: `urllib.parse.unquote(s.replace('-', '%'))`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/pull/1589#issuecomment-1059875687,https://api.github.com/repos/simonw/datasette/issues/1589,1059875687,IC_kwDOBm6k_c4_LGtn,9599,simonw,2022-03-06T01:58:25Z,2022-03-06T01:58:25Z,OWNER,Thanks for catching this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1098275181,Typo in docs about default redirect status code, https://github.com/simonw/datasette/issues/1439#issuecomment-1059903309,https://api.github.com/repos/simonw/datasette/issues/1439,1059903309,IC_kwDOBm6k_c4_LNdN,9599,simonw,2022-03-06T06:17:51Z,2022-03-06T06:17:51Z,OWNER,"Suggestion from a conversation with Seth Michael Larson: it would be neat if plugins could easily integrate with whatever scheme this ends up using, maybe with the `/db/table/-/plugin-name` standardized pattern or similar. Making it easy for plugins to do the right, consistent thing is a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/pull/1648#issuecomment-1060016221,https://api.github.com/repos/simonw/datasette/issues/1648,1060016221,IC_kwDOBm6k_c4_LpBd,9599,simonw,2022-03-06T18:37:59Z,2022-03-06T18:37:59Z,OWNER,"Change of plan: based on extensive conversations on Twitter - see https://github.com/simonw/datasette/issues/1439#issuecomment-1059851259 - I'm going to try a variant of this which is basically percent-encoding but with a hyphen instead of a percent symbol. Reason being that the current scheme doesn't handle the case of `%` being part of the table name, which could cause weird breakage due to some proxies decoding percent encoding before it gets to Datasette.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/pull/1648#issuecomment-1060034562,https://api.github.com/repos/simonw/datasette/issues/1648,1060034562,IC_kwDOBm6k_c4_LtgC,9599,simonw,2022-03-06T20:36:12Z,2022-03-06T20:36:12Z,OWNER,"Updated documentation: ![image](https://user-images.githubusercontent.com/9599/156941171-89778c12-41bc-4951-97f2-ecc805025a53.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/pull/1648#issuecomment-1060044592,https://api.github.com/repos/simonw/datasette/issues/1648,1060044592,IC_kwDOBm6k_c4_Lv8w,9599,simonw,2022-03-06T21:42:35Z,2022-03-06T21:42:35Z,OWNER,"For consistency, I'm going to change how `?_next=` tokens work too. Right now they work like this: https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/views/table.py#L501-L507 https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/utils/__init__.py#L114-L116 I'm going to change those to use dash-encoding instead. I considered looking for `%` in those values and replacing that as `-` too, but since Datasette isn't 1.0 yet I'm going to risk breaking any pagination tokens that people might have saved away somewhere!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/issues/1439#issuecomment-1060044007,https://api.github.com/repos/simonw/datasette/issues/1439,1060044007,IC_kwDOBm6k_c4_Lvzn,9599,simonw,2022-03-06T21:38:15Z,2022-03-06T21:38:15Z,OWNER,"Test: https://github.com/simonw/datasette/blob/d2e3fe3facf0ed0abf8b00cd54463af90dd6904d/tests/test_utils.py#L651-L666 One big advantage to this scheme is that redirecting old links to `%2F` pages (e.g. https://fivethirtyeight.datasettes.com/fivethirtyeight/twitter-ratio%2Fsenators) is easy - if you see a `%` in the `raw_path`, redirect to that page with the `%` replaced by `-`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/pull/1648#issuecomment-1060056510,https://api.github.com/repos/simonw/datasette/issues/1648,1060056510,IC_kwDOBm6k_c4_Ly2-,9599,simonw,2022-03-06T23:02:05Z,2022-03-06T23:04:24Z,OWNER,"Just spotted this: https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/views/base.py#L203-L216 Maybe the db name should use dash encoding too? If so, relevant code includes this bit: https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/url_builder.py#L30-L38","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/issues/1651#issuecomment-1060853226,https://api.github.com/repos/simonw/datasette/issues/1651,1060853226,IC_kwDOBm6k_c4_O1Xq,9599,simonw,2022-03-07T16:04:26Z,2022-03-07T16:04:26Z,OWNER,"Here's the relevant code: https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/utils/__init__.py#L753-L772 https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L451-L479","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1650#issuecomment-1060863311,https://api.github.com/repos/simonw/datasette/issues/1650,1060863311,IC_kwDOBm6k_c4_O31P,9599,simonw,2022-03-07T16:13:17Z,2022-03-07T16:13:17Z,OWNER,"This doesn't seem to work. https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv should be redirecting now that this is deployed - which it is, because https://latest.datasette.io/-/versions shows 644d25d1de78a36b105cca479e7b3e4375a6eadc - but I'm not getting that redirect.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1650#issuecomment-1060864823,https://api.github.com/repos/simonw/datasette/issues/1650,1060864823,IC_kwDOBm6k_c4_O4M3,9599,simonw,2022-03-07T16:14:33Z,2022-03-07T16:14:33Z,OWNER,Same problem here: https://fivethirtyeight.datasettes.com/fivethirtyeight/ahca-2Dpolls%2Fahca_polls should redirect but doesn't.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1439#issuecomment-1060870237,https://api.github.com/repos/simonw/datasette/issues/1439,1060870237,IC_kwDOBm6k_c4_O5hd,9599,simonw,2022-03-07T16:19:22Z,2022-03-07T16:19:22Z,OWNER,"I didn't need to do any of the fancy regular expression routing stuff after all, since the new dash encoding format avoids using `/` so a simple `[^/]+` can capture the correct segments from the URL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1650#issuecomment-1060836262,https://api.github.com/repos/simonw/datasette/issues/1650,1060836262,IC_kwDOBm6k_c4_OxOm,9599,simonw,2022-03-07T15:52:09Z,2022-03-07T15:52:09Z,OWNER,"This is a bit tricky. I tried this, sending a redirect only if a 404 happens: ```diff diff --git a/datasette/app.py b/datasette/app.py index 8c5480c..420664c 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1211,6 +1211,10 @@ class DatasetteRouter: return await self.handle_404(request, send) async def handle_404(self, request, send, exception=None): + # If path contains % encoding, redirect to dash encoding + if '%' in request.scope[""path""]: + await asgi_send_redirect(send, request.scope[""path""].replace(""%"", ""-"")) + return # If URL has a trailing slash, redirect to URL without it path = request.scope.get( ""raw_path"", request.scope[""path""].encode(""utf8"") ``` But this URL didn't work: - http://127.0.0.1:8001/fivethirtyeight/twitter-ratio%2Fsenators I was expecting that to redirect to this page: - http://127.0.0.1:8001/fivethirtyeight/twitter-2Dratio-2Fsenators But instead it took me to another 404: - http://127.0.0.1:8001/fivethirtyeight/twitter-ratio%2Fsenators This is because that URL contains both a %-escaped `/` AND a plain `-` - which was not escaped in the old system but is escaped in the new system.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1653#issuecomment-1061150672,https://api.github.com/repos/simonw/datasette/issues/1653,1061150672,IC_kwDOBm6k_c4_P9_Q,9599,simonw,2022-03-07T21:23:39Z,2022-03-07T21:23:39Z,OWNER,"There may be a short-term fix for this: table view could start accepting a `?_sort_sql=SQLfragment` parameter, similar to the `?_where=` parameter described here: https://docs.datasette.io/en/stable/json_api.html#special-table-arguments That fragment could then be pre-populated in `metadata`. Makes me think maybe that `?_where=` should be optionally settable in metadata too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161937073,Mechanism to default a table to sorting by multiple columns, https://github.com/simonw/datasette/issues/1653#issuecomment-1061148807,https://api.github.com/repos/simonw/datasette/issues/1653,1061148807,IC_kwDOBm6k_c4_P9iH,9599,simonw,2022-03-07T21:21:23Z,2022-03-07T21:21:23Z,OWNER,"This is currently blocked on the fact that Datasette doesn't have a mechanism for sorting by more than one column: - #197","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161937073,Mechanism to default a table to sorting by multiple columns, https://github.com/simonw/datasette/issues/1650#issuecomment-1061038414,https://api.github.com/repos/simonw/datasette/issues/1650,1061038414,IC_kwDOBm6k_c4_PilO,9599,simonw,2022-03-07T19:14:04Z,2022-03-07T19:14:04Z,OWNER,"The problem seems to be that `http://127.0.0.1:8002/fixtures/table%2Fwith%2Fslashes.csv` doesn't result in a 404 at all. If it did, it would be redirected.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1650#issuecomment-1061041034,https://api.github.com/repos/simonw/datasette/issues/1650,1061041034,IC_kwDOBm6k_c4_PjOK,9599,simonw,2022-03-07T19:16:51Z,2022-03-07T19:16:51Z,OWNER,"Here's the problem: https://github.com/simonw/datasette/blob/020effe47bf89f35182960a9645f2383a42ebd54/datasette/utils/__init__.py#L1173-L1175 Which is called here: https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L469-L473 So `table%2Fwith%2Fslashes` ends up decoded as if it was using dash encoding.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1651#issuecomment-1061053094,https://api.github.com/repos/simonw/datasette/issues/1651,1061053094,IC_kwDOBm6k_c4_PmKm,9599,simonw,2022-03-07T19:29:01Z,2022-03-07T19:29:01Z,OWNER,"I found an obscure bug in #1650 which I can fix with this too. The following test should pass: ```python @pytest.mark.parametrize( ""path,expected"", ( ( ""/fivethirtyeight/twitter-ratio%2Fsenators"", ""/fivethirtyeight/twitter-2Dratio-2Fsenators"", ), ( ""/fixtures/table%2Fwith%2Fslashes.csv"", ""/fixtures/table-2Fwith-2Fslashes-2Ecsv"", ), # query string should be preserved (""/foo/bar%2Fbaz?id=5"", ""/foo/bar-2Fbaz?id=5""), ), ) def test_redirect_percent_encoding_to_dash_encoding(app_client, path, expected): response = app_client.get(path) assert response.status == 302 assert response.headers[""location""] == expected ``` It currently fails like this: ``` > assert response.headers[""location""] == expected E AssertionError: assert '/fixtures/table-2Fwith-2Fslashes.csv?_nofacet=1&_nocount=1' == '/fixtures/table-2Fwith-2Fslashes-2Ecsv' E - /fixtures/table-2Fwith-2Fslashes-2Ecsv E + /fixtures/table-2Fwith-2Fslashes.csv?_nofacet=1&_nocount=1 ``` Because the logic in that `get_format()` function notices that the table exists, and then weird things happen here: https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L288-L303 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1651#issuecomment-1061170897,https://api.github.com/repos/simonw/datasette/issues/1651,1061170897,IC_kwDOBm6k_c4_QC7R,9599,simonw,2022-03-07T21:48:35Z,2022-03-07T21:48:35Z,OWNER,"My attempts to simplify `get_format()` keep resulting in errors like this one: ``` File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 474, in view_get response_or_template_contexts = await self.data( TypeError: TableView.data() missing 1 required positional argument: 'table' ``` I really need to clean this up.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1651#issuecomment-1061169528,https://api.github.com/repos/simonw/datasette/issues/1651,1061169528,IC_kwDOBm6k_c4_QCl4,9599,simonw,2022-03-07T21:47:01Z,2022-03-07T21:47:01Z,OWNER,"Wow, this code is difficult to follow! Look at this bit inside the `get_format()` method: https://github.com/simonw/datasette/blob/bb499942c15c4e2cfa4b6afab8f8debe5948c009/datasette/views/base.py#L469-L478 That's modifying the arguments that were extracted from the path by the routing regular expressions to have `table` as ` dash-decoded value! So calling `.get_format()` has the side effect of decoding the table names for you. Nasty.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1654#issuecomment-1061181089,https://api.github.com/repos/simonw/datasette/issues/1654,1061181089,IC_kwDOBm6k_c4_QFah,9599,simonw,2022-03-07T22:01:38Z,2022-03-07T22:01:38Z,OWNER,"I'm going to use the [widely adopted](https://www.contributor-covenant.org/adopters/) Contributor Covenant: https://www.contributor-covenant.org/version/1/4/code-of-conduct/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1654#issuecomment-1061181530,https://api.github.com/repos/simonw/datasette/issues/1654,1061181530,IC_kwDOBm6k_c4_QFha,9599,simonw,2022-03-07T22:02:06Z,2022-03-07T22:02:06Z,OWNER,https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/adding-a-code-of-conduct-to-your-project says this should be called `CODE_OF_CONDUCT.md` in order for GitHub to pick it up.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1654#issuecomment-1061182132,https://api.github.com/repos/simonw/datasette/issues/1654,1061182132,IC_kwDOBm6k_c4_QFq0,9599,simonw,2022-03-07T22:02:43Z,2022-03-07T22:02:43Z,OWNER,"Neat, GitHub have a template for this https://github.com/simonw/datasette/community/code-of-conduct/new?template=contributor-covenant","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1654#issuecomment-1061184206,https://api.github.com/repos/simonw/datasette/issues/1654,1061184206,IC_kwDOBm6k_c4_QGLO,9599,simonw,2022-03-07T22:04:51Z,2022-03-07T22:04:51Z,OWNER,I'm going to add this to the main Datasette repo (done) and the `datasette.io` website too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1654#issuecomment-1061197133,https://api.github.com/repos/simonw/datasette/issues/1654,1061197133,IC_kwDOBm6k_c4_QJVN,9599,simonw,2022-03-07T22:19:35Z,2022-03-07T22:19:35Z,OWNER,"Also now live on https://datasette.io ![CleanShot 2022-03-07 at 14 18 30@2x](https://user-images.githubusercontent.com/9599/157127424-805b3166-f0a8-4fac-be87-c055740af580.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1651#issuecomment-1061223822,https://api.github.com/repos/simonw/datasette/issues/1651,1061223822,IC_kwDOBm6k_c4_QP2O,9599,simonw,2022-03-07T22:54:54Z,2022-03-07T22:54:54Z,OWNER,"I'm going to do a review of how URL routing works at the moment for the various views. I edited down [the full list](https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/app.py#L997-L1107) a bit - these are the most relevant: ```python add_route(IndexView.as_view(self), r""/(?P<as_format>(\.jsono?)?$)"") add_route( DatabaseView.as_view(self), r""/(?P<db_name>[^/]+?)(?P<as_format>"" + renderer_regex + r""|.jsono|\.csv)?$"", ) add_route( TableView.as_view(self), r""/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)"", ) add_route( RowView.as_view(self), r""/(?P<db_name>[^/]+)/(?P<table>[^/]+?)/(?P<pk_path>[^/]+?)(?P<as_format>"" + renderer_regex + r"")?$"", ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/647#issuecomment-1061226942,https://api.github.com/repos/simonw/datasette/issues/647,1061226942,IC_kwDOBm6k_c4_QQm-,9599,simonw,2022-03-07T23:00:06Z,2022-03-07T23:00:06Z,OWNER,"This needs to take into account the changes made here: - #1439 In the new encoding scheme, `-` has a special meaning in a table name: https://docs.datasette.io/en/latest/internals.html#dash-encoding I think `~` is the right character to use to separate a database name from its hash. `~` should be a URL safe character according to Python's implementation of percent-encoding, see comment here: https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/utils/__init__.py#L1146-L1152 So the plugin could check for `dbname~hash` and react based on that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061267615,https://api.github.com/repos/simonw/datasette/issues/647,1061267615,IC_kwDOBm6k_c4_Qaif,9599,simonw,2022-03-08T00:05:43Z,2022-03-08T00:05:43Z,OWNER,"Built a prototype of that plugin: ```python from datasette import hookimpl from functools import wraps @hookimpl def asgi_wrapper(datasette): def wrap_with_hashed_urls(app): @wraps(app) async def hashed_urls(scope, receive, send): # Only triggers on pages with a path not starting in /-/ # and where the first page component matches a database name if scope.get(""type"") != ""http"": await app(scope, receive, send) return path = scope[""path""].lstrip(""/"") if not path or path.startswith(""-/""): await app(scope, receive, send) return potential_database = path.split(""/"")[0] # It may or may not be already dbname~hash if ""~"" in potential_database: db_name, hash = potential_database.split(""~"", 1) else: db_name = potential_database hash = """" # Is db_name a database we have a hash for? try: db = datasette.get_database(db_name) except KeyError: await app(scope, receive, send) return if db.hash is not None: # TODO: make sure db.hash is documented if db.hash[:7] != hash: # Send a redirect path_bits = path.split(""/"") new_path = ""/"" + ""/"".join([""{}-{}"".format(db_name, db.hash[:7])] + path_bits[1:]) if scope.get(""query_string""): new_path += ""?"" + scope[""query_string""].decode(""latin-1"") await send({ ""type"": ""http.response.start"", ""status"": 302, ""headers"": [ [b""location"", new_path.encode(""latin1"")] ], }) await send({""type"": ""http.response.body"", ""body"": b""""}) return else: # Add a far-future cache header async def wrapped_send(event): if event[""type""] == ""http.response.start"": original_headers = event.get(""headers"") or [] event = { ""type"": event[""type""], ""status"": event[""status""], ""headers"": original_headers + [ [b""Cache-Control"", b""max-age=31536000""] ], } await send(event) await app(scope, receive, wrapped_send) return await app(scope, receive, send) return hashed_urls return wrap_with_hashed_urls ``` One catch: it doesn't affect the way URLs are generated - so every internal link within Datasette links to the non-hash version and then triggers a 302 redirect to the hashed version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061272544,https://api.github.com/repos/simonw/datasette/issues/647,1061272544,IC_kwDOBm6k_c4_Qbvg,9599,simonw,2022-03-08T00:14:42Z,2022-03-08T00:14:42Z,OWNER,Maybe the plugin should interfere with `datasette.databases` on startup and change the registered name for each one?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061276399,https://api.github.com/repos/simonw/datasette/issues/647,1061276399,IC_kwDOBm6k_c4_Qcrv,9599,simonw,2022-03-08T00:21:47Z,2022-03-08T00:21:47Z,OWNER,"This seems to do the job: ```python @hookimpl def startup(datasette): for name, database in datasette.databases.items(): if database.hash: new_name = ""{}_{}"".format(name, database.hash[:7]) del datasette.databases[name] datasette.databases[new_name] = database ``` Would have to teach the rest of the plugin to split on `_` and to only redirect if the user seems to be hitting the URL for an old hash after which Datasette has been restarted with an updated database.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061276646,https://api.github.com/repos/simonw/datasette/issues/647,1061276646,IC_kwDOBm6k_c4_Qcvm,9599,simonw,2022-03-08T00:22:11Z,2022-03-08T00:22:11Z,OWNER,I'm now convinced this is feasible enough that it's worth doing in time for Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061282743,https://api.github.com/repos/simonw/datasette/issues/647,1061282743,IC_kwDOBm6k_c4_QeO3,9599,simonw,2022-03-08T00:32:34Z,2022-03-08T00:32:47Z,OWNER,It would be neat if the plugin could spot old-style hyphen hash URLs (maybe on 404) and redirect those too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/1645#issuecomment-1061355871,https://api.github.com/repos/simonw/datasette/issues/1645,1061355871,IC_kwDOBm6k_c4_QwFf,9599,simonw,2022-03-08T02:59:28Z,2022-03-08T02:59:28Z,OWNER,"Hah, found a TODO about this: https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/app.py#L997-L999","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1154399841,"Sensible `cache-control` headers for static assets, including those served by plugins", https://github.com/simonw/datasette/issues/1651#issuecomment-1061359915,https://api.github.com/repos/simonw/datasette/issues/1651,1061359915,IC_kwDOBm6k_c4_QxEr,9599,simonw,2022-03-08T03:08:14Z,2022-03-08T03:09:24Z,OWNER,"A lot of the code complexity here is caused by `DataView` ([here](https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/views/base.py#L182-L669)), which has the logic for CSV streaming and plugin formats such that it can be shared between tables and custom queries. It would be good to get rid of that subclassed shared code, figure out how to do it via a utility function instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/932#issuecomment-1061891851,https://api.github.com/repos/simonw/datasette/issues/932,1061891851,IC_kwDOBm6k_c4_Sy8L,9599,simonw,2022-03-08T15:20:48Z,2022-03-08T15:20:48Z,OWNER,Made a start on this here: https://datasette.io/tutorials ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation, https://github.com/simonw/datasette/issues/1655#issuecomment-1062445113,https://api.github.com/repos/simonw/datasette/issues/1655,1062445113,IC_kwDOBm6k_c4_U6A5,9599,simonw,2022-03-09T01:01:24Z,2022-03-09T01:01:24Z,OWNER,"https://labordata.bunkum.us/-/settings shows `max_returned_rows` had been increased to 5,000 for that instance - the default of 1,000 would help a bit here. Any thoughts on how Datasette could handle this kind of thing better?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data, https://github.com/simonw/datasette/issues/1439#issuecomment-1065987808,https://api.github.com/repos/simonw/datasette/issues/1439,1065987808,IC_kwDOBm6k_c4_ia7g,9599,simonw,2022-03-13T00:02:32Z,2022-03-13T00:02:32Z,OWNER,"OK, this has broken a lot more than I expected it would. Turns out `-` is a very common character in existing Datasette database names! https://datasette.io/-/databases for example has two: ```json [ { ""name"": ""docs-index"", ""path"": ""docs-index.db"", ""size"": 1007616, ""is_mutable"": false, ""is_memory"": false, ""hash"": ""0ac6c3de2762fcd174fd249fed8a8fa6046ea345173d22c2766186bf336462b2"" }, { ""name"": ""dogsheep-index"", ""path"": ""dogsheep-index.db"", ""size"": 5496832, ""is_mutable"": false, ""is_memory"": false, ""hash"": ""d1ea238d204e5b9ae783c86e4af5bcdf21267c1f391de3e468d9665494ee012a"" } ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1065988403,https://api.github.com/repos/simonw/datasette/issues/1439,1065988403,IC_kwDOBm6k_c4_ibEz,9599,simonw,2022-03-13T00:06:38Z,2022-03-13T00:07:19Z,OWNER,"If I want to reserve `-` as a character that CAN be used in URLs, the only remaining character that might make sense for escape sequences is `~` - based on this last line of characters that are escape from percentage encoding: ```python _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-~') ``` So I'd add both `-` and `_` back to the safe list, but use `~` to escape `.` and `/` and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1657#issuecomment-1067413691,https://api.github.com/repos/simonw/datasette/issues/1657,1067413691,IC_kwDOBm6k_c4_n3C7,9599,simonw,2022-03-14T23:37:42Z,2022-03-14T23:37:42Z,OWNER,"Relevant: https://datatracker.ietf.org/doc/html/rfc3986#section-2.1 ``` reserved = gen-delims / sub-delims gen-delims = "":"" / ""/"" / ""?"" / ""#"" / ""["" / ""]"" / ""@"" sub-delims = ""!"" / ""$"" / ""&"" / ""'"" / ""("" / "")"" / ""*"" / ""+"" / "","" / "";"" / ""="" ``` Notably `~` is not in either of those lists.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1067414156,https://api.github.com/repos/simonw/datasette/issues/1657,1067414156,IC_kwDOBm6k_c4_n3KM,9599,simonw,2022-03-14T23:38:41Z,2022-03-14T23:38:41Z,OWNER,"And in https://datatracker.ietf.org/doc/html/rfc3986#section-2.3 ""Unreserved Characters"": unreserved = ALPHA / DIGIT / ""-"" / ""."" / ""_"" / ""~""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1067423720,https://api.github.com/repos/simonw/datasette/issues/1657,1067423720,IC_kwDOBm6k_c4_n5fo,9599,simonw,2022-03-14T23:59:56Z,2022-03-14T23:59:56Z,OWNER,"Updated test: ```python @pytest.mark.parametrize( ""original,expected"", ( (""abc"", ""abc""), (""/foo/bar"", ""~2Ffoo~2Fbar""), (""/-/bar"", ""~2F-~2Fbar""), (""-/db-/table.csv"", ""-~2Fdb-~2Ftable~2Ecsv""), (r""%~-/"", ""~25~7E-~2F""), (""~25~7E~2D~2F"", ""~7E25~7E7E~7E2D~7E2F""), ), ) def test_tilde_encoding(original, expected): actual = utils.tilde_encode(original) assert actual == expected # And test round-trip assert original == utils.tilde_decode(actual) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1067381556,https://api.github.com/repos/simonw/datasette/issues/1657,1067381556,IC_kwDOBm6k_c4_nvM0,9599,simonw,2022-03-14T22:57:27Z,2022-03-14T22:57:45Z,OWNER,"The problem with the [dash encoding mechanism](https://simonwillison.net/2022/Mar/5/dash-encoding/) is that it turns out dashes are used in a LOT of existing Datasette instances - much of https://fivethirtyeight.datasettes.com/fivethirtyeight for example, and even https://datasette.io/ itself: https://datasette.io/dogsheep-index It's pretty ugly to force all of those to change to their dash-encoded equivalent - and in fact it broke https://datasette.io/ in a subtle way: - https://github.com/simonw/datasette.io/issues/94 I'm going to try using `~` instead and see if that works as well and causes less breakage to existing sites.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1067382232,https://api.github.com/repos/simonw/datasette/issues/1657,1067382232,IC_kwDOBm6k_c4_nvXY,9599,simonw,2022-03-14T22:58:47Z,2022-03-14T22:58:47Z,OWNER,"Asked about this [on Twitter](https://twitter.com/simonw/status/1503499169775849473): > Anyone ever seen a proxy or other URL handling system do anything surprising with the tilde ""~"" character? > > I'm considering it as an escaping character, in place of ""-"" as described in Replies so far seem like it should be OK - Apache has supported this for home directories for a couple of decades now without any problems.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1651#issuecomment-1067382442,https://api.github.com/repos/simonw/datasette/issues/1651,1067382442,IC_kwDOBm6k_c4_nvaq,9599,simonw,2022-03-14T22:59:10Z,2022-03-14T22:59:10Z,OWNER,"This work is now blocked on: - https://github.com/simonw/datasette/issues/1657","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1657#issuecomment-1068125636,https://api.github.com/repos/simonw/datasette/issues/1657,1068125636,IC_kwDOBm6k_c4_qk3E,9599,simonw,2022-03-15T15:30:54Z,2022-03-15T15:30:54Z,OWNER,I've made a real mess of this. I'm going to revert Datasette`main` back to the last commit that passed the tests and try this again in a branch.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068126821,https://api.github.com/repos/simonw/datasette/issues/1657,1068126821,IC_kwDOBm6k_c4_qlJl,9599,simonw,2022-03-15T15:31:54Z,2022-03-15T15:31:54Z,OWNER,The state I had got to prior to that revert is in https://github.com/simonw/datasette/tree/issue-1657-wip,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1658#issuecomment-1068138578,https://api.github.com/repos/simonw/datasette/issues/1658,1068138578,IC_kwDOBm6k_c4_qoBS,9599,simonw,2022-03-15T15:42:49Z,2022-03-15T15:42:49Z,OWNER,"Easiest way to do this was with three reverts, then cherry-pick back the code of conduct.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1169840669,Revert main to version that passes tests, https://github.com/simonw/datasette/issues/1657#issuecomment-1068148013,https://api.github.com/repos/simonw/datasette/issues/1657,1068148013,IC_kwDOBm6k_c4_qqUt,9599,simonw,2022-03-15T15:50:15Z,2022-03-15T15:50:15Z,OWNER,"The thing that broke everything was this change: <img width=""569"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/158417416-285018c1-af45-49bc-9687-829c0c17ff05.png""> I'm going to bring back the horrible `get_format()` method for the moment, with its weird mutations of the `args` object, then try and get rid of it again later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068181623,https://api.github.com/repos/simonw/datasette/issues/1657,1068181623,IC_kwDOBm6k_c4_qyh3,9599,simonw,2022-03-15T16:18:23Z,2022-03-15T16:18:23Z,OWNER,Moving this to a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/857#issuecomment-1068450483,https://api.github.com/repos/simonw/datasette/issues/857,1068450483,IC_kwDOBm6k_c4_r0Kz,9599,simonw,2022-03-15T20:43:55Z,2022-03-15T20:43:55Z,OWNER,Dupe of #1510.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642297505,Comprehensive documentation for variables made available to templates, https://github.com/simonw/datasette/issues/1439#issuecomment-1068461449,https://api.github.com/repos/simonw/datasette/issues/1439,1068461449,IC_kwDOBm6k_c4_r22J,9599,simonw,2022-03-15T20:51:26Z,2022-03-15T20:51:26Z,OWNER,I'm happy with this now that I've landed Tilde encoding in #1657.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1657#issuecomment-1068296042,https://api.github.com/repos/simonw/datasette/issues/1657,1068296042,IC_kwDOBm6k_c4_rOdq,9599,simonw,2022-03-15T18:05:54Z,2022-03-15T18:05:54Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#tilde-encoding,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068306916,https://api.github.com/repos/simonw/datasette/issues/1657,1068306916,IC_kwDOBm6k_c4_rRHk,9599,simonw,2022-03-15T18:15:11Z,2022-03-15T18:15:11Z,OWNER,Now live here: https://fivethirtyeight.datasettes.com/fivethirtyeight/august-senate-polls~2Faugust_senate_polls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068318454,https://api.github.com/repos/simonw/datasette/issues/1657,1068318454,IC_kwDOBm6k_c4_rT72,9599,simonw,2022-03-15T18:25:11Z,2022-03-15T18:25:11Z,OWNER,"Demo: - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.csv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1651#issuecomment-1068319530,https://api.github.com/repos/simonw/datasette/issues/1651,1068319530,IC_kwDOBm6k_c4_rUMq,9599,simonw,2022-03-15T18:25:42Z,2022-03-15T18:25:42Z,OWNER,"Done: - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.csv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1062#issuecomment-1068327874,https://api.github.com/repos/simonw/datasette/issues/1062,1068327874,IC_kwDOBm6k_c4_rWPC,9599,simonw,2022-03-15T18:33:49Z,2022-03-15T18:33:49Z,OWNER,"I can get regular `.json` to stream too, using the pattern described in this TIL: https://til.simonwillison.net/python/output-json-array-streaming","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows, https://github.com/simonw/datasette/issues/1660#issuecomment-1068415072,https://api.github.com/repos/simonw/datasette/issues/1660,1068415072,IC_kwDOBm6k_c4_rrhg,9599,simonw,2022-03-15T20:02:36Z,2022-03-15T20:02:36Z,OWNER,"This is one of the worst bits - the `get_format()` method on the `DataView` base class actually modifies `args`, including removing keys! Really confusing: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L454-L482 Then `BaseView` has some surprising responsibilities. It has a utility helper for checking multiple permissions at once: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L81-L105 And its own render method that adds extra stuff to the template context and handles the rel: alternate header: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L131-L157 Then `DataView` does all sorts of weird stuff - from handling database hashes (which I want to remove, see #647): https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L206-L219 To streaming CSV responses: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L286-L308 To handling SQLite exceptions: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L514-L526 And a ton more. It' s a big mess.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1660#issuecomment-1068417357,https://api.github.com/repos/simonw/datasette/issues/1660,1068417357,IC_kwDOBm6k_c4_rsFN,9599,simonw,2022-03-15T20:05:08Z,2022-03-15T20:05:08Z,OWNER,"`DataView` is used as the base class for: - `DatabaseView` - `DatabaseDownload` (just so the permissions checks can be called) - `QueryView` - which isn't routed to directly, it's called from `DatabaseView` if `?sql=` is available and `TableView` for canned queries - `RowTableShared` which is the base class for `TableView` and `RowView`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1660#issuecomment-1068418619,https://api.github.com/repos/simonw/datasette/issues/1660,1068418619,IC_kwDOBm6k_c4_rsY7,9599,simonw,2022-03-15T20:06:19Z,2022-03-15T20:06:19Z,OWNER,"Also related: - #878 - #1512 - #1518 - #870 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1509#issuecomment-1068445412,https://api.github.com/repos/simonw/datasette/issues/1509,1068445412,IC_kwDOBm6k_c4_ry7k,9599,simonw,2022-03-15T20:37:50Z,2022-03-15T20:38:56Z,OWNER,"... maybe Datasette itself should include interactive API documentation, in addition to documenting it in the manual? `/dbname/table/-/apidocs` could return documentation about the specific table, taking into account columns and types.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054243511,Datasette 1.0 JSON API (and documentation), https://github.com/simonw/datasette/issues/1510#issuecomment-1068443509,https://api.github.com/repos/simonw/datasette/issues/1510,1068443509,IC_kwDOBm6k_c4_ryd1,9599,simonw,2022-03-15T20:35:29Z,2022-03-15T20:35:29Z,OWNER,If I set a rule that everything available in the template context MUST also be available via the JSON API (maybe through an extras mechanism) I can combine this with API documentation and solve both at once.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054244712,Datasette 1.0 documented template context (maybe via API docs), https://github.com/simonw/datasette/issues/1509#issuecomment-1068444767,https://api.github.com/repos/simonw/datasette/issues/1509,1068444767,IC_kwDOBm6k_c4_ryxf,9599,simonw,2022-03-15T20:37:03Z,2022-03-15T20:37:03Z,OWNER,"Idea: I could add Pydantic https://pydantic-docs.helpmanual.io/usage/schema/ as an optional test dependency and use it to generate JSON schemas and run validation against examples in the API documentation. Maybe generate API documentation from it too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054243511,Datasette 1.0 JSON API (and documentation), https://github.com/simonw/datasette/issues/1661#issuecomment-1068728484,https://api.github.com/repos/simonw/datasette/issues/1661,1068728484,IC_kwDOBm6k_c4_s4Ck,9599,simonw,2022-03-16T04:47:39Z,2022-03-16T04:47:39Z,OWNER,https://datasette.io/plugins/datasette-hashed-urls is released now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1663#issuecomment-1068742624,https://api.github.com/repos/simonw/datasette/issues/1663,1068742624,IC_kwDOBm6k_c4_s7fg,9599,simonw,2022-03-16T05:17:45Z,2022-03-16T05:17:45Z,OWNER,Should be documented here: https://docs.datasette.io/en/stable/internals.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170554975,Document the internals that were used in datasette-hashed-urls, https://github.com/simonw/datasette/issues/647#issuecomment-1068539404,https://api.github.com/repos/simonw/datasette/issues/647,1068539404,IC_kwDOBm6k_c4_sJ4M,9599,simonw,2022-03-15T22:49:01Z,2022-03-15T22:49:01Z,OWNER,"I shipped the first version of this: https://github.com/simonw/datasette-hashed-urls Next step: test it with a live demo: - https://github.com/simonw/datasette-hashed-urls/issues/2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1068552696,https://api.github.com/repos/simonw/datasette/issues/647,1068552696,IC_kwDOBm6k_c4_sNH4,9599,simonw,2022-03-15T23:13:06Z,2022-03-15T23:13:06Z,OWNER,"The plugin works. I'm going to implement one last feature for it: - https://github.com/simonw/datasette-hashed-urls/issues/3 Then I can remove hashed URL mode in a separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/1661#issuecomment-1068553454,https://api.github.com/repos/simonw/datasette/issues/1661,1068553454,IC_kwDOBm6k_c4_sNTu,9599,simonw,2022-03-15T23:14:37Z,2022-03-15T23:14:37Z,OWNER,"This is going to simplify the code in the various view classes substantially: - #1660","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1068554827,https://api.github.com/repos/simonw/datasette/issues/1661,1068554827,IC_kwDOBm6k_c4_sNpL,9599,simonw,2022-03-15T23:16:58Z,2022-03-15T23:18:58Z,OWNER,"If you attempt to use the [old setting](https://docs.datasette.io/en/stable/settings.html#hash-urls): datasette mydatabase.db --setting hash_urls 1 It should error with a message saying that the feature has been moved to a plugin. I'll do this with a `deprecated_settings` mechanism so the error can be detected even though `datasette --help-settings` will no longer return the setting. https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/cli.py#L479-L489","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1068628839,https://api.github.com/repos/simonw/datasette/issues/1661,1068628839,IC_kwDOBm6k_c4_sftn,9599,simonw,2022-03-16T01:21:36Z,2022-03-16T01:21:48Z,OWNER,"From https://docs.datasette.io/en/0.60.2/performance.html#hashed-url-mode > You can enable these hashed URLs in two ways: using the [hash_urls](https://docs.datasette.io/en/0.60.2/settings.html#setting-hash-urls) configuration setting (which affects all requests to Datasette) or via the `?_hash=1` query string parameter (which only applies to the current request). I'm going to drop` ?_hash=1` entirely. I'd actually forgotten that feature existed!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1068630353,https://api.github.com/repos/simonw/datasette/issues/1661,1068630353,IC_kwDOBm6k_c4_sgFR,9599,simonw,2022-03-16T01:24:56Z,2022-03-16T01:25:49Z,OWNER,"Here's the only bit of code that references that `_hash` mechanism: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L259-L265 And here's the test: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/tests/test_api.py#L828-L854 Related issue: - #471","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1532#issuecomment-1069570893,https://api.github.com/repos/simonw/datasette/issues/1532,1069570893,IC_kwDOBm6k_c4_wFtN,9599,simonw,2022-03-16T20:11:41Z,2022-03-16T20:13:34Z,OWNER,"Could also build a CLI Rich/Textual app to exercise the API - which could embed Datasette as a dependency and work using `datasette.client.get(...)` calls. Could be a plugin that adds a `datasette tui` command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936,Use datasette-table Web Component to guide the design of the JSON API for 1.0, https://github.com/simonw/datasette/issues/1177#issuecomment-1074017633,https://api.github.com/repos/simonw/datasette/issues/1177,1074017633,IC_kwDOBm6k_c5ABDVh,9599,simonw,2022-03-21T15:08:51Z,2022-03-21T15:08:51Z,OWNER,"Related: - #1062 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780153562,Ability to stream all rows as newline-delimited JSON, https://github.com/simonw/datasette/issues/526#issuecomment-1074019047,https://api.github.com/repos/simonw/datasette/issues/526,1074019047,IC_kwDOBm6k_c5ABDrn,9599,simonw,2022-03-21T15:09:56Z,2022-03-21T15:09:56Z,OWNER,I should research how much overhead creating a new connection costs - it may be that an easy way to solve this is to create A dedicated connection for the query and then close that connection at the end.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/1660#issuecomment-1074136176,https://api.github.com/repos/simonw/datasette/issues/1660,1074136176,IC_kwDOBm6k_c5ABgRw,9599,simonw,2022-03-21T16:38:46Z,2022-03-21T16:38:46Z,OWNER,"I'm going to refactor this stuff out and document it so it can be easily used by plugins: https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L69-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1675#issuecomment-1074142617,https://api.github.com/repos/simonw/datasette/issues/1675,1074142617,IC_kwDOBm6k_c5ABh2Z,9599,simonw,2022-03-21T16:45:27Z,2022-03-21T16:45:27Z,OWNER,"Though at that point `check_permission` is such a light wrapper around `self.ds.permission_allowed()` that there's little point in it existing at all. So maybe `check_permisions()` becomes `ds.permissions_allowed()`. `permission_allowed()` v.s. `permissions_allowed()` is a bit of a subtle naming difference, but I think it works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074143209,https://api.github.com/repos/simonw/datasette/issues/1675,1074143209,IC_kwDOBm6k_c5ABh_p,9599,simonw,2022-03-21T16:46:05Z,2022-03-21T16:46:05Z,OWNER,"The other difference though is that `ds.permission_allowed(...)` works against an actor, while `check_permission()` works against a request (though just to access `request.actor`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074141457,https://api.github.com/repos/simonw/datasette/issues/1675,1074141457,IC_kwDOBm6k_c5ABhkR,9599,simonw,2022-03-21T16:44:09Z,2022-03-21T16:44:09Z,OWNER,"A slightly odd thing about these methods is that they either fail silently or they raise a `Forbidden` exception. Maybe they should instead return `True` or `False` and the calling code could decide if it wants to raise the exception? That would make them more usable and a little less surprising.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074158890,https://api.github.com/repos/simonw/datasette/issues/1675,1074158890,IC_kwDOBm6k_c5ABl0q,9599,simonw,2022-03-21T16:57:15Z,2022-03-21T16:57:15Z,OWNER,"Idea: `ds.permission_allowed()` continues to just return `True` or `False`. A new `ds.ensure_permissions(...)` method is added which raises a `Forbidden` exception if a check fails (hence the different name)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074156779,https://api.github.com/repos/simonw/datasette/issues/1675,1074156779,IC_kwDOBm6k_c5ABlTr,9599,simonw,2022-03-21T16:55:08Z,2022-03-21T16:56:02Z,OWNER,"One benefit of the current design of `check_permissions` that raises an exception is that the exception includes information on WHICH of the permission checks failed. Returning just `True` or `False` loses that information. I could return an object which evaluates to `False` but also carries extra information? Bit weird, I've never seen anything like that in other Python code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074161523,https://api.github.com/repos/simonw/datasette/issues/1675,1074161523,IC_kwDOBm6k_c5ABmdz,9599,simonw,2022-03-21T16:59:55Z,2022-03-21T17:00:03Z,OWNER,Also calling that function `permissions_allowed()` is confusing because there is a plugin hook with a similar name already: https://docs.datasette.io/en/stable/plugin_hooks.html#permission-allowed-datasette-actor-action-resource,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074177827,https://api.github.com/repos/simonw/datasette/issues/1675,1074177827,IC_kwDOBm6k_c5ABqcj,9599,simonw,2022-03-21T17:14:31Z,2022-03-21T17:14:31Z,OWNER,"Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions > This method allows multiple permissions to be checked at onced. It raises a `datasette.Forbidden` exception if any of the checks are denied before one of them is explicitly granted. > > This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns `True` or not a single one of them returns `False`: That's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1676#issuecomment-1074178865,https://api.github.com/repos/simonw/datasette/issues/1676,1074178865,IC_kwDOBm6k_c5ABqsx,9599,simonw,2022-03-21T17:15:27Z,2022-03-21T17:15:27Z,OWNER,This method here: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/app.py#L632-L664,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,"Reconsider ensure_permissions() logic, can it be less confusing?", https://github.com/simonw/datasette/issues/1676#issuecomment-1074180312,https://api.github.com/repos/simonw/datasette/issues/1676,1074180312,IC_kwDOBm6k_c5ABrDY,9599,simonw,2022-03-21T17:16:45Z,2022-03-21T17:16:45Z,OWNER,"When looking at this code earlier I assumed that the following would check each permission in turn and fail if any of them failed: ```python await self.ds.ensure_permissions( request.actor, [ (""view-table"", (database, table)), (""view-database"", database), ""view-instance"", ] ) ``` But it's not quite that simple: if any of them fail, it fails... but if an earlier one returns `True` the whole stack passes even if there would have been a failure later on! If that is indeed the right abstraction, I need to work to make the documentation as clear as possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,"Reconsider ensure_permissions() logic, can it be less confusing?", https://github.com/simonw/datasette/issues/1677#issuecomment-1074184240,https://api.github.com/repos/simonw/datasette/issues/1677,1074184240,IC_kwDOBm6k_c5ABsAw,9599,simonw,2022-03-21T17:20:17Z,2022-03-21T17:20:17Z,OWNER,"https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/views/base.py#L69-L77 This is weirdly different from how `check_permissions()` used to work, in that it doesn't differentiate between `None` and `False`. https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L79-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175694248,Remove `check_permission()` from `BaseView`, https://github.com/simonw/datasette/issues/276#issuecomment-1074479768,https://api.github.com/repos/simonw/datasette/issues/276,1074479768,IC_kwDOBm6k_c5AC0KY,9599,simonw,2022-03-21T22:22:20Z,2022-03-21T22:22:20Z,OWNER,"I'm closing this issue because this is now solved by a number of neat plugins: - https://datasette.io/plugins/datasette-geojson-map shows the geometry from SpatiaLite columns on a map - https://datasette.io/plugins/datasette-leaflet-geojson can be used to display inline maps next to each column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,Handle spatialite geometry columns better, https://github.com/simonw/datasette/issues/339#issuecomment-1074479932,https://api.github.com/repos/simonw/datasette/issues/339,1074479932,IC_kwDOBm6k_c5AC0M8,9599,simonw,2022-03-21T22:22:34Z,2022-03-21T22:22:34Z,OWNER,Closing this as obsolete since Datasette no longer uses Sanic.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",340396247,Expose SANIC_RESPONSE_TIMEOUT config option in a sensible way, https://github.com/simonw/datasette/issues/1660#issuecomment-1074287177,https://api.github.com/repos/simonw/datasette/issues/1660,1074287177,IC_kwDOBm6k_c5ACFJJ,9599,simonw,2022-03-21T18:51:42Z,2022-03-21T18:51:42Z,OWNER,`BaseView` is looking a LOT slimmer now that I've moved all of the permissions stuff out of it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1678#issuecomment-1074302559,https://api.github.com/repos/simonw/datasette/issues/1678,1074302559,IC_kwDOBm6k_c5ACI5f,9599,simonw,2022-03-21T19:04:03Z,2022-03-21T19:04:03Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#await-check-visibility-actor-action-resource-none,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175715988,Make `check_visibility()` a documented API, https://github.com/simonw/datasette/issues/1660#issuecomment-1074321862,https://api.github.com/repos/simonw/datasette/issues/1660,1074321862,IC_kwDOBm6k_c5ACNnG,9599,simonw,2022-03-21T19:19:01Z,2022-03-21T19:19:01Z,OWNER,I've simplified this a ton now. I'm going to keep working on this in the long-term but I think this issue can be closed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1679#issuecomment-1074331743,https://api.github.com/repos/simonw/datasette/issues/1679,1074331743,IC_kwDOBm6k_c5ACQBf,9599,simonw,2022-03-21T19:30:05Z,2022-03-21T19:30:05Z,OWNER,"https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/app.py#L118-L122 sets it to 50ms for facet suggestion but that's not going to pass `ms < 50`: ```python Setting( ""facet_suggest_time_limit_ms"", 50, ""Time limit for calculating a suggested facet"", ), ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074332325,https://api.github.com/repos/simonw/datasette/issues/1679,1074332325,IC_kwDOBm6k_c5ACQKl,9599,simonw,2022-03-21T19:30:44Z,2022-03-21T19:30:44Z,OWNER,So it looks like even for facet suggestion `n=1000` always - it's never reduced to `n=1`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074332718,https://api.github.com/repos/simonw/datasette/issues/1679,1074332718,IC_kwDOBm6k_c5ACQQu,9599,simonw,2022-03-21T19:31:10Z,2022-03-21T19:31:10Z,OWNER,How long does it take for SQLite to execute 1000 opcodes anyway?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074337997,https://api.github.com/repos/simonw/datasette/issues/1679,1074337997,IC_kwDOBm6k_c5ACRjN,9599,simonw,2022-03-21T19:37:08Z,2022-03-21T19:37:08Z,OWNER,"This is weird: ```python import sqlite3 db = sqlite3.connect("":memory:"") i = 0 def count(): global i i += 1 db.set_progress_handler(count, 1) db.execute("""""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """""") print(i) ``` Outputs `24`. But if you try the same thing in the SQLite console: ``` sqlite> .stats vmstep sqlite> with recursive counter(x) as ( ...> select 0 ...> union ...> select x + 1 from counter ...> ) ...> select * from counter limit 10000; ... VM-steps: 200007 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074341924,https://api.github.com/repos/simonw/datasette/issues/1679,1074341924,IC_kwDOBm6k_c5ACSgk,9599,simonw,2022-03-21T19:42:08Z,2022-03-21T19:42:08Z,OWNER,"Here's the Python-C implementation of `set_progress_handler`: https://github.com/python/cpython/blob/4674fd4e938eb4a29ccd5b12c15455bd2a41c335/Modules/_sqlite/connection.c#L1177-L1201 It calls `sqlite3_progress_handler(self->db, n, progress_callback, ctx);` https://www.sqlite.org/c3ref/progress_handler.html says: > The parameter N is the approximate number of [virtual machine instructions](https://www.sqlite.org/opcode.html) that are evaluated between successive invocations of the callback X So maybe VM-steps and virtual machine instructions are different things?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074347023,https://api.github.com/repos/simonw/datasette/issues/1679,1074347023,IC_kwDOBm6k_c5ACTwP,9599,simonw,2022-03-21T19:48:59Z,2022-03-21T19:48:59Z,OWNER,Posed a question about that here: https://sqlite.org/forum/forumpost/de9ff10fa7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1676#issuecomment-1074378472,https://api.github.com/repos/simonw/datasette/issues/1676,1074378472,IC_kwDOBm6k_c5ACbbo,9599,simonw,2022-03-21T20:18:10Z,2022-03-21T20:18:10Z,OWNER,Maybe there is a better name for this method that helps emphasize its cascading nature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,"Reconsider ensure_permissions() logic, can it be less confusing?", https://github.com/simonw/datasette/issues/1679#issuecomment-1074439309,https://api.github.com/repos/simonw/datasette/issues/1679,1074439309,IC_kwDOBm6k_c5ACqSN,9599,simonw,2022-03-21T21:28:58Z,2022-03-21T21:28:58Z,OWNER,"David Raymond solved it there: https://sqlite.org/forum/forumpost/330c8532d8a88bcd > Don't forget to step through the results. All .execute() has done is prepared it. > > db.execute(query).fetchall() Sure enough, adding that gets the VM steps number up to 190,007 which is close enough that I'm happy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074446576,https://api.github.com/repos/simonw/datasette/issues/1679,1074446576,IC_kwDOBm6k_c5ACsDw,9599,simonw,2022-03-21T21:38:27Z,2022-03-21T21:38:27Z,OWNER,"OK here's a microbenchmark script: ```python import sqlite3 import timeit db = sqlite3.connect("":memory:"") db_with_progress_handler_1 = sqlite3.connect("":memory:"") db_with_progress_handler_1000 = sqlite3.connect("":memory:"") db_with_progress_handler_1.set_progress_handler(lambda: None, 1) db_with_progress_handler_1000.set_progress_handler(lambda: None, 1000) def execute_query(db): cursor = db.execute("""""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """""") list(cursor.fetchall()) print(""Without progress_handler"") print(timeit.timeit(lambda: execute_query(db), number=100)) print(""progress_handler every 1000 ops"") print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1000), number=100)) print(""progress_handler every 1 op"") print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1), number=100)) ``` Results: ``` % python3 bench.py Without progress_handler 0.8789225700311363 progress_handler every 1000 ops 0.8829826560104266 progress_handler every 1 op 2.8892734259716235 ``` So running every 1000 ops makes almost no difference at all, but running every single op is a 3.2x performance degradation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074458506,https://api.github.com/repos/simonw/datasette/issues/1679,1074458506,IC_kwDOBm6k_c5ACu-K,9599,simonw,2022-03-21T21:53:47Z,2022-03-21T21:53:47Z,OWNER,"Oh interesting, it turns out there is ONE place in the code that sets the `ms` to less than 20 - this test fixture: https://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/tests/fixtures.py#L224-L226","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074454687,https://api.github.com/repos/simonw/datasette/issues/1679,1074454687,IC_kwDOBm6k_c5ACuCf,9599,simonw,2022-03-21T21:48:02Z,2022-03-21T21:48:02Z,OWNER,"Here's another microbenchmark that measures how many nanoseconds it takes to run 1,000 vmops: ```python import sqlite3 import time db = sqlite3.connect("":memory:"") i = 0 out = [] def count(): global i i += 1000 out.append(((i, time.perf_counter_ns()))) db.set_progress_handler(count, 1000) print(""Start:"", time.perf_counter_ns()) all = db.execute("""""" with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter limit 10000; """""").fetchall() print(""End:"", time.perf_counter_ns()) print() print(""So how long does it take to execute 1000 ops?"") prev_time_ns = None for i, time_ns in out: if prev_time_ns is not None: print(time_ns - prev_time_ns, ""ns"") prev_time_ns = time_ns ``` Running it: ``` % python nanobench.py Start: 330877620374821 End: 330877632515822 So how long does it take to execute 1000 ops? 47290 ns 49573 ns 48226 ns 45674 ns 53238 ns 47313 ns 52346 ns 48689 ns 47092 ns 87596 ns 69999 ns 52522 ns 52809 ns 53259 ns 52478 ns 53478 ns 65812 ns ``` 87596ns is 0.087596ms - so even a measure rate of every 1000 ops is easily finely grained enough to capture differences of less than 0.1ms. If anything I could bump that default 1000 up - and I can definitely eliminate the `if ms < 50` branch entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1679#issuecomment-1074459746,https://api.github.com/repos/simonw/datasette/issues/1679,1074459746,IC_kwDOBm6k_c5ACvRi,9599,simonw,2022-03-21T21:55:45Z,2022-03-21T21:55:45Z,OWNER,I'm going to change the original logic to set n=1 for times that are `<= 20ms` - and update the comments to make it more obvious what is happening.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,Research: how much overhead does the n=1 time limit have?, https://github.com/simonw/datasette/issues/1671#issuecomment-1074465536,https://api.github.com/repos/simonw/datasette/issues/1671,1074465536,IC_kwDOBm6k_c5ACwsA,9599,simonw,2022-03-21T22:04:31Z,2022-03-21T22:04:31Z,OWNER,"Oh this is fascinating! I replicated the bug (thanks for the steps to reproduce) and it looks like this is down to the following: <img width=""1276"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/159370986-1e2fc513-6d6c-4a2f-96dd-dccf5a680fe4.png""> Against views, `where has_expired = 1` returns different results from `where has_expired = '1'` This doesn't happen against tables because of SQLite's [type affinity](https://www.sqlite.org/datatype3.html#type_affinity) mechanism, which handles the type conversion automatically.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1074470568,https://api.github.com/repos/simonw/datasette/issues/1671,1074470568,IC_kwDOBm6k_c5ACx6o,9599,simonw,2022-03-21T22:11:14Z,2022-03-21T22:12:49Z,OWNER,"I wonder if this will be a problem with generated columns, or with SQLite strict tables? My hunch is that strict tables will continue to work without any changes, because https://www.sqlite.org/stricttables.html says nothing about their impact on comparison operations. I should test this to make absolutely sure though. Generated columns have a type, so my hunch is they will continue to work fine too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1074468450,https://api.github.com/repos/simonw/datasette/issues/1671,1074468450,IC_kwDOBm6k_c5ACxZi,9599,simonw,2022-03-21T22:08:35Z,2022-03-21T22:10:00Z,OWNER,"Relevant section of the SQLite documentation: [3.2. Affinity Of Expressions](https://www.sqlite.org/datatype3.html#affinity_of_expressions): > When an expression is a simple reference to a column of a real table (not a [VIEW](https://www.sqlite.org/lang_createview.html) or subquery) then the expression has the same affinity as the table column. In your example, `has_expired` is no longer a simple reference to a column of a real table, hence the bug. Then [4.2. Type Conversions Prior To Comparison](https://www.sqlite.org/datatype3.html#type_conversions_prior_to_comparison) fills in the rest: > SQLite may attempt to convert values between the storage classes INTEGER, REAL, and/or TEXT before performing a comparison. Whether or not any conversions are attempted before the comparison takes place depends on the type affinity of the operands. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1074478299,https://api.github.com/repos/simonw/datasette/issues/1671,1074478299,IC_kwDOBm6k_c5ACzzb,9599,simonw,2022-03-21T22:20:26Z,2022-03-21T22:20:26Z,OWNER,"Thinking about options for fixing this... The following query works fine: ```sql select * from test_view where cast(has_expired as text) = '1' ``` I don't want to start using this for every query, because one of the goals of Datasette is to help people who are learning SQL: - #1613 If someone clicks on ""View and edit SQL"" from a filtered table page I don't want them to have to wonder why that `cast` is there. But... for querying views, the `cast` turns out to be necessary. So one fix would be to get the SQL generating logic to use casts like this any time it is operating against a view. An even better fix would be to detect which columns in a view come from a table and which ones might not, and only use casts for the columns that aren't definitely from a table. The trick I was exploring here might be able to help with that: - #1293 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1075425513,https://api.github.com/repos/simonw/datasette/issues/1671,1075425513,IC_kwDOBm6k_c5AGbDp,9599,simonw,2022-03-22T17:31:53Z,2022-03-22T17:31:53Z,OWNER,"The alternative to using `cast` here would be for Datasette to convert the `""1""` to a `1` in Python code before passing it as a param. This feels a bit neater to me, but I still then need to solve the problem of how to identify the ""type"" of a column that I want to use in a query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1075428030,https://api.github.com/repos/simonw/datasette/issues/1671,1075428030,IC_kwDOBm6k_c5AGbq-,9599,simonw,2022-03-22T17:34:30Z,2022-03-22T17:34:30Z,OWNER,"No, I think I need to use `cast` - I can't think of any way to ask SQLite ""for this query, what types are the columns that will come back from it?"" Even the details from the `explain` trick explored in #1293 don't seem to come back with column type information: https://latest.datasette.io/fixtures?sql=explain+select+pk%2C+text1%2C+text2%2C+[name+with+.+and+spaces]+from+searchable_view+where+%22pk%22+%3D+%3Ap0&p0=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1075432283,https://api.github.com/repos/simonw/datasette/issues/1671,1075432283,IC_kwDOBm6k_c5AGctb,9599,simonw,2022-03-22T17:39:04Z,2022-03-22T17:43:12Z,OWNER,"Note that Datasette does already have special logic to convert parameters to integers for numeric comparisons like `>`: https://github.com/simonw/datasette/blob/c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c/datasette/filters.py#L203-L212 Though... it looks like there's a bug in that? It doesn't account for `float` values - `""3.5"".isdigit()` return `False` - probably for the best, because `int(3.5)` would break that value anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1671#issuecomment-1075435185,https://api.github.com/repos/simonw/datasette/issues/1671,1075435185,IC_kwDOBm6k_c5AGdax,9599,simonw,2022-03-22T17:42:09Z,2022-03-22T17:42:09Z,OWNER,"Also made me realize that this query: ```sql select * from sortable where sortable > :p0 ``` Only works here thanks to the column affinity thing kicking in too: https://latest.datasette.io/fixtures?sql=select+*+from+sortable+where+sortable+%3E+%3Ap0&p0=70","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,Filters fail to work correctly against calculated numeric columns returned by SQL views because type affinity rules do not apply, https://github.com/simonw/datasette/issues/1681#issuecomment-1075437598,https://api.github.com/repos/simonw/datasette/issues/1681,1075437598,IC_kwDOBm6k_c5AGeAe,9599,simonw,2022-03-22T17:44:42Z,2022-03-22T17:45:04Z,OWNER,"My hunch is that this mechanism doesn't actually do anything useful at all, because of the type conversion that automatically happens for data from tables based on the column type affinities, see: - #1671 So either remove the `self.numeric` type conversion bit entirely, or prove that it is necessary and upgrade it to be able to handle floating point values too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1177101697,Potential bug in numeric handling where_clause for filters, https://github.com/simonw/datasette/issues/1681#issuecomment-1075438684,https://api.github.com/repos/simonw/datasette/issues/1681,1075438684,IC_kwDOBm6k_c5AGeRc,9599,simonw,2022-03-22T17:45:50Z,2022-03-22T17:49:09Z,OWNER,"I would expect this to break against SQL views that include calculated columns though - something like this: ```sql create view this_will_break as select pk + 1 as pk_plus_one, 0.5 as score from searchable; ``` Confirmed: the filter interface for that view plain doesn't work for any comparison against that table - except for `score > 0` since `0` is converted to an integer. `0.1` breaks though because it doesn't get converted as it doesn't match `.isdigit()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1177101697,Potential bug in numeric handling where_clause for filters, https://github.com/simonw/datasette/issues/1670#issuecomment-1076638278,https://api.github.com/repos/simonw/datasette/issues/1670,1076638278,IC_kwDOBm6k_c5ALDJG,9599,simonw,2022-03-23T17:50:55Z,2022-03-23T17:50:55Z,OWNER,"Release notes are mostly written for the alpha, just need to clean them up a bit https://github.com/simonw/datasette/blob/c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c/docs/changelog.rst#061a0-2022-03-19","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,Ship Datasette 0.61, https://github.com/simonw/datasette/pull/1574#issuecomment-1076645636,https://api.github.com/repos/simonw/datasette/issues/1574,1076645636,IC_kwDOBm6k_c5ALE8E,9599,simonw,2022-03-23T17:56:35Z,2022-03-23T17:56:35Z,OWNER,I'd actually like to switch to slim as the default - I think Datasette should ship the smallest possible container that can still support extra packages being installed using `apt-get install`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084193403,introduce new option for datasette package to use a slim base image, https://github.com/simonw/datasette/pull/1665#issuecomment-1076644362,https://api.github.com/repos/simonw/datasette/issues/1665,1076644362,IC_kwDOBm6k_c5ALEoK,9599,simonw,2022-03-23T17:55:39Z,2022-03-23T17:55:39Z,OWNER,Thanks for the PR - I spotted an error about this and went through and fixed this in all of my repos the other day: https://github.com/search?o=desc&q=user%3Asimonw+google-github-actions%2Fsetup-gcloud%40v0&s=committer-date&type=Commits,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173828092,Pin setup-gcloud to v0 instead of master, https://github.com/simonw/datasette/issues/1670#issuecomment-1076647495,https://api.github.com/repos/simonw/datasette/issues/1670,1076647495,IC_kwDOBm6k_c5ALFZH,9599,simonw,2022-03-23T17:58:16Z,2022-03-23T17:58:16Z,OWNER,"I think the release notes are fine, but they need an opening paragraph highlighting the changes that are most likely to break backwards compatibility.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,Ship Datasette 0.61, https://github.com/simonw/datasette/issues/1670#issuecomment-1076652046,https://api.github.com/repos/simonw/datasette/issues/1670,1076652046,IC_kwDOBm6k_c5ALGgO,9599,simonw,2022-03-23T18:02:30Z,2022-03-23T18:02:30Z,OWNER,"Two new things to add to the release notes from https://github.com/simonw/datasette/compare/0.61a0...main - https://github.com/simonw/datasette/issues/1678 - https://github.com/simonw/datasette/issues/1675 (now also a documented API)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,Ship Datasette 0.61, https://github.com/simonw/datasette/issues/1670#issuecomment-1076666293,https://api.github.com/repos/simonw/datasette/issues/1670,1076666293,IC_kwDOBm6k_c5ALJ-1,9599,simonw,2022-03-23T18:16:29Z,2022-03-23T18:16:29Z,OWNER,https://docs.datasette.io/en/stable/changelog.html#v0-61,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,Ship Datasette 0.61, https://github.com/simonw/datasette/issues/1670#issuecomment-1076665837,https://api.github.com/repos/simonw/datasette/issues/1670,1076665837,IC_kwDOBm6k_c5ALJ3t,9599,simonw,2022-03-23T18:16:01Z,2022-03-23T18:16:01Z,OWNER,"https://github.com/simonw/datasette/releases/tag/0.61 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,Ship Datasette 0.61, https://github.com/simonw/datasette/issues/1670#issuecomment-1076683297,https://api.github.com/repos/simonw/datasette/issues/1670,1076683297,IC_kwDOBm6k_c5ALOIh,9599,simonw,2022-03-23T18:32:32Z,2022-03-23T18:32:32Z,OWNER,Added this to news on https://datasette.io/ https://github.com/simonw/datasette.io/commit/fd3ec57cdd5b935f75cbf52a86b3aabf2c97d217,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,Ship Datasette 0.61, https://github.com/simonw/datasette/issues/1682#issuecomment-1076696791,https://api.github.com/repos/simonw/datasette/issues/1682,1076696791,IC_kwDOBm6k_c5ALRbX,9599,simonw,2022-03-23T18:45:49Z,2022-03-23T18:45:49Z,OWNER,"The problem is here in `QueryView`: https://github.com/simonw/datasette/blob/d7c793d7998388d915f8d270079c68a77a785051/datasette/views/database.py#L206-L238 It should be resolving `database` based on the route path, as seen in other methods like this one: https://github.com/simonw/datasette/blob/d7c793d7998388d915f8d270079c68a77a785051/datasette/views/table.py#L270-L279 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178521513,SQL queries against databases with different routes are broken, https://github.com/simonw/datasette/issues/1688#issuecomment-1079582485,https://api.github.com/repos/simonw/datasette/issues/1688,1079582485,IC_kwDOBm6k_c5AWR8V,9599,simonw,2022-03-26T03:15:34Z,2022-03-26T03:15:34Z,OWNER,"Yup, you're right in what you figured out here: stand-alone plugins can't currently package static assets other then using the static folder. The `datasette-plugin` cookiecutter template should make creating a Python package pretty easy though: https://github.com/simonw/datasette-plugin You can run that yourself, or you can run it using this GitHub template repository: https://github.com/simonw/datasette-plugin-template-repository ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1181432624,[plugins][documentation] Is it possible to serve per-plugin static folders when writing one-off (single file) plugins?, https://github.com/simonw/datasette/issues/1689#issuecomment-1079779040,https://api.github.com/repos/simonw/datasette/issues/1689,1079779040,IC_kwDOBm6k_c5AXB7g,9599,simonw,2022-03-26T21:35:57Z,2022-03-26T21:35:57Z,OWNER,Fixed: https://docs.datasette.io/en/latest/internals.html#add-message-request-message-type-datasette-info,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182065616,datasette.add_message() documentation is incorrect, https://github.com/simonw/datasette/issues/1690#issuecomment-1079788346,https://api.github.com/repos/simonw/datasette/issues/1690,1079788346,IC_kwDOBm6k_c5AXEM6,9599,simonw,2022-03-26T22:42:40Z,2022-03-26T22:42:40Z,OWNER,"I don't want to do a `response.set_actor_cookie()` method because I like `Response` not to carry too many Datasette-specific features. So `datasette.set_actor_cookie(response, actor, expire_after=None)` would be a better place for this I think.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182141761,"Idea: `datasette.set_actor_cookie(response, actor)`", https://github.com/simonw/datasette/issues/1690#issuecomment-1079788375,https://api.github.com/repos/simonw/datasette/issues/1690,1079788375,IC_kwDOBm6k_c5AXENX,9599,simonw,2022-03-26T22:43:00Z,2022-03-26T22:43:00Z,OWNER,Then I can update this section of the documentation which currently recommends the above pattern: https://docs.datasette.io/en/stable/authentication.html#the-ds-actor-cookie,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182141761,"Idea: `datasette.set_actor_cookie(response, actor)`", https://github.com/simonw/datasette/issues/1695#issuecomment-1082617241,https://api.github.com/repos/simonw/datasette/issues/1695,1082617241,IC_kwDOBm6k_c5Ah22Z,9599,simonw,2022-03-30T04:45:55Z,2022-03-30T04:45:55Z,OWNER,"Relevant template: https://github.com/simonw/datasette/blob/e73fa72917ca28c152208d62d07a490c81cadf52/datasette/templates/table.html#L168-L172 Populated from here: https://github.com/simonw/datasette/blob/c496f2b663ff0cef908ffaaa68b8cb63111fb5f2/datasette/facets.py#L246-L253","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1185868354,Option to un-filter facet not shown for `?col__exact=value`, https://github.com/simonw/datasette/issues/1695#issuecomment-1082617386,https://api.github.com/repos/simonw/datasette/issues/1695,1082617386,IC_kwDOBm6k_c5Ah24q,9599,simonw,2022-03-30T04:46:18Z,2022-03-30T04:46:18Z,OWNER,"` selected = (column_qs, str(row[""value""])) in qs_pairs` is wrong.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1185868354,Option to un-filter facet not shown for `?col__exact=value`, https://github.com/simonw/datasette/issues/1692#issuecomment-1082661795,https://api.github.com/repos/simonw/datasette/issues/1692,1082661795,IC_kwDOBm6k_c5AiBuj,9599,simonw,2022-03-30T06:11:41Z,2022-03-30T06:11:41Z,OWNER,This is a good idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182227211,[plugins][feature request]: Support additional script tag attributes when loading custom JS, https://github.com/simonw/datasette/issues/1692#issuecomment-1082663746,https://api.github.com/repos/simonw/datasette/issues/1692,1082663746,IC_kwDOBm6k_c5AiCNC,9599,simonw,2022-03-30T06:14:39Z,2022-03-30T06:14:51Z,OWNER,"I like your design, though I think it should be `""nomodule"": True` for consistency with the other options. I think `""async"": True` is worth supporting too.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182227211,[plugins][feature request]: Support additional script tag attributes when loading custom JS, https://github.com/simonw/datasette/issues/1696#issuecomment-1083351437,https://api.github.com/repos/simonw/datasette/issues/1696,1083351437,IC_kwDOBm6k_c5AkqGN,9599,simonw,2022-03-30T16:20:49Z,2022-03-30T16:21:02Z,OWNER,"Maybe like this: <img width=""499"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/160883280-c19d5a22-e923-491f-8bf4-1a4f5215d684.png""> ```html <h3>283 rows where dcode = 3 <span style=""color: #aaa; font-size: 0.9em"">(Human Related: Other)</span> </h3> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1186696202,Show foreign key label when filtering, https://github.com/simonw/datasette/issues/1697#issuecomment-1085323192,https://api.github.com/repos/simonw/datasette/issues/1697,1085323192,IC_kwDOBm6k_c5AsLe4,9599,simonw,2022-04-01T02:01:51Z,2022-04-01T02:01:51Z,OWNER,"Huh, turns out `Request.fake()` wasn't yet documented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1189113609,"`Request.fake(..., url_vars={})`", https://github.com/simonw/datasette/issues/1698#issuecomment-1086784547,https://api.github.com/repos/simonw/datasette/issues/1698,1086784547,IC_kwDOBm6k_c5AxwQj,9599,simonw,2022-04-03T06:10:24Z,2022-04-03T06:10:24Z,OWNER,Warning added here: https://docs.datasette.io/en/latest/publish.html#publishing-to-google-cloud-run,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1190828163,Add a warning about bots and Cloud Run, https://github.com/simonw/datasette/issues/1715#issuecomment-1106989581,https://api.github.com/repos/simonw/datasette/issues/1715,1106989581,IC_kwDOBm6k_c5B-1IN,9599,simonw,2022-04-22T23:03:29Z,2022-04-22T23:03:29Z,OWNER,I'm having second thoughts about injecting `request` - might be better to have the view function pull the relevant pieces out of the request before triggering the rest of the resolution.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1715#issuecomment-1106908642,https://api.github.com/repos/simonw/datasette/issues/1715,1106908642,IC_kwDOBm6k_c5B-hXi,9599,simonw,2022-04-22T21:47:55Z,2022-04-22T21:47:55Z,OWNER,"I need a `asyncio.Registry` with functions registered to perform the role of the table view. Something like this perhaps: ```python def table_html_context(facet_results, query, datasette, rows): return {...} ``` That then gets called like this: ```python async def view(request): registry = Registry(facet_results, query, datasette, rows) context = await registry.resolve(table_html, request=request, datasette=datasette) return Reponse.html(await datasette.render(""table.html"", context) ``` It's also interesting to start thinking about this from a Python client library point of view. If I'm writing code outside of the HTTP request cycle, what would it look like? One thing I could do: break out is the code that turns a request into a list of pairs extracted from the request - this code here: https://github.com/simonw/datasette/blob/8338c66a57502ef27c3d7afb2527fbc0663b2570/datasette/views/table.py#L442-L449 I could turn that into a typed dependency injection function like this: ```python def filter_args(request: Request) -> List[Tuple[str, str]]: # Arguments that start with _ and don't contain a __ are # special - things like ?_search= - and should not be # treated as filters. filter_args = [] for key in request.args: if not (key.startswith(""_"") and ""__"" not in key): for v in request.args.getlist(key): filter_args.append((key, v)) return filter_args ``` Then I can either pass a `request` into a `.resolve()` call, or I can instead skip that function by passing: ```python output = registry.resolve(table_context, filter_args=[(""foo"", ""bar"")]) ``` I do need to think about where plugins get executed in all of this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject, https://github.com/simonw/datasette/issues/1716#issuecomment-1106923258,https://api.github.com/repos/simonw/datasette/issues/1716,1106923258,IC_kwDOBm6k_c5B-k76,9599,simonw,2022-04-22T22:02:07Z,2022-04-22T22:02:07Z,OWNER,"https://github.com/simonw/datasette/blame/main/datasette/views/base.py <img width=""1373"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/164801564-d8a11ce9-7d9b-4e85-8947-a547d2986ef3.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212838949,Configure git blame to ignore Black commit, https://github.com/simonw/datasette/issues/1715#issuecomment-1106945876,https://api.github.com/repos/simonw/datasette/issues/1715,1106945876,IC_kwDOBm6k_c5B-qdU,9599,simonw,2022-04-22T22:24:29Z,2022-04-22T22:24:29Z,OWNER,"Looking at the start of `TableView.data()`: https://github.com/simonw/datasette/blob/d57c347f35bcd8cff15f913da851b4b8eb030867/datasette/views/table.py#L333-L346 I'm going to resolve `table_name` and `database` from the URL - `table_name` will be a string, `database` will be the DB object returned by `datasette.get_database()`. Then those can be passed in separately too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212823665,Refactor TableView to use asyncinject,