{"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1053973425", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1053973425, "node_id": "IC_kwDOBm6k_c4-0lux", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-28T07:40:12Z", "updated_at": "2022-02-28T07:40:12Z", "author_association": "OWNER", "body": "If I make this change it will break existing links to one of the oldest Datasette demos: http://fivethirtyeight.datasettes.com/fivethirtyeight/avengers%2Favengers\r\n\r\nA plugin that fixes those by redirecting them on 404 would be neat.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/327#issuecomment-1043609198", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/327", "id": 1043609198, "node_id": "IC_kwDOBm6k_c4-NDZu", "user": {"value": 208018, "label": "dholth"}, "created_at": "2022-02-17T23:21:36Z", "updated_at": "2022-02-17T23:33:01Z", "author_association": "NONE", "body": "On fly.io. This particular database goes from 1.4GB to 200M. Slower, part of that might be having no `--inspect-file`?\r\n\r\n```\r\n$ datasette publish fly ... --generate-dir /tmp/deploy-this\r\n...\r\n$ mksquashfs large.db large.squashfs\r\n$ rm large.db # don't accidentally put it in the image\r\n$ cat Dockerfile\r\nFROM python:3.8\r\nCOPY . /app\r\nWORKDIR /app\r\n\r\nENV DATASETTE_SECRET 'xyzzy'\r\nRUN pip install -U datasette\r\n# RUN datasette inspect large.db --inspect-file inspect-data.json\r\nENV PORT 8080\r\nEXPOSE 8080\r\nCMD mount -o loop -t squashfs large.squashfs /mnt; datasette serve --host 0.0.0.0 -i /mnt/large.db --cors --port $PORT\r\n```\r\n\r\nIt would also be possible to copy the file onto the ~6GB available on the ephemeral container filesystem on startup. A little against the spirit of the thing? On this example the whole docker image is 2.42 GB and the squashfs version is 1.14 GB.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 335200136, "label": "Explore if SquashFS can be used to shrink size of packaged Docker containers"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/327#issuecomment-1043626870", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/327", "id": 1043626870, "node_id": "IC_kwDOBm6k_c4-NHt2", "user": {"value": 208018, "label": "dholth"}, "created_at": "2022-02-17T23:37:24Z", "updated_at": "2022-02-17T23:37:24Z", "author_association": "NONE", "body": "On second thought any kind of quick-to-decompress-on-startup could be helpful if we're paying for the container registry and deployment bandwidth but not ephemeral storage.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 335200136, "label": "Explore if SquashFS can be used to shrink size of packaged Docker containers"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045131086", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045131086, "node_id": "IC_kwDOBm6k_c4-S29O", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T20:22:13Z", "updated_at": "2022-02-18T20:22:47Z", "author_association": "OWNER", "body": "Should it encode `%` symbols too, since they have a special meaning in URLs and we can't guarantee that every single web server / proxy out there will round-trip them safely using percentage encoding? If so, would need to pick a different encoding character for them. Maybe `%` becomes `-p` - and in that case `/` could become `-s` too.\r\n\r\nIs it worth expanding dash-encoding outside of just `/` and `-` and `.` though? Not sure.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045134050", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045134050, "node_id": "IC_kwDOBm6k_c4-S3ri", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T20:25:04Z", "updated_at": "2022-02-18T20:25:04Z", "author_association": "OWNER", "body": "Here's a useful modern spec for how existing URL percentage encoding is supposed to work: https://url.spec.whatwg.org/#percent-encoded-bytes", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045024276", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045024276, "node_id": "IC_kwDOBm6k_c4-Sc4U", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:01:42Z", "updated_at": "2022-02-18T19:55:24Z", "author_association": "OWNER", "body": "> Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly.\r\n```python\r\ndef dash_encode(s):\r\n return s.replace(\"-\", \"--\").replace(\".\", \"-.\").replace(\"/\", \"-/\")\r\n\r\ndef dash_decode(s):\r\n return s.replace(\"-/\", \"/\").replace(\"-.\", \".\").replace(\"--\", \"-\")\r\n```\r\n\r\n```pycon\r\n>>> dash_encode(\"foo/bar/baz.csv\")\r\n'foo-/bar-/baz-.csv'\r\n>>> dash_decode('foo-/bar-/baz-.csv')\r\n'foo/bar/baz.csv'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045027067", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045027067, "node_id": "IC_kwDOBm6k_c4-Sdj7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:03:26Z", "updated_at": "2022-02-18T19:03:26Z", "author_association": "OWNER", "body": "(If I make this change it may break some existing Datasette installations when they upgrade - I could try and build a plugin for them which triggers on 404s and checks to see if the old format would return a 200 response, then returns that.)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045032377", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045032377, "node_id": "IC_kwDOBm6k_c4-Se25", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:06:50Z", "updated_at": "2022-02-18T19:06:50Z", "author_association": "OWNER", "body": "How does URL routing for https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv work?\r\n\r\nRight now it's https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/app.py#L1098-L1101\r\n\r\nThat's not going to capture the dot-dash encoding version of that table name:\r\n```pycon\r\n>>> dot_dash_encode(\"table/with/slashes.csv\")\r\n'table-/with-/slashes-.csv'\r\n```\r\nProbably needs a fancy regex trick like a negative lookbehind assertion or similar.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045055772", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045055772, "node_id": "IC_kwDOBm6k_c4-Skkc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:23:33Z", "updated_at": "2022-02-18T19:25:42Z", "author_association": "OWNER", "body": "I want a match for this URL:\r\n\r\n /db/table-/with-/slashes-.csv\r\n\r\nMaybe this:\r\n\r\n ^/(?P[^/]+)/(?P([^/]*|(\\-/)*|(\\-\\.)*|(\\.\\.)*)*$)\r\n\r\nHere we are matching a sequence of:\r\n\r\n ([^/]*|(\\-/)*|(\\-\\.)*|(\\-\\-)*)*\r\n\r\nSo a combination of not-slashes OR -/ or -. Or -- sequences\r\n\r\n\"image\"\r\n\r\n ^/(?P[^/]+)/(?P([^/]*|(\\-/)*|(\\-\\.)*|(\\-\\-)*)*$)\r\n\r\nTry that with non-capturing bits:\r\n\r\n ^/(?P[^/]+)/(?P(?:[^/]*|(?:\\-/)*|(?:\\-\\.)*|(?:\\-\\-)*)*$)\r\n\r\n`(?:[^/]*|(?:\\-/)*|(?:\\-\\.)*|(?:\\-\\-)*)*` visualized is:\r\n\r\n\"image\"\r\n\r\nHere's the explanation on regex101.com https://regex101.com/r/CPnsIO/1\r\n\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045059427", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045059427, "node_id": "IC_kwDOBm6k_c4-Sldj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:26:25Z", "updated_at": "2022-02-18T19:26:25Z", "author_association": "OWNER", "body": "With this new pattern I could probably extract out the optional `.json` format string as part of the initial route capturing regex too, rather than the current `table_and_format` hack.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045069481, "node_id": "IC_kwDOBm6k_c4-Sn6p", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:34:41Z", "updated_at": "2022-03-05T21:32:22Z", "author_association": "OWNER", "body": "I think I got format extraction working! https://regex101.com/r/A0bW1D/1\r\n\r\n ^/(?P[^/]+)/(?P(?:[^\\/\\-\\.]*|(?:\\-/)*|(?:\\-\\.)*|(?:\\-\\-)*)*?)(?:(?\\w+))?$\r\n\r\nI had to make that crazy inner one even more complicated to stop it from capturing `.` that was not part of `-.`.\r\n\r\n (?:[^\\/\\-\\.]*|(?:\\-/)*|(?:\\-\\.)*|(?:\\-\\-)*)*\r\n\r\nVisualized:\r\n\r\n\"image\"\r\n\r\nSo now I have a regex which can extract out the dot-encoded table name AND spot if there is an optional `.format` at the end:\r\n\r\n\"image\"\r\n\r\nIf I end up using this in Datasette it's going to need VERY comprehensive unit tests and inline documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045077590", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045077590, "node_id": "IC_kwDOBm6k_c4-Sp5W", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:41:37Z", "updated_at": "2022-02-18T19:42:41Z", "author_association": "OWNER", "body": "Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where \"system\" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence.\r\n\r\nAnd I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too.\r\n\r\nMaybe change this system to use `.` as the escaping character instead of `-`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045075207", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045075207, "node_id": "IC_kwDOBm6k_c4-SpUH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:39:35Z", "updated_at": "2022-02-18T19:40:13Z", "author_association": "OWNER", "body": "> And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this:\r\n> \r\n> * `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version\r\n> * `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version\r\n> * `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version\r\n\r\nHere's what those look like with the updated version of `dot_dash_encode()` that also encodes `/` as `-/`:\r\n\r\n- `/db/-/db-/table---.csv-.csv` - HTML\r\n- `/db/-/db-/table---.csv-.csv.csv` - CSV\r\n- `/db/-/db-/table---.csv-.csv.json` - JSON\r\n\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045081042", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045081042, "node_id": "IC_kwDOBm6k_c4-SqvS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:44:12Z", "updated_at": "2022-02-18T19:51:34Z", "author_association": "OWNER", "body": "```python\r\ndef dot_encode(s):\r\n return s.replace(\".\", \"..\").replace(\"/\", \"./\")\r\n\r\ndef dot_decode(s):\r\n return s.replace(\"./\", \"/\").replace(\"..\", \".\")\r\n```\r\nNo need for hyphen encoding in this variant at all, which simplifies things a bit.\r\n\r\n(Update: this is flawed, see https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045086033, "node_id": "IC_kwDOBm6k_c4-Sr9R", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:47:43Z", "updated_at": "2022-02-18T19:51:11Z", "author_association": "OWNER", "body": "- https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv\r\n- https://til.simonwillison.net/-/asgi-scope/db/./db./table-..csv..csv\r\n\r\nDo both of those survive the round-trip to populate `raw_path` correctly?\r\n\r\nNo! In both cases the `/./` bit goes missing.\r\n\r\nIt looks like this might even be a client issue - `curl` shows me this:\r\n\r\n```\r\n~ % curl -vv -i 'https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv'\r\n* Trying 216.239.32.21:443...\r\n* Connected to datasette.io (216.239.32.21) port 443 (#0)\r\n* ALPN, offering http/1.1\r\n* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\r\n* Server certificate: datasette.io\r\n* Server certificate: R3\r\n* Server certificate: ISRG Root X1\r\n> GET /-/asgi-scope/db/db./table-..csv..csv HTTP/1.1\r\n```\r\nSo `curl` decided to turn `/-/asgi-scope/db/./db./table` into `/-/asgi-scope/db/db./table` before even sending the request.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045082891", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045082891, "node_id": "IC_kwDOBm6k_c4-SrML", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:45:32Z", "updated_at": "2022-02-18T19:45:32Z", "author_association": "OWNER", "body": "```pycon\r\n>>> dot_encode(\"/db/table-.csv.csv\")\r\n'./db./table-..csv..csv'\r\n>>> dot_decode('./db./table-..csv..csv')\r\n'/db/table-.csv.csv'\r\n```\r\nI worry that web servers might treat `./` in a special way though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045095348", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045095348, "node_id": "IC_kwDOBm6k_c4-SuO0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:53:48Z", "updated_at": "2022-02-18T19:53:48Z", "author_association": "OWNER", "body": "> Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where \"system\" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence.\r\n> \r\n> And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too.\r\n\r\nI don't think this matters. The new regex does indeed capture that kind of page:\r\n\r\n\"image\"\r\n\r\nBut Datasette goes through configured route regular expressions in order - so I can have the regex that captures `/db/-/special` routes listed before the one that captures tables and formats.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045099290", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045099290, "node_id": "IC_kwDOBm6k_c4-SvMa", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T19:56:18Z", "updated_at": "2022-02-18T19:56:30Z", "author_association": "OWNER", "body": "> ```python\r\n> def dash_encode(s):\r\n> return s.replace(\"-\", \"--\").replace(\".\", \"-.\").replace(\"/\", \"-/\")\r\n> \r\n> def dash_decode(s):\r\n> return s.replace(\"-/\", \"/\").replace(\"-.\", \".\").replace(\"--\", \"-\")\r\n> ```\r\n\r\nI think **dash-encoding** (new name for this) is the right way forward here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045108611", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045108611, "node_id": "IC_kwDOBm6k_c4-SxeD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T20:02:19Z", "updated_at": "2022-02-18T20:08:34Z", "author_association": "OWNER", "body": "One other potential variant:\r\n```python\r\ndef dash_encode(s):\r\n return s.replace(\"-\", \"-dash-\").replace(\".\", \"-dot-\").replace(\"/\", \"-slash-\")\r\n\r\ndef dash_decode(s):\r\n return s.replace(\"-slash-\", \"/\").replace(\"-dot-\", \".\").replace(\"-dash-\", \"-\")\r\n```\r\nExcept this has bugs - it doesn't round-trip safely, because it can get confused about things like `-dash-slash-` in terms of is that a `-dash-` or a `-slash-`?\r\n```pycon\r\n>>> dash_encode(\"/db/table-.csv.csv\")\r\n'-slash-db-slash-table-dash--dot-csv-dot-csv'\r\n>>> dash_decode('-slash-db-slash-table-dash--dot-csv-dot-csv')\r\n'/db/table-.csv.csv'\r\n>>> dash_encode('-slash-db-slash-table-dash--dot-csv-dot-csv')\r\n'-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv'\r\n>>> dash_decode('-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv')\r\n'-dash/dash-db-dash/dash-table-dash--dash.dash-csv-dash.dash-csv'\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045111309", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045111309, "node_id": "IC_kwDOBm6k_c4-SyIN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T20:04:24Z", "updated_at": "2022-02-18T20:05:40Z", "author_association": "OWNER", "body": "This made me worry that my current `dash_decode()` implementation had unknown round-trip bugs, but thankfully this works OK:\r\n```pycon\r\n>>> dash_encode(\"/db/table-.csv.csv\")\r\n'-/db-/table---.csv-.csv'\r\n>>> dash_encode('-/db-/table---.csv-.csv')\r\n'---/db---/table-------.csv---.csv'\r\n>>> dash_decode('---/db---/table-------.csv---.csv')\r\n'-/db-/table---.csv-.csv'\r\n>>> dash_decode('-/db-/table---.csv-.csv')\r\n'/db/table-.csv.csv'\r\n``` \r\nThe regex still works against that double-encoded example too:\r\n\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045117304", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045117304, "node_id": "IC_kwDOBm6k_c4-Szl4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T20:09:22Z", "updated_at": "2022-02-18T20:09:22Z", "author_association": "OWNER", "body": "Adopting this could result in supporting database files with surprising characters in their filename too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1045269544", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1045269544, "node_id": "IC_kwDOBm6k_c4-TYwo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-18T22:19:29Z", "updated_at": "2022-02-18T22:19:29Z", "author_association": "OWNER", "body": "Note that I've ruled out using `Accept: application/json` to return JSON because it turns out Cloudflare and potentially other CDNs ignore the `Vary: Accept` header entirely:\r\n- https://github.com/simonw/datasette/issues/1534", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1049114724", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1049114724, "node_id": "IC_kwDOBm6k_c4-iDhk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-23T19:04:40Z", "updated_at": "2022-02-23T19:04:40Z", "author_association": "OWNER", "body": "I'm going to try dash encoding for table names (and row IDs) in a branch and see how I like it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1049124390", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1049124390, "node_id": "IC_kwDOBm6k_c4-iF4m", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-23T19:15:00Z", "updated_at": "2022-02-23T19:15:00Z", "author_association": "OWNER", "body": "I'll start by modifying this function: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/utils/__init__.py#L732-L749\r\n\r\nLater I want to move this to the routing layer to split out `format` automatically, as seen in the regexes here: https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1049126151", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1049126151, "node_id": "IC_kwDOBm6k_c4-iGUH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-23T19:17:01Z", "updated_at": "2022-02-23T19:17:01Z", "author_association": "OWNER", "body": "Actually the relevant code looks to be: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/views/base.py#L481-L498", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1641#issuecomment-1049879118", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1641", "id": 1049879118, "node_id": "IC_kwDOBm6k_c4-k-JO", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-02-24T13:49:26Z", "updated_at": "2022-02-24T13:49:26Z", "author_association": "CONTRIBUTOR", "body": "maybe worth considering adding buttons for paren, asterisk, etc. under the input text box on mobile?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1149310456, "label": "Tweak mobile keyboard settings"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/260#issuecomment-1051473892", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/260", "id": 1051473892, "node_id": "IC_kwDOBm6k_c4-rDfk", "user": {"value": 596279, "label": "zaneselvans"}, "created_at": "2022-02-26T02:24:15Z", "updated_at": "2022-02-26T02:24:15Z", "author_association": "NONE", "body": "Is there already functionality that can be used to validate the `metadata.json` file? Is there a JSON Schema that defines it? Or a validation that's available via datasette with Python? We're working on [automatically building the metadata](https://github.com/catalyst-cooperative/pudl/pull/1479) in CI and when we deploy to cloud run, and it would be nice to be able to check whether the the metadata we're outputting is valid in our tests.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323223872, "label": "Validate metadata.json on startup"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1405#issuecomment-888694144", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1405", "id": 888694144, "node_id": "IC_kwDOBm6k_c40-GWA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-28T23:51:59Z", "updated_at": "2021-07-28T23:51:59Z", "author_association": "OWNER", "body": "https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/datasette/utils/__init__.py#L918-L926", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 955316250, "label": "utils.parse_metadata() should be a documented internal function"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1405#issuecomment-888694261", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1405", "id": 888694261, "node_id": "IC_kwDOBm6k_c40-GX1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-28T23:52:21Z", "updated_at": "2021-07-28T23:52:21Z", "author_association": "OWNER", "body": "Document that it can raise a `BadMetadataError` exception.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 955316250, "label": "utils.parse_metadata() should be a documented internal function"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1402#issuecomment-886968648", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1402", "id": 886968648, "node_id": "IC_kwDOBm6k_c403hFI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-26T19:30:14Z", "updated_at": "2021-07-26T19:30:14Z", "author_association": "OWNER", "body": "I really like this idea. I was thinking it might make a good plugin, but there's not a great mechanism for plugins to inject extra `` content at the moment - plus this actually feels like a reasonable feature for Datasette core itself.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 951185411, "label": "feature request: social meta tags"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1402#issuecomment-886969541", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1402", "id": 886969541, "node_id": "IC_kwDOBm6k_c403hTF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-26T19:31:40Z", "updated_at": "2021-07-26T19:31:40Z", "author_association": "OWNER", "body": "Datasette could do a pretty good job of this by default, using `twitter:card` and `og:url` tags - like on https://til.simonwillison.net/jq/extracting-objects-recursively\r\n\r\nI could also provide a mechanism to customize these - in particular to add images of some sort.\r\n\r\nIt feels like something that should tie in to the metadata mechanism.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 951185411, "label": "feature request: social meta tags"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1404#issuecomment-887095569", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1404", "id": 887095569, "node_id": "IC_kwDOBm6k_c404AER", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-26T23:27:07Z", "updated_at": "2021-07-26T23:27:07Z", "author_association": "OWNER", "body": "Updated documentation: https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/docs/plugin_hooks.rst#register-routes-datasette", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 953352015, "label": "`register_routes()` hook should take `datasette` argument"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/759#issuecomment-881125124", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/759", "id": 881125124, "node_id": "IC_kwDOBm6k_c40hOcE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T02:11:48Z", "updated_at": "2021-07-16T02:11:54Z", "author_association": "OWNER", "body": "I added `\"searchmode\": \"raw\"` as a supported option for table metadata in #1389 and released that in Datasette 0.58.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 612673948, "label": "fts search on a column doesn't work anymore due to escape_fts"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1394#issuecomment-881129149", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1394", "id": 881129149, "node_id": "IC_kwDOBm6k_c40hPa9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T02:23:32Z", "updated_at": "2021-07-16T02:23:32Z", "author_association": "OWNER", "body": "Wrote about this in the annotated release notes for 0.58: https://simonwillison.net/2021/Jul/16/datasette-058/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944870799, "label": "Big performance boost on faceting: skip the inner order by"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881204782", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881204782, "node_id": "IC_kwDOBm6k_c40hh4u", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T06:14:12Z", "updated_at": "2021-07-16T06:14:12Z", "author_association": "OWNER", "body": "Here's the traceback I got from `datasette-graphql` (annoyingly only running the tests in GitHub Actions CI - I've not been able to replicate on my laptop yet):\r\n\r\n```\r\ntests/test_utils.py . [100%]\r\n\r\n=================================== FAILURES ===================================\r\n_________________________ test_graphql_examples[path0] _________________________\r\n\r\nds = \r\npath = PosixPath('/home/runner/work/datasette-graphql/datasette-graphql/examples/filters.md')\r\n\r\n @pytest.mark.asyncio\r\n @pytest.mark.parametrize(\r\n \"path\", (pathlib.Path(__file__).parent.parent / \"examples\").glob(\"*.md\")\r\n )\r\n async def test_graphql_examples(ds, path):\r\n content = path.read_text()\r\n query = graphql_re.search(content)[1]\r\n try:\r\n variables = variables_re.search(content)[1]\r\n except TypeError:\r\n variables = \"{}\"\r\n expected = json.loads(json_re.search(content)[1])\r\n response = await ds.client.post(\r\n \"/graphql\",\r\n json={\r\n \"query\": query,\r\n \"variables\": json.loads(variables),\r\n },\r\n )\r\n> assert response.status_code == 200, response.json()\r\nE AssertionError: {'data': {'repos_arraycontains': None, 'users_contains': None, 'users_date': None, 'users_endswith': None, ...}, 'erro...\", 'path': ['users_gt']}, {'locations': [{'column': 5, 'line': 34}], 'message': \"'rows'\", 'path': ['users_gte']}, ...]}\r\nE assert 500 == 200\r\nE + where 500 = .status_code\r\n\r\ntests/test_graphql.py:142: AssertionError\r\n----------------------------- Captured stderr call -----------------------------\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\ntable databases already exists\r\nTraceback (most recent call last):\r\n File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py\", line 1171, in route_path\r\n response = await view(request, send)\r\n File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py\", line 151, in view\r\n request, **request.scope[\"url_route\"][\"kwargs\"]\r\n File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py\", line 123, in dispatch_request\r\n await self.ds.refresh_schemas()\r\n File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py\", line 338, in refresh_schemas\r\n await init_internal_db(internal_db)\r\n File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/utils/internal_db.py\", line 16, in init_internal_db\r\n block=True,\r\n File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py\", line 102, in execute_write\r\n return await self.execute_write_fn(_inner, block=block)\r\n File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py\", line 118, in execute_write_fn\r\n raise result\r\n File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py\", line 139, in _execute_writes\r\n result = task.fn(conn)\r\n File \"/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py\", line 100, in _inner\r\n return conn.execute(sql, params or [])\r\nsqlite3.OperationalError: table databases already exists\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881204343", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881204343, "node_id": "IC_kwDOBm6k_c40hhx3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T06:13:11Z", "updated_at": "2021-07-16T06:13:11Z", "author_association": "OWNER", "body": "This just broke the `datasette-graphql` test suite: https://github.com/simonw/datasette-graphql/issues/77 - I need to figure out a solution here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881663968", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881663968, "node_id": "IC_kwDOBm6k_c40jR_g", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:18:42Z", "updated_at": "2021-07-16T19:18:42Z", "author_association": "OWNER", "body": "The race condition happens inside this method - initially with the call to `await init_internal_db()`: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/app.py#L334-L359", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881664408", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881664408, "node_id": "IC_kwDOBm6k_c40jSGY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:19:35Z", "updated_at": "2021-07-16T19:19:35Z", "author_association": "OWNER", "body": "The only place that calls `refresh_schemas()` is here: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/views/base.py#L120-L124\r\n\r\nIdeally only one call to `refresh_schemas()` would be running at any one time.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881665383", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881665383, "node_id": "IC_kwDOBm6k_c40jSVn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:21:35Z", "updated_at": "2021-07-16T19:21:35Z", "author_association": "OWNER", "body": "https://stackoverflow.com/a/25799871/6083 has a good example of using `asyncio.Lock()`:\r\n\r\n```python\r\nstuff_lock = asyncio.Lock()\r\n\r\nasync def get_stuff(url):\r\n async with stuff_lock:\r\n if url in cache:\r\n return cache[url]\r\n stuff = await aiohttp.request('GET', url)\r\n cache[url] = stuff\r\n return stuff\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881671706", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881671706, "node_id": "IC_kwDOBm6k_c40jT4a", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:32:05Z", "updated_at": "2021-07-16T19:32:05Z", "author_association": "OWNER", "body": "The test suite passes with that change.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881668759", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881668759, "node_id": "IC_kwDOBm6k_c40jTKX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:27:46Z", "updated_at": "2021-07-16T19:27:46Z", "author_association": "OWNER", "body": "Second attempt at this:\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 5976d8b..5f348cb 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -224,6 +224,7 @@ class Datasette:\r\n self.inspect_data = inspect_data\r\n self.immutables = set(immutables or [])\r\n self.databases = collections.OrderedDict()\r\n+ self._refresh_schemas_lock = asyncio.Lock()\r\n self.crossdb = crossdb\r\n if memory or crossdb or not self.files:\r\n self.add_database(Database(self, is_memory=True), name=\"_memory\")\r\n@@ -332,6 +333,12 @@ class Datasette:\r\n self.client = DatasetteClient(self)\r\n \r\n async def refresh_schemas(self):\r\n+ if self._refresh_schemas_lock.locked():\r\n+ return\r\n+ async with self._refresh_schemas_lock:\r\n+ await self._refresh_schemas()\r\n+\r\n+ async def _refresh_schemas(self):\r\n internal_db = self.databases[\"_internal\"]\r\n if not self.internal_db_created:\r\n await init_internal_db(internal_db)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881674857", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881674857, "node_id": "IC_kwDOBm6k_c40jUpp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:38:39Z", "updated_at": "2021-07-16T19:38:39Z", "author_association": "OWNER", "body": "I can't replicate the race condition locally with or without this patch. I'm going to push the commit and then test the CI run from `datasette-graphql` that was failing against it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1231#issuecomment-881677620", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1231", "id": 881677620, "node_id": "IC_kwDOBm6k_c40jVU0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T19:44:12Z", "updated_at": "2021-07-16T19:44:12Z", "author_association": "OWNER", "body": "That fixed the race condition in the `datasette-graphql` tests, which is the only place that I've been able to successfully replicate this. I'm going to land this change.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 811367257, "label": "Race condition errors in new refresh_schemas() mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1396#issuecomment-881686662", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1396", "id": 881686662, "node_id": "IC_kwDOBm6k_c40jXiG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-16T20:02:44Z", "updated_at": "2021-07-16T20:02:44Z", "author_association": "OWNER", "body": "Confirmed fixed: 0.58.1 was successfully published to Docker Hub in https://github.com/simonw/datasette/runs/3089447346 and the `latest` tag on https://hub.docker.com/r/datasetteproject/datasette/tags was updated.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944903881, "label": "\"invalid reference format\" publishing Docker image"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/123#issuecomment-882096402", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/123", "id": 882096402, "node_id": "IC_kwDOBm6k_c40k7kS", "user": {"value": 921217, "label": "RayBB"}, "created_at": "2021-07-18T18:07:29Z", "updated_at": "2021-07-18T18:07:29Z", "author_association": "NONE", "body": "I also love the idea for this feature and wonder if it could work without having to download the whole database into memory at once if it's a rather large db. Obviously this could be slower but could have many use cases.\r\n\r\nMy comment is partially inspired by this post about streaming sqlite dbs from github pages or such\r\nhttps://news.ycombinator.com/item?id=27016630\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 275125561, "label": "Datasette serve should accept paths/URLs to CSVs and other file formats"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1199#issuecomment-881932880", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1199", "id": 881932880, "node_id": "IC_kwDOBm6k_c40kTpQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-17T17:39:17Z", "updated_at": "2021-07-17T17:39:17Z", "author_association": "OWNER", "body": "I asked about optimizing performance on the SQLite forum and this came up as a suggestion: https://sqlite.org/forum/forumpost/9a6b9ae8e2048c8b?t=c\r\n\r\nI can start by trying this:\r\n\r\n PRAGMA mmap_size=268435456;", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 792652391, "label": "Experiment with PRAGMA mmap_size=N"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/123#issuecomment-882138084", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/123", "id": 882138084, "node_id": "IC_kwDOBm6k_c40lFvk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-19T00:04:31Z", "updated_at": "2021-07-19T00:04:31Z", "author_association": "OWNER", "body": "I've been thinking more about this one today too. An extension of this (touched on in #417, Datasette Library) would be to support pointing Datasette at a directory and having it automatically load any CSV files it finds anywhere in that folder or its descendants - either loading them fully, or providing a UI that allows users to select a file to open it in Datasette.\r\n\r\nFor larger files I think the right thing to do is import them into an on-disk SQLite database, which is limited only by available disk space. For smaller files loading them into an in-memory database should work fine.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 275125561, "label": "Datasette serve should accept paths/URLs to CSVs and other file formats"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1400#issuecomment-882542519", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1400", "id": 882542519, "node_id": "IC_kwDOBm6k_c40moe3", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-07-19T13:20:52Z", "updated_at": "2021-07-19T13:20:52Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#1400](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e95c685) into [main](https://codecov.io/gh/simonw/datasette/commit/c73af5dd72305f6a01ea94a2c76d52e5e26de38b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c73af5d) will **not change** coverage.\n> The diff coverage is `n/a`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1400/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #1400 +/- ##\n=======================================\n Coverage 91.62% 91.62% \n=======================================\n Files 34 34 \n Lines 4371 4371 \n=======================================\n Hits 4005 4005 \n Misses 366 366 \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c73af5d...e95c685](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 947640902, "label": "Bump black from 21.6b0 to 21.7b0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1401#issuecomment-884910320", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1401", "id": 884910320, "node_id": "IC_kwDOBm6k_c40vqjw", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-07-22T13:26:01Z", "updated_at": "2021-07-22T13:26:01Z", "author_association": "CONTRIBUTOR", "body": "ordered lists didn't work either, btw", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 950664971, "label": "unordered list is not rendering bullet points in description_html on database page"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1445#issuecomment-904024939", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1445", "id": 904024939, "node_id": "IC_kwDOBm6k_c414lNr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-23T18:52:35Z", "updated_at": "2021-08-23T18:52:35Z", "author_association": "OWNER", "body": "The downside of the current implementation of this trick is that it only works for exact LIKE partial matches in a specific table - if you search for `dog cat` and `dog` appears in `title` but `cat` appears in `description` you won't get back that result.\r\n\r\nI think that's fine though. If you want more advanced search there are other mechanisms you can use. This is meant to be a very quick and dirty starting point for exploring a brand new table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 977323133, "label": "Ability to search for text across all columns in a table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1445#issuecomment-904026253", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1445", "id": 904026253, "node_id": "IC_kwDOBm6k_c414liN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-23T18:54:49Z", "updated_at": "2021-08-23T18:54:49Z", "author_association": "OWNER", "body": "The bigger problem here is UI design. This feels like a pretty niche requirement to me, so adding a prominent search box to the table page (which already has the filters interface, plus the full-text search box for tables that have FTS configured) feels untidy.\r\n\r\nI could tuck it away in the table cog menu, but that's a weird place for something like this to live.\r\n\r\nMaybe add it as a new type of filter? Filters apply to specific columns though, so this would be the first filter that applied to _all_ columns - which doesn't really fit the existing filter interface very well.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 977323133, "label": "Ability to search for text across all columns in a table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1445#issuecomment-904027166", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1445", "id": 904027166, "node_id": "IC_kwDOBm6k_c414lwe", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-23T18:56:20Z", "updated_at": "2021-08-23T18:56:20Z", "author_association": "OWNER", "body": "A related but potentially even more useful ability would be running a search across every column of every table in a whole database. For anything less than a few 100MB this could be incredibly useful.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 977323133, "label": "Ability to search for text across all columns in a table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1445#issuecomment-904036200", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1445", "id": 904036200, "node_id": "IC_kwDOBm6k_c414n9o", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-23T19:08:54Z", "updated_at": "2021-08-23T19:08:54Z", "author_association": "OWNER", "body": "Figured out a query for searching across every column in every table! https://til.simonwillison.net/datasette/search-all-columns-trick#user-content-same-trick-for-the-entire-database\r\n\r\n```sql\r\nwith tables as (\r\n select\r\n name as table_name\r\n from\r\n sqlite_master\r\n where\r\n type = 'table'\r\n),\r\nqueries as (\r\n select\r\n 'select ''' || tables.table_name || ''' as _table, rowid from \"' || tables.table_name || '\" where ' || group_concat(\r\n '\"' || name || '\" like ''%'' || :search || ''%''',\r\n ' or '\r\n ) as query\r\n from\r\n pragma_table_info(tables.table_name),\r\n tables\r\n group by\r\n tables.table_name\r\n)\r\nselect\r\n group_concat(query, ' union all ')\r\nfrom\r\n queries\r\n```\r\nThe SQL query this generates for larger databases is _extremely_ long - but it does seem to work for smaller databases.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 977323133, "label": "Ability to search for text across all columns in a table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1445#issuecomment-904037087", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1445", "id": 904037087, "node_id": "IC_kwDOBm6k_c414oLf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-23T19:10:17Z", "updated_at": "2021-08-23T19:10:17Z", "author_association": "OWNER", "body": "Rather than trying to run that monstrosity in a single `union all` query, a better approach may be to use `fetch()` requests as seen in https://datasette.io/plugins/datasette-search-all", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 977323133, "label": "Ability to search for text across all columns in a table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1446#issuecomment-904866495", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1446", "id": 904866495, "node_id": "IC_kwDOBm6k_c417yq_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T18:13:49Z", "updated_at": "2021-08-24T18:13:49Z", "author_association": "OWNER", "body": "OK, now the following optional CSS gives us a sticky footer:\r\n```css\r\nhtml, body {\r\n height: 100%;\r\n}\r\nbody {\r\n display: flex;\r\n flex-direction: column;\r\n}\r\n.not-footer {\r\n flex: 1 0 auto;\r\n}\r\nfooter {\r\n flex-shrink: 0;\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978357984, "label": "Modify base.html template to support optional sticky footer"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1446#issuecomment-904954530", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1446", "id": 904954530, "node_id": "IC_kwDOBm6k_c418IKi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-24T20:32:47Z", "updated_at": "2021-08-24T20:32:47Z", "author_association": "OWNER", "body": "Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978357984, "label": "Modify base.html template to support optional sticky footer"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/859#issuecomment-904982056", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/859", "id": 904982056, "node_id": "IC_kwDOBm6k_c418O4o", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2021-08-24T21:15:04Z", "updated_at": "2021-08-24T21:15:30Z", "author_association": "CONTRIBUTOR", "body": "I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642572841, "label": "Database page loads too slowly with many large tables (due to table counts)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1447#issuecomment-905097468", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1447", "id": 905097468, "node_id": "IC_kwDOBm6k_c418rD8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-25T01:28:53Z", "updated_at": "2021-08-25T01:28:53Z", "author_association": "OWNER", "body": "Good catch, thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 978614898, "label": "Remove underscore from search mode parameter name"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1405#issuecomment-889525741", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1405", "id": 889525741, "node_id": "IC_kwDOBm6k_c41BRXt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-29T23:33:30Z", "updated_at": "2021-07-29T23:33:30Z", "author_association": "OWNER", "body": "New documentation section for `datasette.utils` is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 955316250, "label": "utils.parse_metadata() should be a documented internal function"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1241#issuecomment-889539227", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1241", "id": 889539227, "node_id": "IC_kwDOBm6k_c41BUqb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-30T00:15:26Z", "updated_at": "2021-07-30T00:15:26Z", "author_association": "OWNER", "body": "One possible treatment:\r\n\r\n\"fixtures__sortable__201_rows\"\r\n\r\n```html\r\n\r\n

\r\n {% if query.sql and allow_execute_sql %}\r\n \r\n View and edit SQL\r\n \r\n {% endif %}\r\n Copy and share link\r\n

\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 814595021, "label": "Share button for copying current URL"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1406#issuecomment-889548536", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1406", "id": 889548536, "node_id": "IC_kwDOBm6k_c41BW74", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-30T00:43:47Z", "updated_at": "2021-07-30T00:43:47Z", "author_association": "OWNER", "body": "Still couldn't replicate on my laptop. On a hunch, I'm going to add `@pytest.mark.serial` to every test that uses `runner.isolated_filesystem()`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 956303470, "label": "Tests failing with FileNotFoundError in runner.isolated_filesystem"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1406#issuecomment-889547142", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1406", "id": 889547142, "node_id": "IC_kwDOBm6k_c41BWmG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-30T00:39:49Z", "updated_at": "2021-07-30T00:39:49Z", "author_association": "OWNER", "body": "It happens in CI but not on my laptop. I think I need to run the tests on my laptop like this: https://github.com/simonw/datasette/blob/121e10c29c5b412fddf0326939f1fe46c3ad9d4a/.github/workflows/test.yml#L27-L30", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 956303470, "label": "Tests failing with FileNotFoundError in runner.isolated_filesystem"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1406#issuecomment-889550391", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1406", "id": 889550391, "node_id": "IC_kwDOBm6k_c41BXY3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-30T00:49:31Z", "updated_at": "2021-07-30T00:49:31Z", "author_association": "OWNER", "body": "That fixed it. My hunch is that Click's `runner.isolated_filesystem()` mechanism doesn't play well with `pytest-xdist`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 956303470, "label": "Tests failing with FileNotFoundError in runner.isolated_filesystem"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1406#issuecomment-889553052", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1406", "id": 889553052, "node_id": "IC_kwDOBm6k_c41BYCc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-30T00:58:43Z", "updated_at": "2021-07-30T00:58:43Z", "author_association": "OWNER", "body": "Tests are still failing in the job that calculates coverage.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 956303470, "label": "Tests failing with FileNotFoundError in runner.isolated_filesystem"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1406#issuecomment-889555977", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1406", "id": 889555977, "node_id": "IC_kwDOBm6k_c41BYwJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-30T01:06:57Z", "updated_at": "2021-07-30T01:06:57Z", "author_association": "OWNER", "body": "Looking at the source code in Click for `isolated_filesystem()`: https://github.com/pallets/click/blob/9da166957f5848b641231d485467f6140bca2bc0/src/click/testing.py#L450-L468\r\n\r\n```python\r\n @contextlib.contextmanager\r\n def isolated_filesystem(\r\n self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None\r\n ) -> t.Iterator[str]:\r\n \"\"\"A context manager that creates a temporary directory and\r\n changes the current working directory to it. This isolates tests\r\n that affect the contents of the CWD to prevent them from\r\n interfering with each other.\r\n :param temp_dir: Create the temporary directory under this\r\n directory. If given, the created directory is not removed\r\n when exiting.\r\n .. versionchanged:: 8.0\r\n Added the ``temp_dir`` parameter.\r\n \"\"\"\r\n cwd = os.getcwd()\r\n t = tempfile.mkdtemp(dir=temp_dir)\r\n os.chdir(t)\r\n```\r\nHow about if I pass in that optional `temp_dir` as a temp directory created using the `pytest-xdist` aware pytest mechanisms: https://docs.pytest.org/en/6.2.x/tmpdir.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 956303470, "label": "Tests failing with FileNotFoundError in runner.isolated_filesystem"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1398#issuecomment-889599513", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1398", "id": 889599513, "node_id": "IC_kwDOBm6k_c41BjYZ", "user": {"value": 192984, "label": "aitoehigie"}, "created_at": "2021-07-30T03:21:49Z", "updated_at": "2021-07-30T03:21:49Z", "author_association": "NONE", "body": "Does the library support this now?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 947044667, "label": "Documentation on using Datasette as a library"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1406#issuecomment-890259755", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1406", "id": 890259755, "node_id": "IC_kwDOBm6k_c41EEkr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T00:04:54Z", "updated_at": "2021-07-31T00:04:54Z", "author_association": "OWNER", "body": "STILL failing. I'm going to try removing all instances of `isolated_filesystem()` in favour of a different pattern using pytest temporary files, then see if I can get that to work without the serial hack. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 956303470, "label": "Tests failing with FileNotFoundError in runner.isolated_filesystem"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1407#issuecomment-890388200", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1407", "id": 890388200, "node_id": "IC_kwDOBm6k_c41Ej7o", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T18:38:41Z", "updated_at": "2021-07-31T18:38:41Z", "author_association": "OWNER", "body": "The `path` variable there looked like this:\r\n\r\n`/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-696/popen-gw0/uds0/datasette.sock`\r\n\r\nI think what's happening here is that `pytest-xdist` causes `tmp_path_factory.mktemp(\"uds\")` to create significantly longer paths, which in this case is breaking some limit.\r\n\r\nSo for this code to work with `pytest-xdist` I need to make sure the random path to `datasette.sock` is shorter.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957298475, "label": "OSError: AF_UNIX path too long in ds_unix_domain_socket_server"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1407#issuecomment-890388656", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1407", "id": 890388656, "node_id": "IC_kwDOBm6k_c41EkCw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T18:42:41Z", "updated_at": "2021-07-31T18:42:41Z", "author_association": "OWNER", "body": "I'll try `tempfile.gettempdir()` - on macOS it returns something like `'/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T'` which is still long but hopefully not too long.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957298475, "label": "OSError: AF_UNIX path too long in ds_unix_domain_socket_server"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1406#issuecomment-890390198", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1406", "id": 890390198, "node_id": "IC_kwDOBm6k_c41Eka2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T18:55:33Z", "updated_at": "2021-07-31T18:55:33Z", "author_association": "OWNER", "body": "To clarify: the core problem here is that an error is thrown any time you call `os.getcwd()` but the directory you are currently in has been deleted.\r\n\r\n`runner.isolated_filesystem()` assumes that the current directory in has not been deleted. But the various temporary directory utilities in `pytest` work by creating directories and then deleting them.\r\n\r\nMaybe there's a larger problem here that I play a bit fast and loose with `os.chdir()` in both the test suite and in various lines of code in Datasette itself (in particular in the publish commands)?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 956303470, "label": "Tests failing with FileNotFoundError in runner.isolated_filesystem"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1406#issuecomment-890390342", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1406", "id": 890390342, "node_id": "IC_kwDOBm6k_c41EkdG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T18:56:35Z", "updated_at": "2021-07-31T18:56:35Z", "author_association": "OWNER", "body": "But... I've lost enough time to this already, and removing `runner.isolated_filesystem()` has the tests passing again. So I'm not going to work on this any more.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 956303470, "label": "Tests failing with FileNotFoundError in runner.isolated_filesystem"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1408#issuecomment-890390495", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1408", "id": 890390495, "node_id": "IC_kwDOBm6k_c41Ekff", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T18:57:39Z", "updated_at": "2021-07-31T18:57:39Z", "author_association": "OWNER", "body": "Opening this issue as an optional follow-up to the work I did in #1406.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957302085, "label": "Review places in codebase that use os.chdir(), in particularly relating to tests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1408#issuecomment-890390845", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1408", "id": 890390845, "node_id": "IC_kwDOBm6k_c41Ekk9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T19:00:32Z", "updated_at": "2021-07-31T19:00:32Z", "author_association": "OWNER", "body": "When I revisit this I can also look at dropping the `@pytest.mark.serial` hack, and maybe the `restore_working_directory()` fixture hack too:\r\n\r\nhttps://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/pytest.ini#L9-L10\r\n\r\nhttps://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/tests/conftest.py#L62-L75", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957302085, "label": "Review places in codebase that use os.chdir(), in particularly relating to tests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1409#issuecomment-890400059", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1409", "id": 890400059, "node_id": "IC_kwDOBm6k_c41Em07", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T20:21:51Z", "updated_at": "2021-07-31T20:21:51Z", "author_association": "OWNER", "body": "One of these two options:\r\n\r\n- `--setting default_allow_sql off`\r\n- `--setting allow_sql_default off`\r\n\r\nExisting settings from https://docs.datasette.io/en/0.58.1/settings.html with similar names that I need to be consistent with:\r\n\r\n- `default_page_size`\r\n- `allow_facet`\r\n- `default_facet_size`\r\n- `allow_download`\r\n- `default_cache_ttl`\r\n- `default_cache_ttl_hashed`\r\n- `allow_csv_stream`\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957310278, "label": "`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1409#issuecomment-890400121", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1409", "id": 890400121, "node_id": "IC_kwDOBm6k_c41Em15", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T20:22:21Z", "updated_at": "2021-07-31T20:23:34Z", "author_association": "OWNER", "body": "I think `default_allow_sql` is more consistent with the current naming conventions, because both `allow` and `default` are used as prefixes at the moment but neither of them are ever used as a suffix.\r\n\r\nPlus `default_allow_sql off` makes sense to me but `allow_default_sql off` does not - what is \"default SQL\"?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957310278, "label": "`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1409#issuecomment-890400425", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1409", "id": 890400425, "node_id": "IC_kwDOBm6k_c41Em6p", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T20:25:16Z", "updated_at": "2021-07-31T20:26:25Z", "author_association": "OWNER", "body": "If I was prone to over-thinking (which I am) I'd note that `allow_facet` and `allow_download` and `allow_csv_stream` are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent.\r\n\r\nSo maybe there's a future task where I introduce those as both permissions and metadata `\"allow_x\"` blocks, then rename the settings themselves to be called `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`.\r\n\r\nIf I was going to do that I should get it in before Datasette 1.0.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957310278, "label": "`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1409#issuecomment-890397124", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1409", "id": 890397124, "node_id": "IC_kwDOBm6k_c41EmHE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T19:51:10Z", "updated_at": "2021-07-31T19:51:10Z", "author_association": "OWNER", "body": "I think I may like `disable_sql` better. Some options:\r\n\r\n- `--setting allow_sql off` (consistent with `allow_facet` and `allow_download` and `allow_csv_stream` - all which default to `on` already)\r\n- `--setting disable_sql on`\r\n- `--setting disable_custom_sql on`\r\n\r\nThe existence of three `allow_*` settings does make a strong argument for staying consistent with that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957310278, "label": "`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1409#issuecomment-890397169", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1409", "id": 890397169, "node_id": "IC_kwDOBm6k_c41EmHx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T19:51:35Z", "updated_at": "2021-07-31T19:51:35Z", "author_association": "OWNER", "body": "I'm going to stick with `--setting allow_sql off`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957310278, "label": "`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1409#issuecomment-890397261", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1409", "id": 890397261, "node_id": "IC_kwDOBm6k_c41EmJN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T19:52:25Z", "updated_at": "2021-07-31T19:52:25Z", "author_association": "OWNER", "body": "I think I can make this modification by teaching the default permissions code here to take the `allow_sql` setting into account: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/datasette/default_permissions.py#L38-L45", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957310278, "label": "`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1409#issuecomment-890397652", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1409", "id": 890397652, "node_id": "IC_kwDOBm6k_c41EmPU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T19:56:48Z", "updated_at": "2021-07-31T19:56:48Z", "author_association": "OWNER", "body": "The other option would be to use the setting to pick the `default=` argument when calling `self.ds.permission_allowed( request.actor, \"execute-sql\", resource=database, default=True)`.\r\n\r\nThe problem with that is that there are actually a few different places which perform that check, so changing all of them raises the risk of missing one in the future:\r\n\r\nhttps://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L436-L444\r\n\r\nhttps://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L964-L966\r\n\r\nhttps://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L220-L221\r\n\r\nhttps://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L343-L345\r\n\r\nhttps://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L134-L136\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957310278, "label": "`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1409#issuecomment-890397753", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1409", "id": 890397753, "node_id": "IC_kwDOBm6k_c41EmQ5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T19:57:56Z", "updated_at": "2021-07-31T19:57:56Z", "author_association": "OWNER", "body": "I think the correct solution is for the default permissions logic to take the `allow_sql` setting into account, and to return `False` if that setting is set to `off` AND the current actor fails the `actor_matches_allow` checks.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957310278, "label": "`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1409#issuecomment-890399806", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1409", "id": 890399806, "node_id": "IC_kwDOBm6k_c41Emw-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-07-31T20:18:46Z", "updated_at": "2021-07-31T20:18:46Z", "author_association": "OWNER", "body": "My rationale for removing it: https://github.com/simonw/datasette/issues/813#issuecomment-640916290\r\n\r\n> Naming problem: Datasette already has a config option with this name:\r\n>\r\n> $ datasette serve data.db --config allow_sql:1\r\n>\r\n> https://datasette.readthedocs.io/en/stable/config.html#allow-sql\r\n>\r\n> It's confusing to have two things called `allow_sql` that do slightly different things.\r\n>\r\n> I could retire the `--config allow_sql:0` option entirely, since the new `metadata.json` mechanism can be used to achieve the exact same thing.\r\n> \r\n> I'm going to do that.\r\n\r\nThis is true. The `\"allow_sql\"` permissions block in `metadata.json` does indeed have a name that is easily confused with `--setting allow_sql off`.\r\n\r\nSo I definitely need to pick a different name from the setting. `--setting default_allow_sql off` is a good option here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957310278, "label": "`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1411#issuecomment-890441844", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1411", "id": 890441844, "node_id": "IC_kwDOBm6k_c41ExB0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-01T03:27:30Z", "updated_at": "2021-08-01T03:27:30Z", "author_association": "OWNER", "body": "Confirmed: https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1 no longer exhibits the bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 957345476, "label": "Canned query ?sql= is pointlessly echoed in query string starting from hidden mode"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1227#issuecomment-891352132", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1227", "id": 891352132, "node_id": "IC_kwDOBm6k_c41IPRE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-02T21:38:39Z", "updated_at": "2021-08-02T21:38:39Z", "author_association": "OWNER", "body": "Relevant TIL: https://til.simonwillison.net/vscode/vs-code-regular-expressions", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810394616, "label": "Configure sphinx.ext.extlinks for issues"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1417#issuecomment-891979858", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1417", "id": 891979858, "node_id": "IC_kwDOBm6k_c41KohS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-03T16:15:30Z", "updated_at": "2021-08-03T16:15:30Z", "author_association": "OWNER", "body": "Docs: https://pypi.org/project/codespell/\r\n\r\nThere's a `codespell --ignore-words=FILE` option for ignoring words. I don't have any that need ignoring yet but I'm going to add that file anyway, that way I can have codespell be a failing test but still provide a way to work around it if it incorrectly flags a correct word.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959278472, "label": "Use codespell in CI to spot spelling errors"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1418#issuecomment-891984359", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1418", "id": 891984359, "node_id": "IC_kwDOBm6k_c41Kpnn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-03T16:21:56Z", "updated_at": "2021-08-03T16:21:56Z", "author_association": "OWNER", "body": "Failed with:\r\n```\r\ndocs/authentication.rst:63: perfom ==> perform\r\ndocs/authentication.rst:76: perfom ==> perform\r\ndocs/changelog.rst:429: repsonse ==> response\r\ndocs/changelog.rst:503: permissons ==> permissions\r\ndocs/changelog.rst:717: compatibilty ==> compatibility\r\ndocs/changelog.rst:1172: browseable ==> browsable\r\ndocs/deploying.rst:191: similiar ==> similar\r\ndocs/internals.rst:434: Respons ==> Response, respond\r\ndocs/internals.rst:440: Respons ==> Response, respond\r\ndocs/internals.rst:717: tha ==> than, that, the\r\ndocs/performance.rst:42: databse ==> database\r\ndocs/plugin_hooks.rst:667: utilites ==> utilities\r\ndocs/publish.rst:168: countainer ==> container\r\ndocs/settings.rst:352: inalid ==> invalid\r\ndocs/sql_queries.rst:406: preceeded ==> preceded, proceeded\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959284434, "label": "Spelling corrections plus CI job for codespell"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1418#issuecomment-891987129", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1418", "id": 891987129, "node_id": "IC_kwDOBm6k_c41KqS5", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-08-03T16:26:01Z", "updated_at": "2021-08-03T16:32:20Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#1418](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (532170f) into [main](https://codecov.io/gh/simonw/datasette/commit/54b6e96ee8aa553b6671e341a1944f93f3fb89c3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (54b6e96) will **not change** coverage.\n> The diff coverage is `n/a`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1418/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #1418 +/- ##\n=======================================\n Coverage 91.64% 91.64% \n=======================================\n Files 34 34 \n Lines 4382 4382 \n=======================================\n Hits 4016 4016 \n Misses 366 366 \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [54b6e96...532170f](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959284434, "label": "Spelling corrections plus CI job for codespell"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1419#issuecomment-892276385", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1419", "id": 892276385, "node_id": "IC_kwDOBm6k_c41Lw6h", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-08-04T00:58:49Z", "updated_at": "2021-08-04T00:58:49Z", "author_association": "CONTRIBUTOR", "body": "yes, [filter clause on aggregate queries were added to sqlite3 in 3.30](https://www.sqlite.org/releaselog/3_30_1.html)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959710008, "label": "`publish cloudrun` should deploy a more recent SQLite version"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1420#issuecomment-892365639", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1420", "id": 892365639, "node_id": "IC_kwDOBm6k_c41MGtH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-04T05:05:07Z", "updated_at": "2021-08-04T05:05:07Z", "author_association": "OWNER", "body": "https://github.com/simonw/datasette/blob/cd8b7bee8fb5c1cdce7c8dbfeb0166011abc72c6/datasette/publish/cloudrun.py#L153-L158", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959898166, "label": "`datasette publish cloudrun --cpu X` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1420#issuecomment-892372509", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1420", "id": 892372509, "node_id": "IC_kwDOBm6k_c41MIYd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-04T05:22:29Z", "updated_at": "2021-08-04T05:22:29Z", "author_association": "OWNER", "body": "Testing this manually with:\r\n\r\n datasette publish cloudrun fixtures.db --memory 8G --cpu 4 \\\r\n --service fixtures-over-provisioned-issue-1420 --install datasette-psutil\r\n\r\nAnd for comparison:\r\n\r\n datasette publish cloudrun fixtures.db --service fixtures-default-issue-1420 \\\r\n --install datasette-psutil\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959898166, "label": "`datasette publish cloudrun --cpu X` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1420#issuecomment-892374253", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1420", "id": 892374253, "node_id": "IC_kwDOBm6k_c41MIzt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-04T05:27:21Z", "updated_at": "2021-08-04T05:29:59Z", "author_association": "OWNER", "body": "I'll delete these services shortly afterwards. Right now:\r\n\r\nhttps://fixtures-over-provisioned-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil (deployed with `--memory 8G --cpu 4`) returns:\r\n```\r\nprocess.memory_info()\r\n pmem(rss=60456960, vms=518930432, shared=0, text=0, lib=0, data=0, dirty=0)\r\n\r\n...\r\n\r\npsutil.cpu_times(True)\r\n scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)\r\n scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)\r\n scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)\r\n scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)\r\n\r\npsutil.virtual_memory()\r\n svmem(total=2147483648, available=2092531712, percent=2.6, used=33103872, free=2092531712, active=44130304, inactive=10792960, buffers=0, cached=21848064, shared=262144, slab=0)\r\n```\r\nhttps://fixtures-default-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil returns:\r\n```\r\nprocess.memory_info()\r\n pmem(rss=49324032, vms=140595200, shared=0, text=0, lib=0, data=0, dirty=0)\r\n\r\n...\r\n\r\npsutil.cpu_times(True)\r\n scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)\r\n scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)\r\n\r\npsutil.virtual_memory()\r\n svmem(total=2147483648, available=2091188224, percent=2.6, used=40071168, free=2091188224, active=41586688, inactive=7983104, buffers=0, cached=16224256, shared=262144, slab=0)\r\n```\r\nThese numbers are different enough that I assume this works as advertised. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959898166, "label": "`datasette publish cloudrun --cpu X` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1420#issuecomment-892376353", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1420", "id": 892376353, "node_id": "IC_kwDOBm6k_c41MJUh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-04T05:33:12Z", "updated_at": "2021-08-04T05:33:12Z", "author_association": "OWNER", "body": "In the Cloud Run console (before I deleted these services) when I click \"Edit and deploy new revision\" I see this for the default one:\r\n\r\n\"Deploy_new_revision_\u2013_Cloud_Run_\u2013_datasette_\u2013_Google_Cloud_Platform\"\r\n\r\nAnd this for the big one:\r\n\r\n\"Deploy_new_revision_\u2013_Cloud_Run_\u2013_datasette_\u2013_Google_Cloud_Platform\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959898166, "label": "`datasette publish cloudrun --cpu X` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1420#issuecomment-893079520", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1420", "id": 893079520, "node_id": "IC_kwDOBm6k_c41O0_g", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-05T00:54:59Z", "updated_at": "2021-08-05T00:54:59Z", "author_association": "OWNER", "body": "Just saw this error:\r\n\r\n`ERROR: (gcloud.run.deploy) The `--cpu` flag is not supported on the fully managed version of Cloud Run. Specify `--platform gke` or run `gcloud config set run/platform gke` to work with Cloud Run for Anthos deployed on Google Cloud.`\r\n\r\nWhich is weird because I managed to run this successfully the other day. Maybe a region difference thing?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959898166, "label": "`datasette publish cloudrun --cpu X` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1419#issuecomment-893114612", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1419", "id": 893114612, "node_id": "IC_kwDOBm6k_c41O9j0", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-08-05T02:29:06Z", "updated_at": "2021-08-05T02:29:06Z", "author_association": "CONTRIBUTOR", "body": "there's a lot of complexity here, that's probably not worth addressing. i got what i needed by patching the dockerfile that cloudrun uses to install a newer version of sqlite.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959710008, "label": "`publish cloudrun` should deploy a more recent SQLite version"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1422#issuecomment-893122356", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1422", "id": 893122356, "node_id": "IC_kwDOBm6k_c41O_c0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-05T02:52:31Z", "updated_at": "2021-08-05T02:52:44Z", "author_association": "OWNER", "body": "If you do this it should still be possible to view the SQL - which means we need a new parameter. I propose `?_show_sql=1` to over-ride the hidden default.\r\n\r\nI think the configuration should use `hide_sql: true` - looking like this:\r\n```yaml\r\ndatabases:\r\n fixtures:\r\n queries:\r\n neighborhood_search:\r\n hide_sql: true\r\n sql: |-\r\n select neighborhood, facet_cities.name, state\r\n from facetable\r\n join facet_cities on facetable.city_id = facet_cities.id\r\n where neighborhood like '%' || :text || '%'\r\n order by neighborhood\r\n title: Search neighborhoods\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 961367843, "label": "Ability to default to hiding the SQL for a canned query"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1422#issuecomment-893131703", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1422", "id": 893131703, "node_id": "IC_kwDOBm6k_c41PBu3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-05T03:16:46Z", "updated_at": "2021-08-05T03:16:46Z", "author_association": "OWNER", "body": "The logic for this is a little bit fiddly, due to the need to switch to using `?_show_sql=1` on the link depending on the context.\r\n\r\n- If metadata says hide and there's no query string, hide and link to `?_show_sql=1`\r\n- If metadata says hide but query string says `?_show_sql=1`, show and have hide link linking to URL without `?_show_sql=1`\r\n- Otherwise, show and link to `?_hide_sql=1`\r\n- ... or if that query string is there then hide and link to URL without `?_hide_sql=1`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 961367843, "label": "Ability to default to hiding the SQL for a canned query"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1419#issuecomment-893133496", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1419", "id": 893133496, "node_id": "IC_kwDOBm6k_c41PCK4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-05T03:22:44Z", "updated_at": "2021-08-05T03:22:44Z", "author_association": "OWNER", "body": "I ran into this exact same problem today! I only just learned how to use filter on aggregates: https://til.simonwillison.net/sqlite/sqlite-aggregate-filter-clauses\r\n\r\nA workaround I used is to add this to the deploy command:\r\n\r\n datasette publish cloudrun ... --install=pysqlite3-binary\r\n\r\nThis will install the https://pypi.org/project/pysqlite3-binary for package which bundles a more recent SQLite version.", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 959710008, "label": "`publish cloudrun` should deploy a more recent SQLite version"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1423#issuecomment-893996604", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1423", "id": 893996604, "node_id": "IC_kwDOBm6k_c41SU48", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-06T04:43:07Z", "updated_at": "2021-08-06T04:43:37Z", "author_association": "OWNER", "body": "Problem: on a page which doesn't have quite enough facet values to trigger the display of the \"...\" link that links to `?_facet_size=max` the user would still have to manually count the values - up to 30 by default.\r\n\r\nSo maybe the count should always be shown, perhaps as a non-bold light colored number?\r\n\r\nI could even hide it in a non-discoverable tooltip.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 962391325, "label": "Show count of facet values if ?_facet_size=max"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1423#issuecomment-894452990", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1423", "id": 894452990, "node_id": "IC_kwDOBm6k_c41UET-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-06T18:49:37Z", "updated_at": "2021-08-06T18:49:37Z", "author_association": "OWNER", "body": "Could display them always, like this:\r\n\r\n\"fixtures__compound_three_primary_keys__1_001_rows_and_School_enrollment_analysis_Colorado_-_Google_Docs\"\r\n\r\nThat's with `23`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 962391325, "label": "Show count of facet values if ?_facet_size=max"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1423#issuecomment-894453520", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1423", "id": 894453520, "node_id": "IC_kwDOBm6k_c41UEcQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-06T18:50:40Z", "updated_at": "2021-08-06T18:50:40Z", "author_association": "OWNER", "body": "Point of confusion: if only 30 options are shown, but there's a `...` at the end, what would the number be? It can't be the total number of facets because we haven't counted them all - but if it's just the number of displayed facets that's like to be confusing.\r\n\r\nSo the original idea of showing the counts only if `_facet_size=max` is a good one.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 962391325, "label": "Show count of facet values if ?_facet_size=max"}, "performed_via_github_app": null}