issue_comments

Table actions

7,983 rows sorted by node_id

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: author_association, reactions

id html_url issue_url node_id ▼ user created_at updated_at author_association body reactions issue performed_via_github_app
1053973425 https://github.com/simonw/datasette/issues/1439#issuecomment-1053973425 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-0lux simonw 9599 2022-02-28T07:40:12Z 2022-02-28T07:40:12Z OWNER If I make this change it will break existing links to one of the oldest Datasette demos: http://fivethirtyeight.datasettes.com/fivethirtyeight/avengers%2Favengers A plugin that fixes those by redirecting them on 404 would be neat. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1043609198 https://github.com/simonw/datasette/issues/327#issuecomment-1043609198 https://api.github.com/repos/simonw/datasette/issues/327 IC_kwDOBm6k_c4-NDZu dholth 208018 2022-02-17T23:21:36Z 2022-02-17T23:33:01Z NONE On fly.io. This particular database goes from 1.4GB to 200M. Slower, part of that might be having no `--inspect-file`? ``` $ datasette publish fly ... --generate-dir /tmp/deploy-this ... $ mksquashfs large.db large.squashfs $ rm large.db # don't accidentally put it in the image $ cat Dockerfile FROM python:3.8 COPY . /app WORKDIR /app ENV DATASETTE_SECRET 'xyzzy' RUN pip install -U datasette # RUN datasette inspect large.db --inspect-file inspect-data.json ENV PORT 8080 EXPOSE 8080 CMD mount -o loop -t squashfs large.squashfs /mnt; datasette serve --host 0.0.0.0 -i /mnt/large.db --cors --port $PORT ``` It would also be possible to copy the file onto the ~6GB available on the ephemeral container filesystem on startup. A little against the spirit of the thing? On this example the whole docker image is 2.42 GB and the squashfs version is 1.14 GB. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Explore if SquashFS can be used to shrink size of packaged Docker containers 335200136  
1043626870 https://github.com/simonw/datasette/issues/327#issuecomment-1043626870 https://api.github.com/repos/simonw/datasette/issues/327 IC_kwDOBm6k_c4-NHt2 dholth 208018 2022-02-17T23:37:24Z 2022-02-17T23:37:24Z NONE On second thought any kind of quick-to-decompress-on-startup could be helpful if we're paying for the container registry and deployment bandwidth but not ephemeral storage. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Explore if SquashFS can be used to shrink size of packaged Docker containers 335200136  
1045131086 https://github.com/simonw/datasette/issues/1439#issuecomment-1045131086 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-S29O simonw 9599 2022-02-18T20:22:13Z 2022-02-18T20:22:47Z OWNER Should it encode `%` symbols too, since they have a special meaning in URLs and we can't guarantee that every single web server / proxy out there will round-trip them safely using percentage encoding? If so, would need to pick a different encoding character for them. Maybe `%` becomes `-p` - and in that case `/` could become `-s` too. Is it worth expanding dash-encoding outside of just `/` and `-` and `.` though? Not sure. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045134050 https://github.com/simonw/datasette/issues/1439#issuecomment-1045134050 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-S3ri simonw 9599 2022-02-18T20:25:04Z 2022-02-18T20:25:04Z OWNER Here's a useful modern spec for how existing URL percentage encoding is supposed to work: https://url.spec.whatwg.org/#percent-encoded-bytes {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045024276 https://github.com/simonw/datasette/issues/1439#issuecomment-1045024276 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-Sc4U simonw 9599 2022-02-18T19:01:42Z 2022-02-18T19:55:24Z OWNER > Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly. ```python def dash_encode(s): return s.replace("-", "--").replace(".", "-.").replace("/", "-/") def dash_decode(s): return s.replace("-/", "/").replace("-.", ".").replace("--", "-") ``` ```pycon >>> dash_encode("foo/bar/baz.csv") 'foo-/bar-/baz-.csv' >>> dash_decode('foo-/bar-/baz-.csv') 'foo/bar/baz.csv' ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045027067 https://github.com/simonw/datasette/issues/1439#issuecomment-1045027067 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-Sdj7 simonw 9599 2022-02-18T19:03:26Z 2022-02-18T19:03:26Z OWNER (If I make this change it may break some existing Datasette installations when they upgrade - I could try and build a plugin for them which triggers on 404s and checks to see if the old format would return a 200 response, then returns that.) {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045032377 https://github.com/simonw/datasette/issues/1439#issuecomment-1045032377 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-Se25 simonw 9599 2022-02-18T19:06:50Z 2022-02-18T19:06:50Z OWNER How does URL routing for https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv work? Right now it's https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/app.py#L1098-L1101 That's not going to capture the dot-dash encoding version of that table name: ```pycon >>> dot_dash_encode("table/with/slashes.csv") 'table-/with-/slashes-.csv' ``` Probably needs a fancy regex trick like a negative lookbehind assertion or similar. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045055772 https://github.com/simonw/datasette/issues/1439#issuecomment-1045055772 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-Skkc simonw 9599 2022-02-18T19:23:33Z 2022-02-18T19:25:42Z OWNER I want a match for this URL: /db/table-/with-/slashes-.csv Maybe this: ^/(?P<db_name>[^/]+)/(?P<table_and_format>([^/]*|(\-/)*|(\-\.)*|(\.\.)*)*$) Here we are matching a sequence of: ([^/]*|(\-/)*|(\-\.)*|(\-\-)*)* So a combination of not-slashes OR -/ or -. Or -- sequences <img width="224" alt="image" src="https://user-images.githubusercontent.com/9599/154748362-84909d4e-dccf-454b-a9cd-a036f9f66f09.png"> ^/(?P<db_name>[^/]+)/(?P<table_and_format>([^/]*|(\-/)*|(\-\.)*|(\-\-)*)*$) Try that with non-capturing bits: ^/(?P<db_name>[^/]+)/(?P<table_and_format>(?:[^/]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*$) `(?:[^/]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*` visualized is: <img width="193" alt="image" src="https://user-images.githubusercontent.com/9599/154748441-decea502-0d04-44f4-9ca9-fb6883767833.png"> Here's the explanation on regex101.com https://regex101.com/r/CPnsIO/1 <img width="1074" alt="image" src="https://user-images.githubusercontent.com/9599/154748720-cdda61db-5498-49a8-91c2-e726b394fa49.png"> {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045059427 https://github.com/simonw/datasette/issues/1439#issuecomment-1045059427 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-Sldj simonw 9599 2022-02-18T19:26:25Z 2022-02-18T19:26:25Z OWNER With this new pattern I could probably extract out the optional `.json` format string as part of the initial route capturing regex too, rather than the current `table_and_format` hack. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045069481 https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-Sn6p simonw 9599 2022-02-18T19:34:41Z 2022-03-05T21:32:22Z OWNER I think I got format extraction working! https://regex101.com/r/A0bW1D/1 ^/(?P<database>[^/]+)/(?P<table>(?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*?)(?:(?<!\-)\.(?P<format>\w+))?$ I had to make that crazy inner one even more complicated to stop it from capturing `.` that was not part of `-.`. (?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)* Visualized: <img width="222" alt="image" src="https://user-images.githubusercontent.com/9599/154749714-44579899-5dc7-4e5f-ad4f-dc59dac48979.png"> So now I have a regex which can extract out the dot-encoded table name AND spot if there is an optional `.format` at the end: <img width="1090" alt="image" src="https://user-images.githubusercontent.com/9599/156900484-7912073f-28aa-4301-86e2-e5cbe625e1d5.png"> If I end up using this in Datasette it's going to need VERY comprehensive unit tests and inline documentation. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045077590 https://github.com/simonw/datasette/issues/1439#issuecomment-1045077590 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-Sp5W simonw 9599 2022-02-18T19:41:37Z 2022-02-18T19:42:41Z OWNER Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where "system" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence. And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too. Maybe change this system to use `.` as the escaping character instead of `-`? {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045075207 https://github.com/simonw/datasette/issues/1439#issuecomment-1045075207 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-SpUH simonw 9599 2022-02-18T19:39:35Z 2022-02-18T19:40:13Z OWNER > And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this: > > * `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version > * `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version > * `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version Here's what those look like with the updated version of `dot_dash_encode()` that also encodes `/` as `-/`: - `/db/-/db-/table---.csv-.csv` - HTML - `/db/-/db-/table---.csv-.csv.csv` - CSV - `/db/-/db-/table---.csv-.csv.json` - JSON <img width="1050" alt="image" src="https://user-images.githubusercontent.com/9599/154750631-a8a23c62-3dfc-43e4-8026-4d117dc4bf8d.png"> {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045081042 https://github.com/simonw/datasette/issues/1439#issuecomment-1045081042 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-SqvS simonw 9599 2022-02-18T19:44:12Z 2022-02-18T19:51:34Z OWNER ```python def dot_encode(s): return s.replace(".", "..").replace("/", "./") def dot_decode(s): return s.replace("./", "/").replace("..", ".") ``` No need for hyphen encoding in this variant at all, which simplifies things a bit. (Update: this is flawed, see https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033) {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045086033 https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-Sr9R simonw 9599 2022-02-18T19:47:43Z 2022-02-18T19:51:11Z OWNER - https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv - https://til.simonwillison.net/-/asgi-scope/db/./db./table-..csv..csv Do both of those survive the round-trip to populate `raw_path` correctly? No! In both cases the `/./` bit goes missing. It looks like this might even be a client issue - `curl` shows me this: ``` ~ % curl -vv -i 'https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv' * Trying 216.239.32.21:443... * Connected to datasette.io (216.239.32.21) port 443 (#0) * ALPN, offering http/1.1 * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: datasette.io * Server certificate: R3 * Server certificate: ISRG Root X1 > GET /-/asgi-scope/db/db./table-..csv..csv HTTP/1.1 ``` So `curl` decided to turn `/-/asgi-scope/db/./db./table` into `/-/asgi-scope/db/db./table` before even sending the request. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045082891 https://github.com/simonw/datasette/issues/1439#issuecomment-1045082891 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-SrML simonw 9599 2022-02-18T19:45:32Z 2022-02-18T19:45:32Z OWNER ```pycon >>> dot_encode("/db/table-.csv.csv") './db./table-..csv..csv' >>> dot_decode('./db./table-..csv..csv') '/db/table-.csv.csv' ``` I worry that web servers might treat `./` in a special way though. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045095348 https://github.com/simonw/datasette/issues/1439#issuecomment-1045095348 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-SuO0 simonw 9599 2022-02-18T19:53:48Z 2022-02-18T19:53:48Z OWNER > Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where "system" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence. > > And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too. I don't think this matters. The new regex does indeed capture that kind of page: <img width="1052" alt="image" src="https://user-images.githubusercontent.com/9599/154752309-e1787755-3bdb-47c2-867c-7ac5fe65664d.png"> But Datasette goes through configured route regular expressions in order - so I can have the regex that captures `/db/-/special` routes listed before the one that captures tables and formats. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045099290 https://github.com/simonw/datasette/issues/1439#issuecomment-1045099290 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-SvMa simonw 9599 2022-02-18T19:56:18Z 2022-02-18T19:56:30Z OWNER > ```python > def dash_encode(s): > return s.replace("-", "--").replace(".", "-.").replace("/", "-/") > > def dash_decode(s): > return s.replace("-/", "/").replace("-.", ".").replace("--", "-") > ``` I think **dash-encoding** (new name for this) is the right way forward here. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045108611 https://github.com/simonw/datasette/issues/1439#issuecomment-1045108611 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-SxeD simonw 9599 2022-02-18T20:02:19Z 2022-02-18T20:08:34Z OWNER One other potential variant: ```python def dash_encode(s): return s.replace("-", "-dash-").replace(".", "-dot-").replace("/", "-slash-") def dash_decode(s): return s.replace("-slash-", "/").replace("-dot-", ".").replace("-dash-", "-") ``` Except this has bugs - it doesn't round-trip safely, because it can get confused about things like `-dash-slash-` in terms of is that a `-dash-` or a `-slash-`? ```pycon >>> dash_encode("/db/table-.csv.csv") '-slash-db-slash-table-dash--dot-csv-dot-csv' >>> dash_decode('-slash-db-slash-table-dash--dot-csv-dot-csv') '/db/table-.csv.csv' >>> dash_encode('-slash-db-slash-table-dash--dot-csv-dot-csv') '-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv' >>> dash_decode('-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv') '-dash/dash-db-dash/dash-table-dash--dash.dash-csv-dash.dash-csv' ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045111309 https://github.com/simonw/datasette/issues/1439#issuecomment-1045111309 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-SyIN simonw 9599 2022-02-18T20:04:24Z 2022-02-18T20:05:40Z OWNER This made me worry that my current `dash_decode()` implementation had unknown round-trip bugs, but thankfully this works OK: ```pycon >>> dash_encode("/db/table-.csv.csv") '-/db-/table---.csv-.csv' >>> dash_encode('-/db-/table---.csv-.csv') '---/db---/table-------.csv---.csv' >>> dash_decode('---/db---/table-------.csv---.csv') '-/db-/table---.csv-.csv' >>> dash_decode('-/db-/table---.csv-.csv') '/db/table-.csv.csv' ``` The regex still works against that double-encoded example too: <img width="1032" alt="image" src="https://user-images.githubusercontent.com/9599/154753916-b7d2159e-4284-4c92-ae61-110671fa320e.png"> {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045117304 https://github.com/simonw/datasette/issues/1439#issuecomment-1045117304 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-Szl4 simonw 9599 2022-02-18T20:09:22Z 2022-02-18T20:09:22Z OWNER Adopting this could result in supporting database files with surprising characters in their filename too. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1045269544 https://github.com/simonw/datasette/issues/1439#issuecomment-1045269544 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-TYwo simonw 9599 2022-02-18T22:19:29Z 2022-02-18T22:19:29Z OWNER Note that I've ruled out using `Accept: application/json` to return JSON because it turns out Cloudflare and potentially other CDNs ignore the `Vary: Accept` header entirely: - https://github.com/simonw/datasette/issues/1534 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1049114724 https://github.com/simonw/datasette/issues/1439#issuecomment-1049114724 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-iDhk simonw 9599 2022-02-23T19:04:40Z 2022-02-23T19:04:40Z OWNER I'm going to try dash encoding for table names (and row IDs) in a branch and see how I like it. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1049124390 https://github.com/simonw/datasette/issues/1439#issuecomment-1049124390 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-iF4m simonw 9599 2022-02-23T19:15:00Z 2022-02-23T19:15:00Z OWNER I'll start by modifying this function: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/utils/__init__.py#L732-L749 Later I want to move this to the routing layer to split out `format` automatically, as seen in the regexes here: https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1049126151 https://github.com/simonw/datasette/issues/1439#issuecomment-1049126151 https://api.github.com/repos/simonw/datasette/issues/1439 IC_kwDOBm6k_c4-iGUH simonw 9599 2022-02-23T19:17:01Z 2022-02-23T19:17:01Z OWNER Actually the relevant code looks to be: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/views/base.py#L481-L498 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Rethink how .ext formats (v.s. ?_format=) works before 1.0 973139047  
1049879118 https://github.com/simonw/datasette/issues/1641#issuecomment-1049879118 https://api.github.com/repos/simonw/datasette/issues/1641 IC_kwDOBm6k_c4-k-JO fgregg 536941 2022-02-24T13:49:26Z 2022-02-24T13:49:26Z NONE maybe worth considering adding buttons for paren, asterisk, etc. under the input text box on mobile? {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Tweak mobile keyboard settings 1149310456  
1051473892 https://github.com/simonw/datasette/issues/260#issuecomment-1051473892 https://api.github.com/repos/simonw/datasette/issues/260 IC_kwDOBm6k_c4-rDfk zaneselvans 596279 2022-02-26T02:24:15Z 2022-02-26T02:24:15Z NONE Is there already functionality that can be used to validate the `metadata.json` file? Is there a JSON Schema that defines it? Or a validation that's available via datasette with Python? We're working on [automatically building the metadata](https://github.com/catalyst-cooperative/pudl/pull/1479) in CI and when we deploy to cloud run, and it would be nice to be able to check whether the the metadata we're outputting is valid in our tests. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Validate metadata.json on startup 323223872  
888694144 https://github.com/simonw/datasette/issues/1405#issuecomment-888694144 https://api.github.com/repos/simonw/datasette/issues/1405 IC_kwDOBm6k_c40-GWA simonw 9599 2021-07-28T23:51:59Z 2021-07-28T23:51:59Z OWNER https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/datasette/utils/__init__.py#L918-L926 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} utils.parse_metadata() should be a documented internal function 955316250  
888694261 https://github.com/simonw/datasette/issues/1405#issuecomment-888694261 https://api.github.com/repos/simonw/datasette/issues/1405 IC_kwDOBm6k_c40-GX1 simonw 9599 2021-07-28T23:52:21Z 2021-07-28T23:52:21Z OWNER Document that it can raise a `BadMetadataError` exception. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} utils.parse_metadata() should be a documented internal function 955316250  
886968648 https://github.com/simonw/datasette/issues/1402#issuecomment-886968648 https://api.github.com/repos/simonw/datasette/issues/1402 IC_kwDOBm6k_c403hFI simonw 9599 2021-07-26T19:30:14Z 2021-07-26T19:30:14Z OWNER I really like this idea. I was thinking it might make a good plugin, but there's not a great mechanism for plugins to inject extra `<head>` content at the moment - plus this actually feels like a reasonable feature for Datasette core itself. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} feature request: social meta tags 951185411  
886969541 https://github.com/simonw/datasette/issues/1402#issuecomment-886969541 https://api.github.com/repos/simonw/datasette/issues/1402 IC_kwDOBm6k_c403hTF simonw 9599 2021-07-26T19:31:40Z 2021-07-26T19:31:40Z OWNER Datasette could do a pretty good job of this by default, using `twitter:card` and `og:url` tags - like on https://til.simonwillison.net/jq/extracting-objects-recursively I could also provide a mechanism to customize these - in particular to add images of some sort. It feels like something that should tie in to the metadata mechanism. {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0} feature request: social meta tags 951185411  
887095569 https://github.com/simonw/datasette/issues/1404#issuecomment-887095569 https://api.github.com/repos/simonw/datasette/issues/1404 IC_kwDOBm6k_c404AER simonw 9599 2021-07-26T23:27:07Z 2021-07-26T23:27:07Z OWNER Updated documentation: https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/docs/plugin_hooks.rst#register-routes-datasette {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `register_routes()` hook should take `datasette` argument 953352015  
881125124 https://github.com/simonw/datasette/issues/759#issuecomment-881125124 https://api.github.com/repos/simonw/datasette/issues/759 IC_kwDOBm6k_c40hOcE simonw 9599 2021-07-16T02:11:48Z 2021-07-16T02:11:54Z OWNER I added `"searchmode": "raw"` as a supported option for table metadata in #1389 and released that in Datasette 0.58. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} fts search on a column doesn't work anymore due to escape_fts 612673948  
881129149 https://github.com/simonw/datasette/issues/1394#issuecomment-881129149 https://api.github.com/repos/simonw/datasette/issues/1394 IC_kwDOBm6k_c40hPa9 simonw 9599 2021-07-16T02:23:32Z 2021-07-16T02:23:32Z OWNER Wrote about this in the annotated release notes for 0.58: https://simonwillison.net/2021/Jul/16/datasette-058/ {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Big performance boost on faceting: skip the inner order by 944870799  
881204782 https://github.com/simonw/datasette/issues/1231#issuecomment-881204782 https://api.github.com/repos/simonw/datasette/issues/1231 IC_kwDOBm6k_c40hh4u simonw 9599 2021-07-16T06:14:12Z 2021-07-16T06:14:12Z OWNER Here's the traceback I got from `datasette-graphql` (annoyingly only running the tests in GitHub Actions CI - I've not been able to replicate on my laptop yet): ``` tests/test_utils.py . [100%] =================================== FAILURES =================================== _________________________ test_graphql_examples[path0] _________________________ ds = <datasette.app.Datasette object at 0x7f6b8b6f8fd0> path = PosixPath('/home/runner/work/datasette-graphql/datasette-graphql/examples/filters.md') @pytest.mark.asyncio @pytest.mark.parametrize( "path", (pathlib.Path(__file__).parent.parent / "examples").glob("*.md") ) async def test_graphql_examples(ds, path): content = path.read_text() query = graphql_re.search(content)[1] try: variables = variables_re.search(content)[1] except TypeError: variables = "{}" expected = json.loads(json_re.search(content)[1]) response = await ds.client.post( "/graphql", json={ "query": query, "variables": json.loads(variables), }, ) > assert response.status_code == 200, response.json() E AssertionError: {'data': {'repos_arraycontains': None, 'users_contains': None, 'users_date': None, 'users_endswith': None, ...}, 'erro...", 'path': ['users_gt']}, {'locations': [{'column': 5, 'line': 34}], 'message': "'rows'", 'path': ['users_gte']}, ...]} E assert 500 == 200 E + where 500 = <Response [500 Internal Server Error]>.status_code tests/test_graphql.py:142: AssertionError ----------------------------- Captured stderr call ----------------------------- table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists … {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Race condition errors in new refresh_schemas() mechanism 811367257  
881204343 https://github.com/simonw/datasette/issues/1231#issuecomment-881204343 https://api.github.com/repos/simonw/datasette/issues/1231 IC_kwDOBm6k_c40hhx3 simonw 9599 2021-07-16T06:13:11Z 2021-07-16T06:13:11Z OWNER This just broke the `datasette-graphql` test suite: https://github.com/simonw/datasette-graphql/issues/77 - I need to figure out a solution here. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Race condition errors in new refresh_schemas() mechanism 811367257  
881663968 https://github.com/simonw/datasette/issues/1231#issuecomment-881663968 https://api.github.com/repos/simonw/datasette/issues/1231 IC_kwDOBm6k_c40jR_g simonw 9599 2021-07-16T19:18:42Z 2021-07-16T19:18:42Z OWNER The race condition happens inside this method - initially with the call to `await init_internal_db()`: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/app.py#L334-L359 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Race condition errors in new refresh_schemas() mechanism 811367257  
881664408 https://github.com/simonw/datasette/issues/1231#issuecomment-881664408 https://api.github.com/repos/simonw/datasette/issues/1231 IC_kwDOBm6k_c40jSGY simonw 9599 2021-07-16T19:19:35Z 2021-07-16T19:19:35Z OWNER The only place that calls `refresh_schemas()` is here: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/views/base.py#L120-L124 Ideally only one call to `refresh_schemas()` would be running at any one time. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Race condition errors in new refresh_schemas() mechanism 811367257  
881665383 https://github.com/simonw/datasette/issues/1231#issuecomment-881665383 https://api.github.com/repos/simonw/datasette/issues/1231 IC_kwDOBm6k_c40jSVn simonw 9599 2021-07-16T19:21:35Z 2021-07-16T19:21:35Z OWNER https://stackoverflow.com/a/25799871/6083 has a good example of using `asyncio.Lock()`: ```python stuff_lock = asyncio.Lock() async def get_stuff(url): async with stuff_lock: if url in cache: return cache[url] stuff = await aiohttp.request('GET', url) cache[url] = stuff return stuff ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Race condition errors in new refresh_schemas() mechanism 811367257  
881671706 https://github.com/simonw/datasette/issues/1231#issuecomment-881671706 https://api.github.com/repos/simonw/datasette/issues/1231 IC_kwDOBm6k_c40jT4a simonw 9599 2021-07-16T19:32:05Z 2021-07-16T19:32:05Z OWNER The test suite passes with that change. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Race condition errors in new refresh_schemas() mechanism 811367257  
881668759 https://github.com/simonw/datasette/issues/1231#issuecomment-881668759 https://api.github.com/repos/simonw/datasette/issues/1231 IC_kwDOBm6k_c40jTKX simonw 9599 2021-07-16T19:27:46Z 2021-07-16T19:27:46Z OWNER Second attempt at this: ```diff diff --git a/datasette/app.py b/datasette/app.py index 5976d8b..5f348cb 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -224,6 +224,7 @@ class Datasette: self.inspect_data = inspect_data self.immutables = set(immutables or []) self.databases = collections.OrderedDict() + self._refresh_schemas_lock = asyncio.Lock() self.crossdb = crossdb if memory or crossdb or not self.files: self.add_database(Database(self, is_memory=True), name="_memory") @@ -332,6 +333,12 @@ class Datasette: self.client = DatasetteClient(self) async def refresh_schemas(self): + if self._refresh_schemas_lock.locked(): + return + async with self._refresh_schemas_lock: + await self._refresh_schemas() + + async def _refresh_schemas(self): internal_db = self.databases["_internal"] if not self.internal_db_created: await init_internal_db(internal_db) ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Race condition errors in new refresh_schemas() mechanism 811367257  
881674857 https://github.com/simonw/datasette/issues/1231#issuecomment-881674857 https://api.github.com/repos/simonw/datasette/issues/1231 IC_kwDOBm6k_c40jUpp simonw 9599 2021-07-16T19:38:39Z 2021-07-16T19:38:39Z OWNER I can't replicate the race condition locally with or without this patch. I'm going to push the commit and then test the CI run from `datasette-graphql` that was failing against it. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Race condition errors in new refresh_schemas() mechanism 811367257  
881677620 https://github.com/simonw/datasette/issues/1231#issuecomment-881677620 https://api.github.com/repos/simonw/datasette/issues/1231 IC_kwDOBm6k_c40jVU0 simonw 9599 2021-07-16T19:44:12Z 2021-07-16T19:44:12Z OWNER That fixed the race condition in the `datasette-graphql` tests, which is the only place that I've been able to successfully replicate this. I'm going to land this change. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Race condition errors in new refresh_schemas() mechanism 811367257  
881686662 https://github.com/simonw/datasette/issues/1396#issuecomment-881686662 https://api.github.com/repos/simonw/datasette/issues/1396 IC_kwDOBm6k_c40jXiG simonw 9599 2021-07-16T20:02:44Z 2021-07-16T20:02:44Z OWNER Confirmed fixed: 0.58.1 was successfully published to Docker Hub in https://github.com/simonw/datasette/runs/3089447346 and the `latest` tag on https://hub.docker.com/r/datasetteproject/datasette/tags was updated. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} "invalid reference format" publishing Docker image 944903881  
882096402 https://github.com/simonw/datasette/issues/123#issuecomment-882096402 https://api.github.com/repos/simonw/datasette/issues/123 IC_kwDOBm6k_c40k7kS RayBB 921217 2021-07-18T18:07:29Z 2021-07-18T18:07:29Z NONE I also love the idea for this feature and wonder if it could work without having to download the whole database into memory at once if it's a rather large db. Obviously this could be slower but could have many use cases. My comment is partially inspired by this post about streaming sqlite dbs from github pages or such https://news.ycombinator.com/item?id=27016630 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Datasette serve should accept paths/URLs to CSVs and other file formats 275125561  
881932880 https://github.com/simonw/datasette/issues/1199#issuecomment-881932880 https://api.github.com/repos/simonw/datasette/issues/1199 IC_kwDOBm6k_c40kTpQ simonw 9599 2021-07-17T17:39:17Z 2021-07-17T17:39:17Z OWNER I asked about optimizing performance on the SQLite forum and this came up as a suggestion: https://sqlite.org/forum/forumpost/9a6b9ae8e2048c8b?t=c I can start by trying this: PRAGMA mmap_size=268435456; {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Experiment with PRAGMA mmap_size=N 792652391  
882138084 https://github.com/simonw/datasette/issues/123#issuecomment-882138084 https://api.github.com/repos/simonw/datasette/issues/123 IC_kwDOBm6k_c40lFvk simonw 9599 2021-07-19T00:04:31Z 2021-07-19T00:04:31Z OWNER I've been thinking more about this one today too. An extension of this (touched on in #417, Datasette Library) would be to support pointing Datasette at a directory and having it automatically load any CSV files it finds anywhere in that folder or its descendants - either loading them fully, or providing a UI that allows users to select a file to open it in Datasette. For larger files I think the right thing to do is import them into an on-disk SQLite database, which is limited only by available disk space. For smaller files loading them into an in-memory database should work fine. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Datasette serve should accept paths/URLs to CSVs and other file formats 275125561  
882542519 https://github.com/simonw/datasette/pull/1400#issuecomment-882542519 https://api.github.com/repos/simonw/datasette/issues/1400 IC_kwDOBm6k_c40moe3 codecov[bot] 22429695 2021-07-19T13:20:52Z 2021-07-19T13:20:52Z NONE # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1400](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e95c685) into [main](https://codecov.io/gh/simonw/datasette/commit/c73af5dd72305f6a01ea94a2c76d52e5e26de38b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c73af5d) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1400/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1400 +/- ## ======================================= Coverage 91.62% 91.62% ======================================= Files 34 34 Lines 4371 4371 ======================================= Hits 4005 4005 Misses 366 366 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov… {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Bump black from 21.6b0 to 21.7b0 947640902  
884910320 https://github.com/simonw/datasette/issues/1401#issuecomment-884910320 https://api.github.com/repos/simonw/datasette/issues/1401 IC_kwDOBm6k_c40vqjw fgregg 536941 2021-07-22T13:26:01Z 2021-07-22T13:26:01Z NONE ordered lists didn't work either, btw {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} unordered list is not rendering bullet points in description_html on database page 950664971  
904024939 https://github.com/simonw/datasette/issues/1445#issuecomment-904024939 https://api.github.com/repos/simonw/datasette/issues/1445 IC_kwDOBm6k_c414lNr simonw 9599 2021-08-23T18:52:35Z 2021-08-23T18:52:35Z OWNER The downside of the current implementation of this trick is that it only works for exact LIKE partial matches in a specific table - if you search for `dog cat` and `dog` appears in `title` but `cat` appears in `description` you won't get back that result. I think that's fine though. If you want more advanced search there are other mechanisms you can use. This is meant to be a very quick and dirty starting point for exploring a brand new table. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Ability to search for text across all columns in a table 977323133  
904026253 https://github.com/simonw/datasette/issues/1445#issuecomment-904026253 https://api.github.com/repos/simonw/datasette/issues/1445 IC_kwDOBm6k_c414liN simonw 9599 2021-08-23T18:54:49Z 2021-08-23T18:54:49Z OWNER The bigger problem here is UI design. This feels like a pretty niche requirement to me, so adding a prominent search box to the table page (which already has the filters interface, plus the full-text search box for tables that have FTS configured) feels untidy. I could tuck it away in the table cog menu, but that's a weird place for something like this to live. Maybe add it as a new type of filter? Filters apply to specific columns though, so this would be the first filter that applied to _all_ columns - which doesn't really fit the existing filter interface very well. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Ability to search for text across all columns in a table 977323133  
904027166 https://github.com/simonw/datasette/issues/1445#issuecomment-904027166 https://api.github.com/repos/simonw/datasette/issues/1445 IC_kwDOBm6k_c414lwe simonw 9599 2021-08-23T18:56:20Z 2021-08-23T18:56:20Z OWNER A related but potentially even more useful ability would be running a search across every column of every table in a whole database. For anything less than a few 100MB this could be incredibly useful. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Ability to search for text across all columns in a table 977323133  
904036200 https://github.com/simonw/datasette/issues/1445#issuecomment-904036200 https://api.github.com/repos/simonw/datasette/issues/1445 IC_kwDOBm6k_c414n9o simonw 9599 2021-08-23T19:08:54Z 2021-08-23T19:08:54Z OWNER Figured out a query for searching across every column in every table! https://til.simonwillison.net/datasette/search-all-columns-trick#user-content-same-trick-for-the-entire-database ```sql with tables as ( select name as table_name from sqlite_master where type = 'table' ), queries as ( select 'select ''' || tables.table_name || ''' as _table, rowid from "' || tables.table_name || '" where ' || group_concat( '"' || name || '" like ''%'' || :search || ''%''', ' or ' ) as query from pragma_table_info(tables.table_name), tables group by tables.table_name ) select group_concat(query, ' union all ') from queries ``` The SQL query this generates for larger databases is _extremely_ long - but it does seem to work for smaller databases. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Ability to search for text across all columns in a table 977323133  
904037087 https://github.com/simonw/datasette/issues/1445#issuecomment-904037087 https://api.github.com/repos/simonw/datasette/issues/1445 IC_kwDOBm6k_c414oLf simonw 9599 2021-08-23T19:10:17Z 2021-08-23T19:10:17Z OWNER Rather than trying to run that monstrosity in a single `union all` query, a better approach may be to use `fetch()` requests as seen in https://datasette.io/plugins/datasette-search-all {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Ability to search for text across all columns in a table 977323133  
904866495 https://github.com/simonw/datasette/issues/1446#issuecomment-904866495 https://api.github.com/repos/simonw/datasette/issues/1446 IC_kwDOBm6k_c417yq_ simonw 9599 2021-08-24T18:13:49Z 2021-08-24T18:13:49Z OWNER OK, now the following optional CSS gives us a sticky footer: ```css html, body { height: 100%; } body { display: flex; flex-direction: column; } .not-footer { flex: 1 0 auto; } footer { flex-shrink: 0; } ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Modify base.html template to support optional sticky footer 978357984  
904954530 https://github.com/simonw/datasette/issues/1446#issuecomment-904954530 https://api.github.com/repos/simonw/datasette/issues/1446 IC_kwDOBm6k_c418IKi simonw 9599 2021-08-24T20:32:47Z 2021-08-24T20:32:47Z OWNER Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Modify base.html template to support optional sticky footer 978357984  
904982056 https://github.com/simonw/datasette/issues/859#issuecomment-904982056 https://api.github.com/repos/simonw/datasette/issues/859 IC_kwDOBm6k_c418O4o brandonrobertz 2670795 2021-08-24T21:15:04Z 2021-08-24T21:15:30Z CONTRIBUTOR I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Database page loads too slowly with many large tables (due to table counts) 642572841  
905097468 https://github.com/simonw/datasette/pull/1447#issuecomment-905097468 https://api.github.com/repos/simonw/datasette/issues/1447 IC_kwDOBm6k_c418rD8 simonw 9599 2021-08-25T01:28:53Z 2021-08-25T01:28:53Z OWNER Good catch, thanks! {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Remove underscore from search mode parameter name 978614898  
889525741 https://github.com/simonw/datasette/issues/1405#issuecomment-889525741 https://api.github.com/repos/simonw/datasette/issues/1405 IC_kwDOBm6k_c41BRXt simonw 9599 2021-07-29T23:33:30Z 2021-07-29T23:33:30Z OWNER New documentation section for `datasette.utils` is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} utils.parse_metadata() should be a documented internal function 955316250  
889539227 https://github.com/simonw/datasette/issues/1241#issuecomment-889539227 https://api.github.com/repos/simonw/datasette/issues/1241 IC_kwDOBm6k_c41BUqb simonw 9599 2021-07-30T00:15:26Z 2021-07-30T00:15:26Z OWNER One possible treatment: <img width="442" alt="fixtures__sortable__201_rows" src="https://user-images.githubusercontent.com/9599/127581105-dca1193c-1bdd-491e-89d1-ac3136c05c3b.png"> ```html <style> a.action-button { display: inline-block; border: 1.5px solid #666; border-radius: 0.5em; background-color: #ffffff; color: #666; padding: 0.1em 0.8em; text-decoration: none; margin-right: 0.3em; font-size: 0.85em; } </style> <p> {% if query.sql and allow_execute_sql %} <a class="action-button" title="{{ query.sql }}" href="{{ urls.database(database) }}?{{ {'sql': query.sql}|urlencode|safe }}{% if query.params %}&amp;{{ query.params|urlencode|safe }}{% endif %}"> View and edit SQL </a> {% endif %} <a href="#" class="action-button">Copy and share link</a> </p> ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Share button for copying current URL 814595021  
889548536 https://github.com/simonw/datasette/issues/1406#issuecomment-889548536 https://api.github.com/repos/simonw/datasette/issues/1406 IC_kwDOBm6k_c41BW74 simonw 9599 2021-07-30T00:43:47Z 2021-07-30T00:43:47Z OWNER Still couldn't replicate on my laptop. On a hunch, I'm going to add `@pytest.mark.serial` to every test that uses `runner.isolated_filesystem()`. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470  
889547142 https://github.com/simonw/datasette/issues/1406#issuecomment-889547142 https://api.github.com/repos/simonw/datasette/issues/1406 IC_kwDOBm6k_c41BWmG simonw 9599 2021-07-30T00:39:49Z 2021-07-30T00:39:49Z OWNER It happens in CI but not on my laptop. I think I need to run the tests on my laptop like this: https://github.com/simonw/datasette/blob/121e10c29c5b412fddf0326939f1fe46c3ad9d4a/.github/workflows/test.yml#L27-L30 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470  
889550391 https://github.com/simonw/datasette/issues/1406#issuecomment-889550391 https://api.github.com/repos/simonw/datasette/issues/1406 IC_kwDOBm6k_c41BXY3 simonw 9599 2021-07-30T00:49:31Z 2021-07-30T00:49:31Z OWNER That fixed it. My hunch is that Click's `runner.isolated_filesystem()` mechanism doesn't play well with `pytest-xdist`. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470  
889553052 https://github.com/simonw/datasette/issues/1406#issuecomment-889553052 https://api.github.com/repos/simonw/datasette/issues/1406 IC_kwDOBm6k_c41BYCc simonw 9599 2021-07-30T00:58:43Z 2021-07-30T00:58:43Z OWNER Tests are still failing in the job that calculates coverage. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470  
889555977 https://github.com/simonw/datasette/issues/1406#issuecomment-889555977 https://api.github.com/repos/simonw/datasette/issues/1406 IC_kwDOBm6k_c41BYwJ simonw 9599 2021-07-30T01:06:57Z 2021-07-30T01:06:57Z OWNER Looking at the source code in Click for `isolated_filesystem()`: https://github.com/pallets/click/blob/9da166957f5848b641231d485467f6140bca2bc0/src/click/testing.py#L450-L468 ```python @contextlib.contextmanager def isolated_filesystem( self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None ) -> t.Iterator[str]: """A context manager that creates a temporary directory and changes the current working directory to it. This isolates tests that affect the contents of the CWD to prevent them from interfering with each other. :param temp_dir: Create the temporary directory under this directory. If given, the created directory is not removed when exiting. .. versionchanged:: 8.0 Added the ``temp_dir`` parameter. """ cwd = os.getcwd() t = tempfile.mkdtemp(dir=temp_dir) os.chdir(t) ``` How about if I pass in that optional `temp_dir` as a temp directory created using the `pytest-xdist` aware pytest mechanisms: https://docs.pytest.org/en/6.2.x/tmpdir.html {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470  
889599513 https://github.com/simonw/datasette/issues/1398#issuecomment-889599513 https://api.github.com/repos/simonw/datasette/issues/1398 IC_kwDOBm6k_c41BjYZ aitoehigie 192984 2021-07-30T03:21:49Z 2021-07-30T03:21:49Z NONE Does the library support this now? {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Documentation on using Datasette as a library 947044667  
890259755 https://github.com/simonw/datasette/issues/1406#issuecomment-890259755 https://api.github.com/repos/simonw/datasette/issues/1406 IC_kwDOBm6k_c41EEkr simonw 9599 2021-07-31T00:04:54Z 2021-07-31T00:04:54Z OWNER STILL failing. I'm going to try removing all instances of `isolated_filesystem()` in favour of a different pattern using pytest temporary files, then see if I can get that to work without the serial hack. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470  
890388200 https://github.com/simonw/datasette/issues/1407#issuecomment-890388200 https://api.github.com/repos/simonw/datasette/issues/1407 IC_kwDOBm6k_c41Ej7o simonw 9599 2021-07-31T18:38:41Z 2021-07-31T18:38:41Z OWNER The `path` variable there looked like this: `/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-696/popen-gw0/uds0/datasette.sock` I think what's happening here is that `pytest-xdist` causes `tmp_path_factory.mktemp("uds")` to create significantly longer paths, which in this case is breaking some limit. So for this code to work with `pytest-xdist` I need to make sure the random path to `datasette.sock` is shorter. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} OSError: AF_UNIX path too long in ds_unix_domain_socket_server 957298475  
890388656 https://github.com/simonw/datasette/issues/1407#issuecomment-890388656 https://api.github.com/repos/simonw/datasette/issues/1407 IC_kwDOBm6k_c41EkCw simonw 9599 2021-07-31T18:42:41Z 2021-07-31T18:42:41Z OWNER I'll try `tempfile.gettempdir()` - on macOS it returns something like `'/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T'` which is still long but hopefully not too long. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} OSError: AF_UNIX path too long in ds_unix_domain_socket_server 957298475  
890390198 https://github.com/simonw/datasette/issues/1406#issuecomment-890390198 https://api.github.com/repos/simonw/datasette/issues/1406 IC_kwDOBm6k_c41Eka2 simonw 9599 2021-07-31T18:55:33Z 2021-07-31T18:55:33Z OWNER To clarify: the core problem here is that an error is thrown any time you call `os.getcwd()` but the directory you are currently in has been deleted. `runner.isolated_filesystem()` assumes that the current directory in has not been deleted. But the various temporary directory utilities in `pytest` work by creating directories and then deleting them. Maybe there's a larger problem here that I play a bit fast and loose with `os.chdir()` in both the test suite and in various lines of code in Datasette itself (in particular in the publish commands)? {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470  
890390342 https://github.com/simonw/datasette/issues/1406#issuecomment-890390342 https://api.github.com/repos/simonw/datasette/issues/1406 IC_kwDOBm6k_c41EkdG simonw 9599 2021-07-31T18:56:35Z 2021-07-31T18:56:35Z OWNER But... I've lost enough time to this already, and removing `runner.isolated_filesystem()` has the tests passing again. So I'm not going to work on this any more. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Tests failing with FileNotFoundError in runner.isolated_filesystem 956303470  
890390495 https://github.com/simonw/datasette/issues/1408#issuecomment-890390495 https://api.github.com/repos/simonw/datasette/issues/1408 IC_kwDOBm6k_c41Ekff simonw 9599 2021-07-31T18:57:39Z 2021-07-31T18:57:39Z OWNER Opening this issue as an optional follow-up to the work I did in #1406. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Review places in codebase that use os.chdir(), in particularly relating to tests 957302085  
890390845 https://github.com/simonw/datasette/issues/1408#issuecomment-890390845 https://api.github.com/repos/simonw/datasette/issues/1408 IC_kwDOBm6k_c41Ekk9 simonw 9599 2021-07-31T19:00:32Z 2021-07-31T19:00:32Z OWNER When I revisit this I can also look at dropping the `@pytest.mark.serial` hack, and maybe the `restore_working_directory()` fixture hack too: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/pytest.ini#L9-L10 https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/tests/conftest.py#L62-L75 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Review places in codebase that use os.chdir(), in particularly relating to tests 957302085  
890400059 https://github.com/simonw/datasette/issues/1409#issuecomment-890400059 https://api.github.com/repos/simonw/datasette/issues/1409 IC_kwDOBm6k_c41Em07 simonw 9599 2021-07-31T20:21:51Z 2021-07-31T20:21:51Z OWNER One of these two options: - `--setting default_allow_sql off` - `--setting allow_sql_default off` Existing settings from https://docs.datasette.io/en/0.58.1/settings.html with similar names that I need to be consistent with: - `default_page_size` - `allow_facet` - `default_facet_size` - `allow_download` - `default_cache_ttl` - `default_cache_ttl_hashed` - `allow_csv_stream` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278  
890400121 https://github.com/simonw/datasette/issues/1409#issuecomment-890400121 https://api.github.com/repos/simonw/datasette/issues/1409 IC_kwDOBm6k_c41Em15 simonw 9599 2021-07-31T20:22:21Z 2021-07-31T20:23:34Z OWNER I think `default_allow_sql` is more consistent with the current naming conventions, because both `allow` and `default` are used as prefixes at the moment but neither of them are ever used as a suffix. Plus `default_allow_sql off` makes sense to me but `allow_default_sql off` does not - what is "default SQL"? {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278  
890400425 https://github.com/simonw/datasette/issues/1409#issuecomment-890400425 https://api.github.com/repos/simonw/datasette/issues/1409 IC_kwDOBm6k_c41Em6p simonw 9599 2021-07-31T20:25:16Z 2021-07-31T20:26:25Z OWNER If I was prone to over-thinking (which I am) I'd note that `allow_facet` and `allow_download` and `allow_csv_stream` are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent. So maybe there's a future task where I introduce those as both permissions and metadata `"allow_x"` blocks, then rename the settings themselves to be called `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`. If I was going to do that I should get it in before Datasette 1.0. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278  
890397124 https://github.com/simonw/datasette/issues/1409#issuecomment-890397124 https://api.github.com/repos/simonw/datasette/issues/1409 IC_kwDOBm6k_c41EmHE simonw 9599 2021-07-31T19:51:10Z 2021-07-31T19:51:10Z OWNER I think I may like `disable_sql` better. Some options: - `--setting allow_sql off` (consistent with `allow_facet` and `allow_download` and `allow_csv_stream` - all which default to `on` already) - `--setting disable_sql on` - `--setting disable_custom_sql on` The existence of three `allow_*` settings does make a strong argument for staying consistent with that. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278  
890397169 https://github.com/simonw/datasette/issues/1409#issuecomment-890397169 https://api.github.com/repos/simonw/datasette/issues/1409 IC_kwDOBm6k_c41EmHx simonw 9599 2021-07-31T19:51:35Z 2021-07-31T19:51:35Z OWNER I'm going to stick with `--setting allow_sql off`. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278  
890397261 https://github.com/simonw/datasette/issues/1409#issuecomment-890397261 https://api.github.com/repos/simonw/datasette/issues/1409 IC_kwDOBm6k_c41EmJN simonw 9599 2021-07-31T19:52:25Z 2021-07-31T19:52:25Z OWNER I think I can make this modification by teaching the default permissions code here to take the `allow_sql` setting into account: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/datasette/default_permissions.py#L38-L45 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278  
890397652 https://github.com/simonw/datasette/issues/1409#issuecomment-890397652 https://api.github.com/repos/simonw/datasette/issues/1409 IC_kwDOBm6k_c41EmPU simonw 9599 2021-07-31T19:56:48Z 2021-07-31T19:56:48Z OWNER The other option would be to use the setting to pick the `default=` argument when calling `self.ds.permission_allowed( request.actor, "execute-sql", resource=database, default=True)`. The problem with that is that there are actually a few different places which perform that check, so changing all of them raises the risk of missing one in the future: https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L436-L444 https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L964-L966 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L220-L221 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L343-L345 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L134-L136 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278  
890397753 https://github.com/simonw/datasette/issues/1409#issuecomment-890397753 https://api.github.com/repos/simonw/datasette/issues/1409 IC_kwDOBm6k_c41EmQ5 simonw 9599 2021-07-31T19:57:56Z 2021-07-31T19:57:56Z OWNER I think the correct solution is for the default permissions logic to take the `allow_sql` setting into account, and to return `False` if that setting is set to `off` AND the current actor fails the `actor_matches_allow` checks. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278  
890399806 https://github.com/simonw/datasette/issues/1409#issuecomment-890399806 https://api.github.com/repos/simonw/datasette/issues/1409 IC_kwDOBm6k_c41Emw- simonw 9599 2021-07-31T20:18:46Z 2021-07-31T20:18:46Z OWNER My rationale for removing it: https://github.com/simonw/datasette/issues/813#issuecomment-640916290 > Naming problem: Datasette already has a config option with this name: > > $ datasette serve data.db --config allow_sql:1 > > https://datasette.readthedocs.io/en/stable/config.html#allow-sql > > It's confusing to have two things called `allow_sql` that do slightly different things. > > I could retire the `--config allow_sql:0` option entirely, since the new `metadata.json` mechanism can be used to achieve the exact same thing. > > I'm going to do that. This is true. The `"allow_sql"` permissions block in `metadata.json` does indeed have a name that is easily confused with `--setting allow_sql off`. So I definitely need to pick a different name from the setting. `--setting default_allow_sql off` is a good option here. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting) 957310278  
890441844 https://github.com/simonw/datasette/issues/1411#issuecomment-890441844 https://api.github.com/repos/simonw/datasette/issues/1411 IC_kwDOBm6k_c41ExB0 simonw 9599 2021-08-01T03:27:30Z 2021-08-01T03:27:30Z OWNER Confirmed: https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1 no longer exhibits the bug. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Canned query ?sql= is pointlessly echoed in query string starting from hidden mode 957345476  
891352132 https://github.com/simonw/datasette/issues/1227#issuecomment-891352132 https://api.github.com/repos/simonw/datasette/issues/1227 IC_kwDOBm6k_c41IPRE simonw 9599 2021-08-02T21:38:39Z 2021-08-02T21:38:39Z OWNER Relevant TIL: https://til.simonwillison.net/vscode/vs-code-regular-expressions {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Configure sphinx.ext.extlinks for issues 810394616  
891979858 https://github.com/simonw/datasette/issues/1417#issuecomment-891979858 https://api.github.com/repos/simonw/datasette/issues/1417 IC_kwDOBm6k_c41KohS simonw 9599 2021-08-03T16:15:30Z 2021-08-03T16:15:30Z OWNER Docs: https://pypi.org/project/codespell/ There's a `codespell --ignore-words=FILE` option for ignoring words. I don't have any that need ignoring yet but I'm going to add that file anyway, that way I can have codespell be a failing test but still provide a way to work around it if it incorrectly flags a correct word. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Use codespell in CI to spot spelling errors 959278472  
891984359 https://github.com/simonw/datasette/pull/1418#issuecomment-891984359 https://api.github.com/repos/simonw/datasette/issues/1418 IC_kwDOBm6k_c41Kpnn simonw 9599 2021-08-03T16:21:56Z 2021-08-03T16:21:56Z OWNER Failed with: ``` docs/authentication.rst:63: perfom ==> perform docs/authentication.rst:76: perfom ==> perform docs/changelog.rst:429: repsonse ==> response docs/changelog.rst:503: permissons ==> permissions docs/changelog.rst:717: compatibilty ==> compatibility docs/changelog.rst:1172: browseable ==> browsable docs/deploying.rst:191: similiar ==> similar docs/internals.rst:434: Respons ==> Response, respond docs/internals.rst:440: Respons ==> Response, respond docs/internals.rst:717: tha ==> than, that, the docs/performance.rst:42: databse ==> database docs/plugin_hooks.rst:667: utilites ==> utilities docs/publish.rst:168: countainer ==> container docs/settings.rst:352: inalid ==> invalid docs/sql_queries.rst:406: preceeded ==> preceded, proceeded ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Spelling corrections plus CI job for codespell 959284434  
891987129 https://github.com/simonw/datasette/pull/1418#issuecomment-891987129 https://api.github.com/repos/simonw/datasette/issues/1418 IC_kwDOBm6k_c41KqS5 codecov[bot] 22429695 2021-08-03T16:26:01Z 2021-08-03T16:32:20Z NONE # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1418](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (532170f) into [main](https://codecov.io/gh/simonw/datasette/commit/54b6e96ee8aa553b6671e341a1944f93f3fb89c3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (54b6e96) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1418/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1418 +/- ## ======================================= Coverage 91.64% 91.64% ======================================= Files 34 34 Lines 4382 4382 ======================================= Hits 4016 4016 Misses 366 366 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov… {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Spelling corrections plus CI job for codespell 959284434  
892276385 https://github.com/simonw/datasette/issues/1419#issuecomment-892276385 https://api.github.com/repos/simonw/datasette/issues/1419 IC_kwDOBm6k_c41Lw6h fgregg 536941 2021-08-04T00:58:49Z 2021-08-04T00:58:49Z NONE yes, [filter clause on aggregate queries were added to sqlite3 in 3.30](https://www.sqlite.org/releaselog/3_30_1.html) {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `publish cloudrun` should deploy a more recent SQLite version 959710008  
892365639 https://github.com/simonw/datasette/issues/1420#issuecomment-892365639 https://api.github.com/repos/simonw/datasette/issues/1420 IC_kwDOBm6k_c41MGtH simonw 9599 2021-08-04T05:05:07Z 2021-08-04T05:05:07Z OWNER https://github.com/simonw/datasette/blob/cd8b7bee8fb5c1cdce7c8dbfeb0166011abc72c6/datasette/publish/cloudrun.py#L153-L158 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `datasette publish cloudrun --cpu X` option 959898166  
892372509 https://github.com/simonw/datasette/issues/1420#issuecomment-892372509 https://api.github.com/repos/simonw/datasette/issues/1420 IC_kwDOBm6k_c41MIYd simonw 9599 2021-08-04T05:22:29Z 2021-08-04T05:22:29Z OWNER Testing this manually with: datasette publish cloudrun fixtures.db --memory 8G --cpu 4 \ --service fixtures-over-provisioned-issue-1420 --install datasette-psutil And for comparison: datasette publish cloudrun fixtures.db --service fixtures-default-issue-1420 \ --install datasette-psutil {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `datasette publish cloudrun --cpu X` option 959898166  
892374253 https://github.com/simonw/datasette/issues/1420#issuecomment-892374253 https://api.github.com/repos/simonw/datasette/issues/1420 IC_kwDOBm6k_c41MIzt simonw 9599 2021-08-04T05:27:21Z 2021-08-04T05:29:59Z OWNER I'll delete these services shortly afterwards. Right now: https://fixtures-over-provisioned-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil (deployed with `--memory 8G --cpu 4`) returns: ``` process.memory_info() pmem(rss=60456960, vms=518930432, shared=0, text=0, lib=0, data=0, dirty=0) ... psutil.cpu_times(True) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) psutil.virtual_memory() svmem(total=2147483648, available=2092531712, percent=2.6, used=33103872, free=2092531712, active=44130304, inactive=10792960, buffers=0, cached=21848064, shared=262144, slab=0) ``` https://fixtures-default-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil returns: ``` process.memory_info() pmem(rss=49324032, vms=140595200, shared=0, text=0, lib=0, data=0, dirty=0) ... psutil.cpu_times(True) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) psutil.virtual_memory() svmem(total=2147483648, available=2091188224, percent=2.6, used=40071168, free=2091188224, active=41586688, inactive=7983104, buffers=0, cached=16224256, shared=262144, slab=0) ``` These numbers are different enough that I assume this works as advertised. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `datasette publish cloudrun --cpu X` option 959898166  
892376353 https://github.com/simonw/datasette/issues/1420#issuecomment-892376353 https://api.github.com/repos/simonw/datasette/issues/1420 IC_kwDOBm6k_c41MJUh simonw 9599 2021-08-04T05:33:12Z 2021-08-04T05:33:12Z OWNER In the Cloud Run console (before I deleted these services) when I click "Edit and deploy new revision" I see this for the default one: <img width="551" alt="Deploy_new_revision_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/128127309-7ec2064f-b107-4d99-b630-ee58734c4a72.png"> And this for the big one: <img width="546" alt="Deploy_new_revision_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/128127366-7cdcb245-2e83-4b88-9d4f-0c595a486139.png"> {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `datasette publish cloudrun --cpu X` option 959898166  
893079520 https://github.com/simonw/datasette/issues/1420#issuecomment-893079520 https://api.github.com/repos/simonw/datasette/issues/1420 IC_kwDOBm6k_c41O0_g simonw 9599 2021-08-05T00:54:59Z 2021-08-05T00:54:59Z OWNER Just saw this error: `ERROR: (gcloud.run.deploy) The `--cpu` flag is not supported on the fully managed version of Cloud Run. Specify `--platform gke` or run `gcloud config set run/platform gke` to work with Cloud Run for Anthos deployed on Google Cloud.` Which is weird because I managed to run this successfully the other day. Maybe a region difference thing? {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `datasette publish cloudrun --cpu X` option 959898166  
893114612 https://github.com/simonw/datasette/issues/1419#issuecomment-893114612 https://api.github.com/repos/simonw/datasette/issues/1419 IC_kwDOBm6k_c41O9j0 fgregg 536941 2021-08-05T02:29:06Z 2021-08-05T02:29:06Z NONE there's a lot of complexity here, that's probably not worth addressing. i got what i needed by patching the dockerfile that cloudrun uses to install a newer version of sqlite. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `publish cloudrun` should deploy a more recent SQLite version 959710008  
893122356 https://github.com/simonw/datasette/issues/1422#issuecomment-893122356 https://api.github.com/repos/simonw/datasette/issues/1422 IC_kwDOBm6k_c41O_c0 simonw 9599 2021-08-05T02:52:31Z 2021-08-05T02:52:44Z OWNER If you do this it should still be possible to view the SQL - which means we need a new parameter. I propose `?_show_sql=1` to over-ride the hidden default. I think the configuration should use `hide_sql: true` - looking like this: ```yaml databases: fixtures: queries: neighborhood_search: hide_sql: true sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood title: Search neighborhoods ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Ability to default to hiding the SQL for a canned query 961367843  
893131703 https://github.com/simonw/datasette/issues/1422#issuecomment-893131703 https://api.github.com/repos/simonw/datasette/issues/1422 IC_kwDOBm6k_c41PBu3 simonw 9599 2021-08-05T03:16:46Z 2021-08-05T03:16:46Z OWNER The logic for this is a little bit fiddly, due to the need to switch to using `?_show_sql=1` on the link depending on the context. - If metadata says hide and there's no query string, hide and link to `?_show_sql=1` - If metadata says hide but query string says `?_show_sql=1`, show and have hide link linking to URL without `?_show_sql=1` - Otherwise, show and link to `?_hide_sql=1` - ... or if that query string is there then hide and link to URL without `?_hide_sql=1` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Ability to default to hiding the SQL for a canned query 961367843  
893133496 https://github.com/simonw/datasette/issues/1419#issuecomment-893133496 https://api.github.com/repos/simonw/datasette/issues/1419 IC_kwDOBm6k_c41PCK4 simonw 9599 2021-08-05T03:22:44Z 2021-08-05T03:22:44Z OWNER I ran into this exact same problem today! I only just learned how to use filter on aggregates: https://til.simonwillison.net/sqlite/sqlite-aggregate-filter-clauses A workaround I used is to add this to the deploy command: datasette publish cloudrun ... --install=pysqlite3-binary This will install the https://pypi.org/project/pysqlite3-binary for package which bundles a more recent SQLite version. {"total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} `publish cloudrun` should deploy a more recent SQLite version 959710008  
893996604 https://github.com/simonw/datasette/issues/1423#issuecomment-893996604 https://api.github.com/repos/simonw/datasette/issues/1423 IC_kwDOBm6k_c41SU48 simonw 9599 2021-08-06T04:43:07Z 2021-08-06T04:43:37Z OWNER Problem: on a page which doesn't have quite enough facet values to trigger the display of the "..." link that links to `?_facet_size=max` the user would still have to manually count the values - up to 30 by default. So maybe the count should always be shown, perhaps as a non-bold light colored number? I could even hide it in a non-discoverable tooltip. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Show count of facet values if ?_facet_size=max 962391325  
894452990 https://github.com/simonw/datasette/issues/1423#issuecomment-894452990 https://api.github.com/repos/simonw/datasette/issues/1423 IC_kwDOBm6k_c41UET- simonw 9599 2021-08-06T18:49:37Z 2021-08-06T18:49:37Z OWNER Could display them always, like this: <img width="1299" alt="fixtures__compound_three_primary_keys__1_001_rows_and_School_enrollment_analysis_Colorado_-_Google_Docs" src="https://user-images.githubusercontent.com/9599/128557997-6bedb52b-45f1-4ab1-9a79-ddf6c01f6036.png"> That's with `<span style="font-size: 0.8em;color: #666;">23</span>` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Show count of facet values if ?_facet_size=max 962391325  
894453520 https://github.com/simonw/datasette/issues/1423#issuecomment-894453520 https://api.github.com/repos/simonw/datasette/issues/1423 IC_kwDOBm6k_c41UEcQ simonw 9599 2021-08-06T18:50:40Z 2021-08-06T18:50:40Z OWNER Point of confusion: if only 30 options are shown, but there's a `...` at the end, what would the number be? It can't be the total number of facets because we haven't counted them all - but if it's just the number of displayed facets that's like to be confusing. So the original idea of showing the counts only if `_facet_size=max` is a good one. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Show count of facet values if ?_facet_size=max 962391325  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);