github
| html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
|---|---|---|---|---|---|---|---|---|---|---|---|
| https://github.com/simonw/datasette/issues/1439#issuecomment-1053973425 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1053973425 | IC_kwDOBm6k_c4-0lux | 9599 | 2022-02-28T07:40:12Z | 2022-02-28T07:40:12Z | OWNER | If I make this change it will break existing links to one of the oldest Datasette demos: http://fivethirtyeight.datasettes.com/fivethirtyeight/avengers%2Favengers A plugin that fixes those by redirecting them on 404 would be neat. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/327#issuecomment-1043609198 | https://api.github.com/repos/simonw/datasette/issues/327 | 1043609198 | IC_kwDOBm6k_c4-NDZu | 208018 | 2022-02-17T23:21:36Z | 2022-02-17T23:33:01Z | NONE | On fly.io. This particular database goes from 1.4GB to 200M. Slower, part of that might be having no `--inspect-file`? ``` $ datasette publish fly ... --generate-dir /tmp/deploy-this ... $ mksquashfs large.db large.squashfs $ rm large.db # don't accidentally put it in the image $ cat Dockerfile FROM python:3.8 COPY . /app WORKDIR /app ENV DATASETTE_SECRET 'xyzzy' RUN pip install -U datasette # RUN datasette inspect large.db --inspect-file inspect-data.json ENV PORT 8080 EXPOSE 8080 CMD mount -o loop -t squashfs large.squashfs /mnt; datasette serve --host 0.0.0.0 -i /mnt/large.db --cors --port $PORT ``` It would also be possible to copy the file onto the ~6GB available on the ephemeral container filesystem on startup. A little against the spirit of the thing? On this example the whole docker image is 2.42 GB and the squashfs version is 1.14 GB. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 335200136 | |
| https://github.com/simonw/datasette/issues/327#issuecomment-1043626870 | https://api.github.com/repos/simonw/datasette/issues/327 | 1043626870 | IC_kwDOBm6k_c4-NHt2 | 208018 | 2022-02-17T23:37:24Z | 2022-02-17T23:37:24Z | NONE | On second thought any kind of quick-to-decompress-on-startup could be helpful if we're paying for the container registry and deployment bandwidth but not ephemeral storage. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 335200136 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045131086 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045131086 | IC_kwDOBm6k_c4-S29O | 9599 | 2022-02-18T20:22:13Z | 2022-02-18T20:22:47Z | OWNER | Should it encode `%` symbols too, since they have a special meaning in URLs and we can't guarantee that every single web server / proxy out there will round-trip them safely using percentage encoding? If so, would need to pick a different encoding character for them. Maybe `%` becomes `-p` - and in that case `/` could become `-s` too. Is it worth expanding dash-encoding outside of just `/` and `-` and `.` though? Not sure. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045134050 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045134050 | IC_kwDOBm6k_c4-S3ri | 9599 | 2022-02-18T20:25:04Z | 2022-02-18T20:25:04Z | OWNER | Here's a useful modern spec for how existing URL percentage encoding is supposed to work: https://url.spec.whatwg.org/#percent-encoded-bytes | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045024276 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045024276 | IC_kwDOBm6k_c4-Sc4U | 9599 | 2022-02-18T19:01:42Z | 2022-02-18T19:55:24Z | OWNER | > Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly. ```python def dash_encode(s): return s.replace("-", "--").replace(".", "-.").replace("/", "-/") def dash_decode(s): return s.replace("-/", "/").replace("-.", ".").replace("--", "-") ``` ```pycon >>> dash_encode("foo/bar/baz.csv") 'foo-/bar-/baz-.csv' >>> dash_decode('foo-/bar-/baz-.csv') 'foo/bar/baz.csv' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045027067 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045027067 | IC_kwDOBm6k_c4-Sdj7 | 9599 | 2022-02-18T19:03:26Z | 2022-02-18T19:03:26Z | OWNER | (If I make this change it may break some existing Datasette installations when they upgrade - I could try and build a plugin for them which triggers on 404s and checks to see if the old format would return a 200 response, then returns that.) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045032377 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045032377 | IC_kwDOBm6k_c4-Se25 | 9599 | 2022-02-18T19:06:50Z | 2022-02-18T19:06:50Z | OWNER | How does URL routing for https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv work? Right now it's https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/app.py#L1098-L1101 That's not going to capture the dot-dash encoding version of that table name: ```pycon >>> dot_dash_encode("table/with/slashes.csv") 'table-/with-/slashes-.csv' ``` Probably needs a fancy regex trick like a negative lookbehind assertion or similar. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045055772 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045055772 | IC_kwDOBm6k_c4-Skkc | 9599 | 2022-02-18T19:23:33Z | 2022-02-18T19:25:42Z | OWNER | I want a match for this URL: /db/table-/with-/slashes-.csv Maybe this: ^/(?P<db_name>[^/]+)/(?P<table_and_format>([^/]*|(\-/)*|(\-\.)*|(\.\.)*)*$) Here we are matching a sequence of: ([^/]*|(\-/)*|(\-\.)*|(\-\-)*)* So a combination of not-slashes OR -/ or -. Or -- sequences <img width="224" alt="image" src="https://user-images.githubusercontent.com/9599/154748362-84909d4e-dccf-454b-a9cd-a036f9f66f09.png"> ^/(?P<db_name>[^/]+)/(?P<table_and_format>([^/]*|(\-/)*|(\-\.)*|(\-\-)*)*$) Try that with non-capturing bits: ^/(?P<db_name>[^/]+)/(?P<table_and_format>(?:[^/]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*$) `(?:[^/]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*` visualized is: <img width="193" alt="image" src="https://user-images.githubusercontent.com/9599/154748441-decea502-0d04-44f4-9ca9-fb6883767833.png"> Here's the explanation on regex101.com https://regex101.com/r/CPnsIO/1 <img width="1074" alt="image" src="https://user-images.githubusercontent.com/9599/154748720-cdda61db-5498-49a8-91c2-e726b394fa49.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045059427 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045059427 | IC_kwDOBm6k_c4-Sldj | 9599 | 2022-02-18T19:26:25Z | 2022-02-18T19:26:25Z | OWNER | With this new pattern I could probably extract out the optional `.json` format string as part of the initial route capturing regex too, rather than the current `table_and_format` hack. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045069481 | IC_kwDOBm6k_c4-Sn6p | 9599 | 2022-02-18T19:34:41Z | 2022-03-05T21:32:22Z | OWNER | I think I got format extraction working! https://regex101.com/r/A0bW1D/1 ^/(?P<database>[^/]+)/(?P<table>(?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)*?)(?:(?<!\-)\.(?P<format>\w+))?$ I had to make that crazy inner one even more complicated to stop it from capturing `.` that was not part of `-.`. (?:[^\/\-\.]*|(?:\-/)*|(?:\-\.)*|(?:\-\-)*)* Visualized: <img width="222" alt="image" src="https://user-images.githubusercontent.com/9599/154749714-44579899-5dc7-4e5f-ad4f-dc59dac48979.png"> So now I have a regex which can extract out the dot-encoded table name AND spot if there is an optional `.format` at the end: <img width="1090" alt="image" src="https://user-images.githubusercontent.com/9599/156900484-7912073f-28aa-4301-86e2-e5cbe625e1d5.png"> If I end up using this in Datasette it's going to need VERY comprehensive unit tests and inline documentation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045077590 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045077590 | IC_kwDOBm6k_c4-Sp5W | 9599 | 2022-02-18T19:41:37Z | 2022-02-18T19:42:41Z | OWNER | Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where "system" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence. And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too. Maybe change this system to use `.` as the escaping character instead of `-`? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045075207 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045075207 | IC_kwDOBm6k_c4-SpUH | 9599 | 2022-02-18T19:39:35Z | 2022-02-18T19:40:13Z | OWNER | > And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this: > > * `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version > * `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version > * `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version Here's what those look like with the updated version of `dot_dash_encode()` that also encodes `/` as `-/`: - `/db/-/db-/table---.csv-.csv` - HTML - `/db/-/db-/table---.csv-.csv.csv` - CSV - `/db/-/db-/table---.csv-.csv.json` - JSON <img width="1050" alt="image" src="https://user-images.githubusercontent.com/9599/154750631-a8a23c62-3dfc-43e4-8026-4d117dc4bf8d.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045081042 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045081042 | IC_kwDOBm6k_c4-SqvS | 9599 | 2022-02-18T19:44:12Z | 2022-02-18T19:51:34Z | OWNER | ```python def dot_encode(s): return s.replace(".", "..").replace("/", "./") def dot_decode(s): return s.replace("./", "/").replace("..", ".") ``` No need for hyphen encoding in this variant at all, which simplifies things a bit. (Update: this is flawed, see https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045086033 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045086033 | IC_kwDOBm6k_c4-Sr9R | 9599 | 2022-02-18T19:47:43Z | 2022-02-18T19:51:11Z | OWNER | - https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv - https://til.simonwillison.net/-/asgi-scope/db/./db./table-..csv..csv Do both of those survive the round-trip to populate `raw_path` correctly? No! In both cases the `/./` bit goes missing. It looks like this might even be a client issue - `curl` shows me this: ``` ~ % curl -vv -i 'https://datasette.io/-/asgi-scope/db/./db./table-..csv..csv' * Trying 216.239.32.21:443... * Connected to datasette.io (216.239.32.21) port 443 (#0) * ALPN, offering http/1.1 * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: datasette.io * Server certificate: R3 * Server certificate: ISRG Root X1 > GET /-/asgi-scope/db/db./table-..csv..csv HTTP/1.1 ``` So `curl` decided to turn `/-/asgi-scope/db/./db./table` into `/-/asgi-scope/db/db./table` before even sending the request. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045082891 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045082891 | IC_kwDOBm6k_c4-SrML | 9599 | 2022-02-18T19:45:32Z | 2022-02-18T19:45:32Z | OWNER | ```pycon >>> dot_encode("/db/table-.csv.csv") './db./table-..csv..csv' >>> dot_decode('./db./table-..csv..csv') '/db/table-.csv.csv' ``` I worry that web servers might treat `./` in a special way though. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045095348 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045095348 | IC_kwDOBm6k_c4-SuO0 | 9599 | 2022-02-18T19:53:48Z | 2022-02-18T19:53:48Z | OWNER | > Ugh, one disadvantage I just spotted with this: Datasette already has a `/-/versions.json` convention where "system" URLs are namespaced under `/-/` - but that could be confused under this new scheme with the `-/` escaping sequence. > > And I've thought about adding `/db/-/special` and `/db/table/-/special` URLs in the past too. I don't think this matters. The new regex does indeed capture that kind of page: <img width="1052" alt="image" src="https://user-images.githubusercontent.com/9599/154752309-e1787755-3bdb-47c2-867c-7ac5fe65664d.png"> But Datasette goes through configured route regular expressions in order - so I can have the regex that captures `/db/-/special` routes listed before the one that captures tables and formats. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045099290 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045099290 | IC_kwDOBm6k_c4-SvMa | 9599 | 2022-02-18T19:56:18Z | 2022-02-18T19:56:30Z | OWNER | > ```python > def dash_encode(s): > return s.replace("-", "--").replace(".", "-.").replace("/", "-/") > > def dash_decode(s): > return s.replace("-/", "/").replace("-.", ".").replace("--", "-") > ``` I think **dash-encoding** (new name for this) is the right way forward here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045108611 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045108611 | IC_kwDOBm6k_c4-SxeD | 9599 | 2022-02-18T20:02:19Z | 2022-02-18T20:08:34Z | OWNER | One other potential variant: ```python def dash_encode(s): return s.replace("-", "-dash-").replace(".", "-dot-").replace("/", "-slash-") def dash_decode(s): return s.replace("-slash-", "/").replace("-dot-", ".").replace("-dash-", "-") ``` Except this has bugs - it doesn't round-trip safely, because it can get confused about things like `-dash-slash-` in terms of is that a `-dash-` or a `-slash-`? ```pycon >>> dash_encode("/db/table-.csv.csv") '-slash-db-slash-table-dash--dot-csv-dot-csv' >>> dash_decode('-slash-db-slash-table-dash--dot-csv-dot-csv') '/db/table-.csv.csv' >>> dash_encode('-slash-db-slash-table-dash--dot-csv-dot-csv') '-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv' >>> dash_decode('-dash-slash-dash-db-dash-slash-dash-table-dash-dash-dash--dash-dot-dash-csv-dash-dot-dash-csv') '-dash/dash-db-dash/dash-table-dash--dash.dash-csv-dash.dash-csv' ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045111309 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045111309 | IC_kwDOBm6k_c4-SyIN | 9599 | 2022-02-18T20:04:24Z | 2022-02-18T20:05:40Z | OWNER | This made me worry that my current `dash_decode()` implementation had unknown round-trip bugs, but thankfully this works OK: ```pycon >>> dash_encode("/db/table-.csv.csv") '-/db-/table---.csv-.csv' >>> dash_encode('-/db-/table---.csv-.csv') '---/db---/table-------.csv---.csv' >>> dash_decode('---/db---/table-------.csv---.csv') '-/db-/table---.csv-.csv' >>> dash_decode('-/db-/table---.csv-.csv') '/db/table-.csv.csv' ``` The regex still works against that double-encoded example too: <img width="1032" alt="image" src="https://user-images.githubusercontent.com/9599/154753916-b7d2159e-4284-4c92-ae61-110671fa320e.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045117304 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045117304 | IC_kwDOBm6k_c4-Szl4 | 9599 | 2022-02-18T20:09:22Z | 2022-02-18T20:09:22Z | OWNER | Adopting this could result in supporting database files with surprising characters in their filename too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1045269544 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1045269544 | IC_kwDOBm6k_c4-TYwo | 9599 | 2022-02-18T22:19:29Z | 2022-02-18T22:19:29Z | OWNER | Note that I've ruled out using `Accept: application/json` to return JSON because it turns out Cloudflare and potentially other CDNs ignore the `Vary: Accept` header entirely: - https://github.com/simonw/datasette/issues/1534 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1049114724 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1049114724 | IC_kwDOBm6k_c4-iDhk | 9599 | 2022-02-23T19:04:40Z | 2022-02-23T19:04:40Z | OWNER | I'm going to try dash encoding for table names (and row IDs) in a branch and see how I like it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1049124390 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1049124390 | IC_kwDOBm6k_c4-iF4m | 9599 | 2022-02-23T19:15:00Z | 2022-02-23T19:15:00Z | OWNER | I'll start by modifying this function: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/utils/__init__.py#L732-L749 Later I want to move this to the routing layer to split out `format` automatically, as seen in the regexes here: https://github.com/simonw/datasette/issues/1439#issuecomment-1045069481 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1439#issuecomment-1049126151 | https://api.github.com/repos/simonw/datasette/issues/1439 | 1049126151 | IC_kwDOBm6k_c4-iGUH | 9599 | 2022-02-23T19:17:01Z | 2022-02-23T19:17:01Z | OWNER | Actually the relevant code looks to be: https://github.com/simonw/datasette/blob/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b/datasette/views/base.py#L481-L498 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 973139047 | |
| https://github.com/simonw/datasette/issues/1641#issuecomment-1049879118 | https://api.github.com/repos/simonw/datasette/issues/1641 | 1049879118 | IC_kwDOBm6k_c4-k-JO | 536941 | 2022-02-24T13:49:26Z | 2022-02-24T13:49:26Z | NONE | maybe worth considering adding buttons for paren, asterisk, etc. under the input text box on mobile? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 1149310456 | |
| https://github.com/simonw/datasette/issues/260#issuecomment-1051473892 | https://api.github.com/repos/simonw/datasette/issues/260 | 1051473892 | IC_kwDOBm6k_c4-rDfk | 596279 | 2022-02-26T02:24:15Z | 2022-02-26T02:24:15Z | NONE | Is there already functionality that can be used to validate the `metadata.json` file? Is there a JSON Schema that defines it? Or a validation that's available via datasette with Python? We're working on [automatically building the metadata](https://github.com/catalyst-cooperative/pudl/pull/1479) in CI and when we deploy to cloud run, and it would be nice to be able to check whether the the metadata we're outputting is valid in our tests. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 323223872 | |
| https://github.com/simonw/datasette/issues/1405#issuecomment-888694144 | https://api.github.com/repos/simonw/datasette/issues/1405 | 888694144 | IC_kwDOBm6k_c40-GWA | 9599 | 2021-07-28T23:51:59Z | 2021-07-28T23:51:59Z | OWNER | https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/datasette/utils/__init__.py#L918-L926 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 955316250 | |
| https://github.com/simonw/datasette/issues/1405#issuecomment-888694261 | https://api.github.com/repos/simonw/datasette/issues/1405 | 888694261 | IC_kwDOBm6k_c40-GX1 | 9599 | 2021-07-28T23:52:21Z | 2021-07-28T23:52:21Z | OWNER | Document that it can raise a `BadMetadataError` exception. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 955316250 | |
| https://github.com/simonw/datasette/issues/1402#issuecomment-886968648 | https://api.github.com/repos/simonw/datasette/issues/1402 | 886968648 | IC_kwDOBm6k_c403hFI | 9599 | 2021-07-26T19:30:14Z | 2021-07-26T19:30:14Z | OWNER | I really like this idea. I was thinking it might make a good plugin, but there's not a great mechanism for plugins to inject extra `<head>` content at the moment - plus this actually feels like a reasonable feature for Datasette core itself. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 951185411 | |
| https://github.com/simonw/datasette/issues/1402#issuecomment-886969541 | https://api.github.com/repos/simonw/datasette/issues/1402 | 886969541 | IC_kwDOBm6k_c403hTF | 9599 | 2021-07-26T19:31:40Z | 2021-07-26T19:31:40Z | OWNER | Datasette could do a pretty good job of this by default, using `twitter:card` and `og:url` tags - like on https://til.simonwillison.net/jq/extracting-objects-recursively I could also provide a mechanism to customize these - in particular to add images of some sort. It feels like something that should tie in to the metadata mechanism. | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0} | 951185411 | |
| https://github.com/simonw/datasette/issues/1404#issuecomment-887095569 | https://api.github.com/repos/simonw/datasette/issues/1404 | 887095569 | IC_kwDOBm6k_c404AER | 9599 | 2021-07-26T23:27:07Z | 2021-07-26T23:27:07Z | OWNER | Updated documentation: https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/docs/plugin_hooks.rst#register-routes-datasette | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 953352015 | |
| https://github.com/simonw/datasette/issues/759#issuecomment-881125124 | https://api.github.com/repos/simonw/datasette/issues/759 | 881125124 | IC_kwDOBm6k_c40hOcE | 9599 | 2021-07-16T02:11:48Z | 2021-07-16T02:11:54Z | OWNER | I added `"searchmode": "raw"` as a supported option for table metadata in #1389 and released that in Datasette 0.58. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 612673948 | |
| https://github.com/simonw/datasette/issues/1394#issuecomment-881129149 | https://api.github.com/repos/simonw/datasette/issues/1394 | 881129149 | IC_kwDOBm6k_c40hPa9 | 9599 | 2021-07-16T02:23:32Z | 2021-07-16T02:23:32Z | OWNER | Wrote about this in the annotated release notes for 0.58: https://simonwillison.net/2021/Jul/16/datasette-058/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 944870799 | |
| https://github.com/simonw/datasette/issues/1231#issuecomment-881204782 | https://api.github.com/repos/simonw/datasette/issues/1231 | 881204782 | IC_kwDOBm6k_c40hh4u | 9599 | 2021-07-16T06:14:12Z | 2021-07-16T06:14:12Z | OWNER | Here's the traceback I got from `datasette-graphql` (annoyingly only running the tests in GitHub Actions CI - I've not been able to replicate on my laptop yet): ``` tests/test_utils.py . [100%] =================================== FAILURES =================================== _________________________ test_graphql_examples[path0] _________________________ ds = <datasette.app.Datasette object at 0x7f6b8b6f8fd0> path = PosixPath('/home/runner/work/datasette-graphql/datasette-graphql/examples/filters.md') @pytest.mark.asyncio @pytest.mark.parametrize( "path", (pathlib.Path(__file__).parent.parent / "examples").glob("*.md") ) async def test_graphql_examples(ds, path): content = path.read_text() query = graphql_re.search(content)[1] try: variables = variables_re.search(content)[1] except TypeError: variables = "{}" expected = json.loads(json_re.search(content)[1]) response = await ds.client.post( "/graphql", json={ "query": query, "variables": json.loads(variables), }, ) > assert response.status_code == 200, response.json() E AssertionError: {'data': {'repos_arraycontains': None, 'users_contains': None, 'users_date': None, 'users_endswith': None, ...}, 'erro...", 'path': ['users_gt']}, {'locations': [{'column': 5, 'line': 34}], 'message': "'rows'", 'path': ['users_gte']}, ...]} E assert 500 == 200 E + where 500 = <Response [500 Internal Server Error]>.status_code tests/test_graphql.py:142: AssertionError ----------------------------- Captured stderr call ----------------------------- table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists table databases already exists Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py", line 1171, in route_path response = await view(request, send) File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py", line 151, in view request, **request.scope["url_route"]["kwargs"] File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py", line 123, in dispatch_request await self.ds.refresh_schemas() File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py", line 338, in refresh_schemas await init_internal_db(internal_db) File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/utils/internal_db.py", line 16, in init_internal_db block=True, File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py", line 102, in execute_write return await self.execute_write_fn(_inner, block=block) File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py", line 118, in execute_write_fn raise result File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py", line 139, in _execute_writes result = task.fn(conn) File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py", line 100, in _inner return conn.execute(sql, params or []) sqlite3.OperationalError: table databases already exists ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 811367257 | |
| https://github.com/simonw/datasette/issues/1231#issuecomment-881204343 | https://api.github.com/repos/simonw/datasette/issues/1231 | 881204343 | IC_kwDOBm6k_c40hhx3 | 9599 | 2021-07-16T06:13:11Z | 2021-07-16T06:13:11Z | OWNER | This just broke the `datasette-graphql` test suite: https://github.com/simonw/datasette-graphql/issues/77 - I need to figure out a solution here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 811367257 | |
| https://github.com/simonw/datasette/issues/1231#issuecomment-881663968 | https://api.github.com/repos/simonw/datasette/issues/1231 | 881663968 | IC_kwDOBm6k_c40jR_g | 9599 | 2021-07-16T19:18:42Z | 2021-07-16T19:18:42Z | OWNER | The race condition happens inside this method - initially with the call to `await init_internal_db()`: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/app.py#L334-L359 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 811367257 | |
| https://github.com/simonw/datasette/issues/1231#issuecomment-881664408 | https://api.github.com/repos/simonw/datasette/issues/1231 | 881664408 | IC_kwDOBm6k_c40jSGY | 9599 | 2021-07-16T19:19:35Z | 2021-07-16T19:19:35Z | OWNER | The only place that calls `refresh_schemas()` is here: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/views/base.py#L120-L124 Ideally only one call to `refresh_schemas()` would be running at any one time. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 811367257 | |
| https://github.com/simonw/datasette/issues/1231#issuecomment-881665383 | https://api.github.com/repos/simonw/datasette/issues/1231 | 881665383 | IC_kwDOBm6k_c40jSVn | 9599 | 2021-07-16T19:21:35Z | 2021-07-16T19:21:35Z | OWNER | https://stackoverflow.com/a/25799871/6083 has a good example of using `asyncio.Lock()`: ```python stuff_lock = asyncio.Lock() async def get_stuff(url): async with stuff_lock: if url in cache: return cache[url] stuff = await aiohttp.request('GET', url) cache[url] = stuff return stuff ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 811367257 | |
| https://github.com/simonw/datasette/issues/1231#issuecomment-881671706 | https://api.github.com/repos/simonw/datasette/issues/1231 | 881671706 | IC_kwDOBm6k_c40jT4a | 9599 | 2021-07-16T19:32:05Z | 2021-07-16T19:32:05Z | OWNER | The test suite passes with that change. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 811367257 | |
| https://github.com/simonw/datasette/issues/1231#issuecomment-881668759 | https://api.github.com/repos/simonw/datasette/issues/1231 | 881668759 | IC_kwDOBm6k_c40jTKX | 9599 | 2021-07-16T19:27:46Z | 2021-07-16T19:27:46Z | OWNER | Second attempt at this: ```diff diff --git a/datasette/app.py b/datasette/app.py index 5976d8b..5f348cb 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -224,6 +224,7 @@ class Datasette: self.inspect_data = inspect_data self.immutables = set(immutables or []) self.databases = collections.OrderedDict() + self._refresh_schemas_lock = asyncio.Lock() self.crossdb = crossdb if memory or crossdb or not self.files: self.add_database(Database(self, is_memory=True), name="_memory") @@ -332,6 +333,12 @@ class Datasette: self.client = DatasetteClient(self) async def refresh_schemas(self): + if self._refresh_schemas_lock.locked(): + return + async with self._refresh_schemas_lock: + await self._refresh_schemas() + + async def _refresh_schemas(self): internal_db = self.databases["_internal"] if not self.internal_db_created: await init_internal_db(internal_db) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 811367257 | |
| https://github.com/simonw/datasette/issues/1231#issuecomment-881674857 | https://api.github.com/repos/simonw/datasette/issues/1231 | 881674857 | IC_kwDOBm6k_c40jUpp | 9599 | 2021-07-16T19:38:39Z | 2021-07-16T19:38:39Z | OWNER | I can't replicate the race condition locally with or without this patch. I'm going to push the commit and then test the CI run from `datasette-graphql` that was failing against it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 811367257 | |
| https://github.com/simonw/datasette/issues/1231#issuecomment-881677620 | https://api.github.com/repos/simonw/datasette/issues/1231 | 881677620 | IC_kwDOBm6k_c40jVU0 | 9599 | 2021-07-16T19:44:12Z | 2021-07-16T19:44:12Z | OWNER | That fixed the race condition in the `datasette-graphql` tests, which is the only place that I've been able to successfully replicate this. I'm going to land this change. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 811367257 | |
| https://github.com/simonw/datasette/issues/1396#issuecomment-881686662 | https://api.github.com/repos/simonw/datasette/issues/1396 | 881686662 | IC_kwDOBm6k_c40jXiG | 9599 | 2021-07-16T20:02:44Z | 2021-07-16T20:02:44Z | OWNER | Confirmed fixed: 0.58.1 was successfully published to Docker Hub in https://github.com/simonw/datasette/runs/3089447346 and the `latest` tag on https://hub.docker.com/r/datasetteproject/datasette/tags was updated. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 944903881 | |
| https://github.com/simonw/datasette/issues/123#issuecomment-882096402 | https://api.github.com/repos/simonw/datasette/issues/123 | 882096402 | IC_kwDOBm6k_c40k7kS | 921217 | 2021-07-18T18:07:29Z | 2021-07-18T18:07:29Z | NONE | I also love the idea for this feature and wonder if it could work without having to download the whole database into memory at once if it's a rather large db. Obviously this could be slower but could have many use cases. My comment is partially inspired by this post about streaming sqlite dbs from github pages or such https://news.ycombinator.com/item?id=27016630 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 275125561 | |
| https://github.com/simonw/datasette/issues/1199#issuecomment-881932880 | https://api.github.com/repos/simonw/datasette/issues/1199 | 881932880 | IC_kwDOBm6k_c40kTpQ | 9599 | 2021-07-17T17:39:17Z | 2021-07-17T17:39:17Z | OWNER | I asked about optimizing performance on the SQLite forum and this came up as a suggestion: https://sqlite.org/forum/forumpost/9a6b9ae8e2048c8b?t=c I can start by trying this: PRAGMA mmap_size=268435456; | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 792652391 | |
| https://github.com/simonw/datasette/issues/123#issuecomment-882138084 | https://api.github.com/repos/simonw/datasette/issues/123 | 882138084 | IC_kwDOBm6k_c40lFvk | 9599 | 2021-07-19T00:04:31Z | 2021-07-19T00:04:31Z | OWNER | I've been thinking more about this one today too. An extension of this (touched on in #417, Datasette Library) would be to support pointing Datasette at a directory and having it automatically load any CSV files it finds anywhere in that folder or its descendants - either loading them fully, or providing a UI that allows users to select a file to open it in Datasette. For larger files I think the right thing to do is import them into an on-disk SQLite database, which is limited only by available disk space. For smaller files loading them into an in-memory database should work fine. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 275125561 | |
| https://github.com/simonw/datasette/pull/1400#issuecomment-882542519 | https://api.github.com/repos/simonw/datasette/issues/1400 | 882542519 | IC_kwDOBm6k_c40moe3 | 22429695 | 2021-07-19T13:20:52Z | 2021-07-19T13:20:52Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1400](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e95c685) into [main](https://codecov.io/gh/simonw/datasette/commit/c73af5dd72305f6a01ea94a2c76d52e5e26de38b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c73af5d) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1400 +/- ## ======================================= Coverage 91.62% 91.62% ======================================= Files 34 34 Lines 4371 4371 ======================================= Hits 4005 4005 Misses 366 366 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c73af5d...e95c685](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 947640902 | |
| https://github.com/simonw/datasette/issues/1401#issuecomment-884910320 | https://api.github.com/repos/simonw/datasette/issues/1401 | 884910320 | IC_kwDOBm6k_c40vqjw | 536941 | 2021-07-22T13:26:01Z | 2021-07-22T13:26:01Z | NONE | ordered lists didn't work either, btw | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 950664971 | |
| https://github.com/simonw/datasette/issues/1445#issuecomment-904024939 | https://api.github.com/repos/simonw/datasette/issues/1445 | 904024939 | IC_kwDOBm6k_c414lNr | 9599 | 2021-08-23T18:52:35Z | 2021-08-23T18:52:35Z | OWNER | The downside of the current implementation of this trick is that it only works for exact LIKE partial matches in a specific table - if you search for `dog cat` and `dog` appears in `title` but `cat` appears in `description` you won't get back that result. I think that's fine though. If you want more advanced search there are other mechanisms you can use. This is meant to be a very quick and dirty starting point for exploring a brand new table. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 977323133 | |
| https://github.com/simonw/datasette/issues/1445#issuecomment-904026253 | https://api.github.com/repos/simonw/datasette/issues/1445 | 904026253 | IC_kwDOBm6k_c414liN | 9599 | 2021-08-23T18:54:49Z | 2021-08-23T18:54:49Z | OWNER | The bigger problem here is UI design. This feels like a pretty niche requirement to me, so adding a prominent search box to the table page (which already has the filters interface, plus the full-text search box for tables that have FTS configured) feels untidy. I could tuck it away in the table cog menu, but that's a weird place for something like this to live. Maybe add it as a new type of filter? Filters apply to specific columns though, so this would be the first filter that applied to _all_ columns - which doesn't really fit the existing filter interface very well. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 977323133 | |
| https://github.com/simonw/datasette/issues/1445#issuecomment-904027166 | https://api.github.com/repos/simonw/datasette/issues/1445 | 904027166 | IC_kwDOBm6k_c414lwe | 9599 | 2021-08-23T18:56:20Z | 2021-08-23T18:56:20Z | OWNER | A related but potentially even more useful ability would be running a search across every column of every table in a whole database. For anything less than a few 100MB this could be incredibly useful. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 977323133 | |
| https://github.com/simonw/datasette/issues/1445#issuecomment-904036200 | https://api.github.com/repos/simonw/datasette/issues/1445 | 904036200 | IC_kwDOBm6k_c414n9o | 9599 | 2021-08-23T19:08:54Z | 2021-08-23T19:08:54Z | OWNER | Figured out a query for searching across every column in every table! https://til.simonwillison.net/datasette/search-all-columns-trick#user-content-same-trick-for-the-entire-database ```sql with tables as ( select name as table_name from sqlite_master where type = 'table' ), queries as ( select 'select ''' || tables.table_name || ''' as _table, rowid from "' || tables.table_name || '" where ' || group_concat( '"' || name || '" like ''%'' || :search || ''%''', ' or ' ) as query from pragma_table_info(tables.table_name), tables group by tables.table_name ) select group_concat(query, ' union all ') from queries ``` The SQL query this generates for larger databases is _extremely_ long - but it does seem to work for smaller databases. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 977323133 | |
| https://github.com/simonw/datasette/issues/1445#issuecomment-904037087 | https://api.github.com/repos/simonw/datasette/issues/1445 | 904037087 | IC_kwDOBm6k_c414oLf | 9599 | 2021-08-23T19:10:17Z | 2021-08-23T19:10:17Z | OWNER | Rather than trying to run that monstrosity in a single `union all` query, a better approach may be to use `fetch()` requests as seen in https://datasette.io/plugins/datasette-search-all | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 977323133 | |
| https://github.com/simonw/datasette/issues/1446#issuecomment-904866495 | https://api.github.com/repos/simonw/datasette/issues/1446 | 904866495 | IC_kwDOBm6k_c417yq_ | 9599 | 2021-08-24T18:13:49Z | 2021-08-24T18:13:49Z | OWNER | OK, now the following optional CSS gives us a sticky footer: ```css html, body { height: 100%; } body { display: flex; flex-direction: column; } .not-footer { flex: 1 0 auto; } footer { flex-shrink: 0; } ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 978357984 | |
| https://github.com/simonw/datasette/issues/1446#issuecomment-904954530 | https://api.github.com/repos/simonw/datasette/issues/1446 | 904954530 | IC_kwDOBm6k_c418IKi | 9599 | 2021-08-24T20:32:47Z | 2021-08-24T20:32:47Z | OWNER | Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 978357984 | |
| https://github.com/simonw/datasette/issues/859#issuecomment-904982056 | https://api.github.com/repos/simonw/datasette/issues/859 | 904982056 | IC_kwDOBm6k_c418O4o | 2670795 | 2021-08-24T21:15:04Z | 2021-08-24T21:15:30Z | CONTRIBUTOR | I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 642572841 | |
| https://github.com/simonw/datasette/pull/1447#issuecomment-905097468 | https://api.github.com/repos/simonw/datasette/issues/1447 | 905097468 | IC_kwDOBm6k_c418rD8 | 9599 | 2021-08-25T01:28:53Z | 2021-08-25T01:28:53Z | OWNER | Good catch, thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 978614898 | |
| https://github.com/simonw/datasette/issues/1405#issuecomment-889525741 | https://api.github.com/repos/simonw/datasette/issues/1405 | 889525741 | IC_kwDOBm6k_c41BRXt | 9599 | 2021-07-29T23:33:30Z | 2021-07-29T23:33:30Z | OWNER | New documentation section for `datasette.utils` is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 955316250 | |
| https://github.com/simonw/datasette/issues/1241#issuecomment-889539227 | https://api.github.com/repos/simonw/datasette/issues/1241 | 889539227 | IC_kwDOBm6k_c41BUqb | 9599 | 2021-07-30T00:15:26Z | 2021-07-30T00:15:26Z | OWNER | One possible treatment: <img width="442" alt="fixtures__sortable__201_rows" src="https://user-images.githubusercontent.com/9599/127581105-dca1193c-1bdd-491e-89d1-ac3136c05c3b.png"> ```html <style> a.action-button { display: inline-block; border: 1.5px solid #666; border-radius: 0.5em; background-color: #ffffff; color: #666; padding: 0.1em 0.8em; text-decoration: none; margin-right: 0.3em; font-size: 0.85em; } </style> <p> {% if query.sql and allow_execute_sql %} <a class="action-button" title="{{ query.sql }}" href="{{ urls.database(database) }}?{{ {'sql': query.sql}|urlencode|safe }}{% if query.params %}&{{ query.params|urlencode|safe }}{% endif %}"> View and edit SQL </a> {% endif %} <a href="#" class="action-button">Copy and share link</a> </p> ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 814595021 | |
| https://github.com/simonw/datasette/issues/1406#issuecomment-889548536 | https://api.github.com/repos/simonw/datasette/issues/1406 | 889548536 | IC_kwDOBm6k_c41BW74 | 9599 | 2021-07-30T00:43:47Z | 2021-07-30T00:43:47Z | OWNER | Still couldn't replicate on my laptop. On a hunch, I'm going to add `@pytest.mark.serial` to every test that uses `runner.isolated_filesystem()`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 956303470 | |
| https://github.com/simonw/datasette/issues/1406#issuecomment-889547142 | https://api.github.com/repos/simonw/datasette/issues/1406 | 889547142 | IC_kwDOBm6k_c41BWmG | 9599 | 2021-07-30T00:39:49Z | 2021-07-30T00:39:49Z | OWNER | It happens in CI but not on my laptop. I think I need to run the tests on my laptop like this: https://github.com/simonw/datasette/blob/121e10c29c5b412fddf0326939f1fe46c3ad9d4a/.github/workflows/test.yml#L27-L30 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 956303470 | |
| https://github.com/simonw/datasette/issues/1406#issuecomment-889550391 | https://api.github.com/repos/simonw/datasette/issues/1406 | 889550391 | IC_kwDOBm6k_c41BXY3 | 9599 | 2021-07-30T00:49:31Z | 2021-07-30T00:49:31Z | OWNER | That fixed it. My hunch is that Click's `runner.isolated_filesystem()` mechanism doesn't play well with `pytest-xdist`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 956303470 | |
| https://github.com/simonw/datasette/issues/1406#issuecomment-889553052 | https://api.github.com/repos/simonw/datasette/issues/1406 | 889553052 | IC_kwDOBm6k_c41BYCc | 9599 | 2021-07-30T00:58:43Z | 2021-07-30T00:58:43Z | OWNER | Tests are still failing in the job that calculates coverage. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 956303470 | |
| https://github.com/simonw/datasette/issues/1406#issuecomment-889555977 | https://api.github.com/repos/simonw/datasette/issues/1406 | 889555977 | IC_kwDOBm6k_c41BYwJ | 9599 | 2021-07-30T01:06:57Z | 2021-07-30T01:06:57Z | OWNER | Looking at the source code in Click for `isolated_filesystem()`: https://github.com/pallets/click/blob/9da166957f5848b641231d485467f6140bca2bc0/src/click/testing.py#L450-L468 ```python @contextlib.contextmanager def isolated_filesystem( self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None ) -> t.Iterator[str]: """A context manager that creates a temporary directory and changes the current working directory to it. This isolates tests that affect the contents of the CWD to prevent them from interfering with each other. :param temp_dir: Create the temporary directory under this directory. If given, the created directory is not removed when exiting. .. versionchanged:: 8.0 Added the ``temp_dir`` parameter. """ cwd = os.getcwd() t = tempfile.mkdtemp(dir=temp_dir) os.chdir(t) ``` How about if I pass in that optional `temp_dir` as a temp directory created using the `pytest-xdist` aware pytest mechanisms: https://docs.pytest.org/en/6.2.x/tmpdir.html | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 956303470 | |
| https://github.com/simonw/datasette/issues/1398#issuecomment-889599513 | https://api.github.com/repos/simonw/datasette/issues/1398 | 889599513 | IC_kwDOBm6k_c41BjYZ | 192984 | 2021-07-30T03:21:49Z | 2021-07-30T03:21:49Z | NONE | Does the library support this now? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 947044667 | |
| https://github.com/simonw/datasette/issues/1406#issuecomment-890259755 | https://api.github.com/repos/simonw/datasette/issues/1406 | 890259755 | IC_kwDOBm6k_c41EEkr | 9599 | 2021-07-31T00:04:54Z | 2021-07-31T00:04:54Z | OWNER | STILL failing. I'm going to try removing all instances of `isolated_filesystem()` in favour of a different pattern using pytest temporary files, then see if I can get that to work without the serial hack. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 956303470 | |
| https://github.com/simonw/datasette/issues/1407#issuecomment-890388200 | https://api.github.com/repos/simonw/datasette/issues/1407 | 890388200 | IC_kwDOBm6k_c41Ej7o | 9599 | 2021-07-31T18:38:41Z | 2021-07-31T18:38:41Z | OWNER | The `path` variable there looked like this: `/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-696/popen-gw0/uds0/datasette.sock` I think what's happening here is that `pytest-xdist` causes `tmp_path_factory.mktemp("uds")` to create significantly longer paths, which in this case is breaking some limit. So for this code to work with `pytest-xdist` I need to make sure the random path to `datasette.sock` is shorter. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957298475 | |
| https://github.com/simonw/datasette/issues/1407#issuecomment-890388656 | https://api.github.com/repos/simonw/datasette/issues/1407 | 890388656 | IC_kwDOBm6k_c41EkCw | 9599 | 2021-07-31T18:42:41Z | 2021-07-31T18:42:41Z | OWNER | I'll try `tempfile.gettempdir()` - on macOS it returns something like `'/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T'` which is still long but hopefully not too long. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957298475 | |
| https://github.com/simonw/datasette/issues/1406#issuecomment-890390198 | https://api.github.com/repos/simonw/datasette/issues/1406 | 890390198 | IC_kwDOBm6k_c41Eka2 | 9599 | 2021-07-31T18:55:33Z | 2021-07-31T18:55:33Z | OWNER | To clarify: the core problem here is that an error is thrown any time you call `os.getcwd()` but the directory you are currently in has been deleted. `runner.isolated_filesystem()` assumes that the current directory in has not been deleted. But the various temporary directory utilities in `pytest` work by creating directories and then deleting them. Maybe there's a larger problem here that I play a bit fast and loose with `os.chdir()` in both the test suite and in various lines of code in Datasette itself (in particular in the publish commands)? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 956303470 | |
| https://github.com/simonw/datasette/issues/1406#issuecomment-890390342 | https://api.github.com/repos/simonw/datasette/issues/1406 | 890390342 | IC_kwDOBm6k_c41EkdG | 9599 | 2021-07-31T18:56:35Z | 2021-07-31T18:56:35Z | OWNER | But... I've lost enough time to this already, and removing `runner.isolated_filesystem()` has the tests passing again. So I'm not going to work on this any more. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 956303470 | |
| https://github.com/simonw/datasette/issues/1408#issuecomment-890390495 | https://api.github.com/repos/simonw/datasette/issues/1408 | 890390495 | IC_kwDOBm6k_c41Ekff | 9599 | 2021-07-31T18:57:39Z | 2021-07-31T18:57:39Z | OWNER | Opening this issue as an optional follow-up to the work I did in #1406. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957302085 | |
| https://github.com/simonw/datasette/issues/1408#issuecomment-890390845 | https://api.github.com/repos/simonw/datasette/issues/1408 | 890390845 | IC_kwDOBm6k_c41Ekk9 | 9599 | 2021-07-31T19:00:32Z | 2021-07-31T19:00:32Z | OWNER | When I revisit this I can also look at dropping the `@pytest.mark.serial` hack, and maybe the `restore_working_directory()` fixture hack too: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/pytest.ini#L9-L10 https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/tests/conftest.py#L62-L75 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957302085 | |
| https://github.com/simonw/datasette/issues/1409#issuecomment-890400059 | https://api.github.com/repos/simonw/datasette/issues/1409 | 890400059 | IC_kwDOBm6k_c41Em07 | 9599 | 2021-07-31T20:21:51Z | 2021-07-31T20:21:51Z | OWNER | One of these two options: - `--setting default_allow_sql off` - `--setting allow_sql_default off` Existing settings from https://docs.datasette.io/en/0.58.1/settings.html with similar names that I need to be consistent with: - `default_page_size` - `allow_facet` - `default_facet_size` - `allow_download` - `default_cache_ttl` - `default_cache_ttl_hashed` - `allow_csv_stream` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957310278 | |
| https://github.com/simonw/datasette/issues/1409#issuecomment-890400121 | https://api.github.com/repos/simonw/datasette/issues/1409 | 890400121 | IC_kwDOBm6k_c41Em15 | 9599 | 2021-07-31T20:22:21Z | 2021-07-31T20:23:34Z | OWNER | I think `default_allow_sql` is more consistent with the current naming conventions, because both `allow` and `default` are used as prefixes at the moment but neither of them are ever used as a suffix. Plus `default_allow_sql off` makes sense to me but `allow_default_sql off` does not - what is "default SQL"? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957310278 | |
| https://github.com/simonw/datasette/issues/1409#issuecomment-890400425 | https://api.github.com/repos/simonw/datasette/issues/1409 | 890400425 | IC_kwDOBm6k_c41Em6p | 9599 | 2021-07-31T20:25:16Z | 2021-07-31T20:26:25Z | OWNER | If I was prone to over-thinking (which I am) I'd note that `allow_facet` and `allow_download` and `allow_csv_stream` are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent. So maybe there's a future task where I introduce those as both permissions and metadata `"allow_x"` blocks, then rename the settings themselves to be called `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`. If I was going to do that I should get it in before Datasette 1.0. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957310278 | |
| https://github.com/simonw/datasette/issues/1409#issuecomment-890397124 | https://api.github.com/repos/simonw/datasette/issues/1409 | 890397124 | IC_kwDOBm6k_c41EmHE | 9599 | 2021-07-31T19:51:10Z | 2021-07-31T19:51:10Z | OWNER | I think I may like `disable_sql` better. Some options: - `--setting allow_sql off` (consistent with `allow_facet` and `allow_download` and `allow_csv_stream` - all which default to `on` already) - `--setting disable_sql on` - `--setting disable_custom_sql on` The existence of three `allow_*` settings does make a strong argument for staying consistent with that. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957310278 | |
| https://github.com/simonw/datasette/issues/1409#issuecomment-890397169 | https://api.github.com/repos/simonw/datasette/issues/1409 | 890397169 | IC_kwDOBm6k_c41EmHx | 9599 | 2021-07-31T19:51:35Z | 2021-07-31T19:51:35Z | OWNER | I'm going to stick with `--setting allow_sql off`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957310278 | |
| https://github.com/simonw/datasette/issues/1409#issuecomment-890397261 | https://api.github.com/repos/simonw/datasette/issues/1409 | 890397261 | IC_kwDOBm6k_c41EmJN | 9599 | 2021-07-31T19:52:25Z | 2021-07-31T19:52:25Z | OWNER | I think I can make this modification by teaching the default permissions code here to take the `allow_sql` setting into account: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/datasette/default_permissions.py#L38-L45 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957310278 | |
| https://github.com/simonw/datasette/issues/1409#issuecomment-890397652 | https://api.github.com/repos/simonw/datasette/issues/1409 | 890397652 | IC_kwDOBm6k_c41EmPU | 9599 | 2021-07-31T19:56:48Z | 2021-07-31T19:56:48Z | OWNER | The other option would be to use the setting to pick the `default=` argument when calling `self.ds.permission_allowed( request.actor, "execute-sql", resource=database, default=True)`. The problem with that is that there are actually a few different places which perform that check, so changing all of them raises the risk of missing one in the future: https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L436-L444 https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L964-L966 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L220-L221 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L343-L345 https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L134-L136 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957310278 | |
| https://github.com/simonw/datasette/issues/1409#issuecomment-890397753 | https://api.github.com/repos/simonw/datasette/issues/1409 | 890397753 | IC_kwDOBm6k_c41EmQ5 | 9599 | 2021-07-31T19:57:56Z | 2021-07-31T19:57:56Z | OWNER | I think the correct solution is for the default permissions logic to take the `allow_sql` setting into account, and to return `False` if that setting is set to `off` AND the current actor fails the `actor_matches_allow` checks. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957310278 | |
| https://github.com/simonw/datasette/issues/1409#issuecomment-890399806 | https://api.github.com/repos/simonw/datasette/issues/1409 | 890399806 | IC_kwDOBm6k_c41Emw- | 9599 | 2021-07-31T20:18:46Z | 2021-07-31T20:18:46Z | OWNER | My rationale for removing it: https://github.com/simonw/datasette/issues/813#issuecomment-640916290 > Naming problem: Datasette already has a config option with this name: > > $ datasette serve data.db --config allow_sql:1 > > https://datasette.readthedocs.io/en/stable/config.html#allow-sql > > It's confusing to have two things called `allow_sql` that do slightly different things. > > I could retire the `--config allow_sql:0` option entirely, since the new `metadata.json` mechanism can be used to achieve the exact same thing. > > I'm going to do that. This is true. The `"allow_sql"` permissions block in `metadata.json` does indeed have a name that is easily confused with `--setting allow_sql off`. So I definitely need to pick a different name from the setting. `--setting default_allow_sql off` is a good option here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957310278 | |
| https://github.com/simonw/datasette/issues/1411#issuecomment-890441844 | https://api.github.com/repos/simonw/datasette/issues/1411 | 890441844 | IC_kwDOBm6k_c41ExB0 | 9599 | 2021-08-01T03:27:30Z | 2021-08-01T03:27:30Z | OWNER | Confirmed: https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1 no longer exhibits the bug. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 957345476 | |
| https://github.com/simonw/datasette/issues/1227#issuecomment-891352132 | https://api.github.com/repos/simonw/datasette/issues/1227 | 891352132 | IC_kwDOBm6k_c41IPRE | 9599 | 2021-08-02T21:38:39Z | 2021-08-02T21:38:39Z | OWNER | Relevant TIL: https://til.simonwillison.net/vscode/vs-code-regular-expressions | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 810394616 | |
| https://github.com/simonw/datasette/issues/1417#issuecomment-891979858 | https://api.github.com/repos/simonw/datasette/issues/1417 | 891979858 | IC_kwDOBm6k_c41KohS | 9599 | 2021-08-03T16:15:30Z | 2021-08-03T16:15:30Z | OWNER | Docs: https://pypi.org/project/codespell/ There's a `codespell --ignore-words=FILE` option for ignoring words. I don't have any that need ignoring yet but I'm going to add that file anyway, that way I can have codespell be a failing test but still provide a way to work around it if it incorrectly flags a correct word. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 959278472 | |
| https://github.com/simonw/datasette/pull/1418#issuecomment-891984359 | https://api.github.com/repos/simonw/datasette/issues/1418 | 891984359 | IC_kwDOBm6k_c41Kpnn | 9599 | 2021-08-03T16:21:56Z | 2021-08-03T16:21:56Z | OWNER | Failed with: ``` docs/authentication.rst:63: perfom ==> perform docs/authentication.rst:76: perfom ==> perform docs/changelog.rst:429: repsonse ==> response docs/changelog.rst:503: permissons ==> permissions docs/changelog.rst:717: compatibilty ==> compatibility docs/changelog.rst:1172: browseable ==> browsable docs/deploying.rst:191: similiar ==> similar docs/internals.rst:434: Respons ==> Response, respond docs/internals.rst:440: Respons ==> Response, respond docs/internals.rst:717: tha ==> than, that, the docs/performance.rst:42: databse ==> database docs/plugin_hooks.rst:667: utilites ==> utilities docs/publish.rst:168: countainer ==> container docs/settings.rst:352: inalid ==> invalid docs/sql_queries.rst:406: preceeded ==> preceded, proceeded ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 959284434 | |
| https://github.com/simonw/datasette/pull/1418#issuecomment-891987129 | https://api.github.com/repos/simonw/datasette/issues/1418 | 891987129 | IC_kwDOBm6k_c41KqS5 | 22429695 | 2021-08-03T16:26:01Z | 2021-08-03T16:32:20Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1418](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (532170f) into [main](https://codecov.io/gh/simonw/datasette/commit/54b6e96ee8aa553b6671e341a1944f93f3fb89c3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (54b6e96) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1418 +/- ## ======================================= Coverage 91.64% 91.64% ======================================= Files 34 34 Lines 4382 4382 ======================================= Hits 4016 4016 Misses 366 366 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [54b6e96...532170f](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 959284434 | |
| https://github.com/simonw/datasette/issues/1419#issuecomment-892276385 | https://api.github.com/repos/simonw/datasette/issues/1419 | 892276385 | IC_kwDOBm6k_c41Lw6h | 536941 | 2021-08-04T00:58:49Z | 2021-08-04T00:58:49Z | NONE | yes, [filter clause on aggregate queries were added to sqlite3 in 3.30](https://www.sqlite.org/releaselog/3_30_1.html) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 959710008 | |
| https://github.com/simonw/datasette/issues/1420#issuecomment-892365639 | https://api.github.com/repos/simonw/datasette/issues/1420 | 892365639 | IC_kwDOBm6k_c41MGtH | 9599 | 2021-08-04T05:05:07Z | 2021-08-04T05:05:07Z | OWNER | https://github.com/simonw/datasette/blob/cd8b7bee8fb5c1cdce7c8dbfeb0166011abc72c6/datasette/publish/cloudrun.py#L153-L158 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 959898166 | |
| https://github.com/simonw/datasette/issues/1420#issuecomment-892372509 | https://api.github.com/repos/simonw/datasette/issues/1420 | 892372509 | IC_kwDOBm6k_c41MIYd | 9599 | 2021-08-04T05:22:29Z | 2021-08-04T05:22:29Z | OWNER | Testing this manually with: datasette publish cloudrun fixtures.db --memory 8G --cpu 4 \ --service fixtures-over-provisioned-issue-1420 --install datasette-psutil And for comparison: datasette publish cloudrun fixtures.db --service fixtures-default-issue-1420 \ --install datasette-psutil | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 959898166 | |
| https://github.com/simonw/datasette/issues/1420#issuecomment-892374253 | https://api.github.com/repos/simonw/datasette/issues/1420 | 892374253 | IC_kwDOBm6k_c41MIzt | 9599 | 2021-08-04T05:27:21Z | 2021-08-04T05:29:59Z | OWNER | I'll delete these services shortly afterwards. Right now: https://fixtures-over-provisioned-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil (deployed with `--memory 8G --cpu 4`) returns: ``` process.memory_info() pmem(rss=60456960, vms=518930432, shared=0, text=0, lib=0, data=0, dirty=0) ... psutil.cpu_times(True) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) psutil.virtual_memory() svmem(total=2147483648, available=2092531712, percent=2.6, used=33103872, free=2092531712, active=44130304, inactive=10792960, buffers=0, cached=21848064, shared=262144, slab=0) ``` https://fixtures-default-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil returns: ``` process.memory_info() pmem(rss=49324032, vms=140595200, shared=0, text=0, lib=0, data=0, dirty=0) ... psutil.cpu_times(True) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0) psutil.virtual_memory() svmem(total=2147483648, available=2091188224, percent=2.6, used=40071168, free=2091188224, active=41586688, inactive=7983104, buffers=0, cached=16224256, shared=262144, slab=0) ``` These numbers are different enough that I assume this works as advertised. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 959898166 | |
| https://github.com/simonw/datasette/issues/1420#issuecomment-892376353 | https://api.github.com/repos/simonw/datasette/issues/1420 | 892376353 | IC_kwDOBm6k_c41MJUh | 9599 | 2021-08-04T05:33:12Z | 2021-08-04T05:33:12Z | OWNER | In the Cloud Run console (before I deleted these services) when I click "Edit and deploy new revision" I see this for the default one: <img width="551" alt="Deploy_new_revision_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/128127309-7ec2064f-b107-4d99-b630-ee58734c4a72.png"> And this for the big one: <img width="546" alt="Deploy_new_revision_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform" src="https://user-images.githubusercontent.com/9599/128127366-7cdcb245-2e83-4b88-9d4f-0c595a486139.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 959898166 | |
| https://github.com/simonw/datasette/issues/1420#issuecomment-893079520 | https://api.github.com/repos/simonw/datasette/issues/1420 | 893079520 | IC_kwDOBm6k_c41O0_g | 9599 | 2021-08-05T00:54:59Z | 2021-08-05T00:54:59Z | OWNER | Just saw this error: `ERROR: (gcloud.run.deploy) The `--cpu` flag is not supported on the fully managed version of Cloud Run. Specify `--platform gke` or run `gcloud config set run/platform gke` to work with Cloud Run for Anthos deployed on Google Cloud.` Which is weird because I managed to run this successfully the other day. Maybe a region difference thing? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 959898166 | |
| https://github.com/simonw/datasette/issues/1419#issuecomment-893114612 | https://api.github.com/repos/simonw/datasette/issues/1419 | 893114612 | IC_kwDOBm6k_c41O9j0 | 536941 | 2021-08-05T02:29:06Z | 2021-08-05T02:29:06Z | NONE | there's a lot of complexity here, that's probably not worth addressing. i got what i needed by patching the dockerfile that cloudrun uses to install a newer version of sqlite. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 959710008 | |
| https://github.com/simonw/datasette/issues/1422#issuecomment-893122356 | https://api.github.com/repos/simonw/datasette/issues/1422 | 893122356 | IC_kwDOBm6k_c41O_c0 | 9599 | 2021-08-05T02:52:31Z | 2021-08-05T02:52:44Z | OWNER | If you do this it should still be possible to view the SQL - which means we need a new parameter. I propose `?_show_sql=1` to over-ride the hidden default. I think the configuration should use `hide_sql: true` - looking like this: ```yaml databases: fixtures: queries: neighborhood_search: hide_sql: true sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood title: Search neighborhoods ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 961367843 | |
| https://github.com/simonw/datasette/issues/1422#issuecomment-893131703 | https://api.github.com/repos/simonw/datasette/issues/1422 | 893131703 | IC_kwDOBm6k_c41PBu3 | 9599 | 2021-08-05T03:16:46Z | 2021-08-05T03:16:46Z | OWNER | The logic for this is a little bit fiddly, due to the need to switch to using `?_show_sql=1` on the link depending on the context. - If metadata says hide and there's no query string, hide and link to `?_show_sql=1` - If metadata says hide but query string says `?_show_sql=1`, show and have hide link linking to URL without `?_show_sql=1` - Otherwise, show and link to `?_hide_sql=1` - ... or if that query string is there then hide and link to URL without `?_hide_sql=1` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 961367843 | |
| https://github.com/simonw/datasette/issues/1419#issuecomment-893133496 | https://api.github.com/repos/simonw/datasette/issues/1419 | 893133496 | IC_kwDOBm6k_c41PCK4 | 9599 | 2021-08-05T03:22:44Z | 2021-08-05T03:22:44Z | OWNER | I ran into this exact same problem today! I only just learned how to use filter on aggregates: https://til.simonwillison.net/sqlite/sqlite-aggregate-filter-clauses A workaround I used is to add this to the deploy command: datasette publish cloudrun ... --install=pysqlite3-binary This will install the https://pypi.org/project/pysqlite3-binary for package which bundles a more recent SQLite version. | {"total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 959710008 | |
| https://github.com/simonw/datasette/issues/1423#issuecomment-893996604 | https://api.github.com/repos/simonw/datasette/issues/1423 | 893996604 | IC_kwDOBm6k_c41SU48 | 9599 | 2021-08-06T04:43:07Z | 2021-08-06T04:43:37Z | OWNER | Problem: on a page which doesn't have quite enough facet values to trigger the display of the "..." link that links to `?_facet_size=max` the user would still have to manually count the values - up to 30 by default. So maybe the count should always be shown, perhaps as a non-bold light colored number? I could even hide it in a non-discoverable tooltip. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 962391325 | |
| https://github.com/simonw/datasette/issues/1423#issuecomment-894452990 | https://api.github.com/repos/simonw/datasette/issues/1423 | 894452990 | IC_kwDOBm6k_c41UET- | 9599 | 2021-08-06T18:49:37Z | 2021-08-06T18:49:37Z | OWNER | Could display them always, like this: <img width="1299" alt="fixtures__compound_three_primary_keys__1_001_rows_and_School_enrollment_analysis_Colorado_-_Google_Docs" src="https://user-images.githubusercontent.com/9599/128557997-6bedb52b-45f1-4ab1-9a79-ddf6c01f6036.png"> That's with `<span style="font-size: 0.8em;color: #666;">23</span>` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 962391325 | |
| https://github.com/simonw/datasette/issues/1423#issuecomment-894453520 | https://api.github.com/repos/simonw/datasette/issues/1423 | 894453520 | IC_kwDOBm6k_c41UEcQ | 9599 | 2021-08-06T18:50:40Z | 2021-08-06T18:50:40Z | OWNER | Point of confusion: if only 30 options are shown, but there's a `...` at the end, what would the number be? It can't be the total number of facets because we haven't counted them all - but if it's just the number of displayed facets that's like to be confusing. So the original idea of showing the counts only if `_facet_size=max` is a good one. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 962391325 | |
| https://github.com/simonw/datasette/issues/1423#issuecomment-894454087 | https://api.github.com/repos/simonw/datasette/issues/1423 | 894454087 | IC_kwDOBm6k_c41UElH | 9599 | 2021-08-06T18:51:42Z | 2021-08-06T18:51:42Z | OWNER | The invisible tooltip could say "Showing 30 items, more available" (helping save you from counting up to 20 if you know about the secret feature). The numbers could then be fully displayed on the "..." page. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 962391325 |