{{ row[""title""] }}
{{ row[""description""] }}
Category: {{ row.display(""category_id"") }}
{% endfor %}
```
Is that a good design? the `.display()` thing feels weird - I wonder if anyone has ever actually used that.
It's documented here: https://docs.datasette.io/en/0.64.2/custom_templates.html#custom-templates
> If you want to output the rendered HTML version of a column, including any links to foreign keys, you can use `{{ row.display(""column_name"") }}`.
I can't see any examples of anyone using it in this code search: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+row.display
It is however useful to have some kind of abstraction layer here that insulates the SQLite `Row` object, since having an extra layer will help if Datasette ever grows support for alternative database backends such as DuckDB or PostgreSQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460943097,https://api.github.com/repos/simonw/datasette/issues/1999,1460943097,IC_kwDOBm6k_c5XFDj5,9599,2023-03-08T22:09:24Z,2023-03-08T22:09:47Z,OWNER,"The ease with which I added that `?_extra=query` feature in https://github.com/simonw/datasette/pull/1999/commits/96e94f9b7b2db53865e61390bcce6761727f26d8 made me feel really confident that this architecture is going in the right direction.
```diff
diff --git a/datasette/views/table.py b/datasette/views/table.py
index 8d3bb2c930..3e1db9c85f 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -1913,6 +1913,13 @@ async def extra_request():
""args"": request.args._data,
}
+ async def extra_query():
+ ""Details of the underlying SQL query""
+ return {
+ ""sql"": sql,
+ ""params"": params,
+ }
+
async def extra_extras():
""Available ?_extra= blocks""
return {
@@ -1938,6 +1945,7 @@ async def extra_extras():
extra_primary_keys,
extra_debug,
extra_request,
+ extra_query,
extra_extras,
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/issues/2036#issuecomment-1460816528,https://api.github.com/repos/simonw/datasette/issues/2036,1460816528,IC_kwDOBm6k_c5XEkqQ,9599,2023-03-08T20:22:50Z,2023-03-08T20:23:20Z,OWNER,"Testing this manually:
```
% datasette publish cloudrun content.db --service new-service
Creating temporary tarball archive of 2 file(s) totalling 13.8 MiB before compression.
Uploading tarball of [.] to [gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz]
Created [https://cloudbuild.googleapis.com/v1/projects/datasette-222320/locations/global/builds/290f41a4-e29a-443c-a1e5-c54513c6143d].
Logs are available at [ https://console.cloud.google.com/cloud-build/builds/290f41a4-e29a-443c-a1e5-c54513c6143d?project=99025868001 ].
---- REMOTE BUILD OUTPUT ----
starting build ""290f41a4-e29a-443c-a1e5-c54513c6143d""
FETCHSOURCE
Fetching storage object: gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483
Copying gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483...
/ [1 files][ 3.9 MiB/ 3.9 MiB]
Operation completed over 1 objects/3.9 MiB.
BUILD
Already have image (with digest): gcr.io/cloud-builders/docker
Sending build context to Docker daemon 14.52MB
Step 1/9 : FROM python:3.11.0-slim-bullseye
...
Installing collected packages: rfc3986, typing-extensions, sniffio, PyYAML, python-multipart, pluggy, pint, mergedeep, MarkupSafe, itsdangerous, idna, hupper, h11, click, certifi, asgiref, aiofiles, uvicorn, Jinja2, janus, click-default-group-wheel, asgi-csrf, anyio, httpcore, httpx, datasette
Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.2 PyYAML-6.0 aiofiles-23.1.0 anyio-3.6.2 asgi-csrf-0.9 asgiref-3.6.0 certifi-2022.12.7 click-8.1.3 click-default-group-wheel-1.2.2 datasette-0.64.1 h11-0.14.0 httpcore-0.16.3 httpx-0.23.3 hupper-1.11 idna-3.4 itsdangerous-2.1.2 janus-1.0.0 mergedeep-1.3.4 pint-0.20.1 pluggy-1.0.0 python-multipart-0.0.6 rfc3986-1.5.0 sniffio-1.3.0 typing-extensions-4.5.0 uvicorn-0.20.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[notice] A new release of pip available: 22.3 -> 23.0.1
[notice] To update, run: pip install --upgrade pip
Removing intermediate container 8ccebfebebc9
---> b972c85b38bb
...
Successfully built 606b7c286d7f
Successfully tagged gcr.io/datasette-222320/datasette-new-service:latest
PUSH
Pushing gcr.io/datasette-222320/datasette-new-service
The push refers to repository [gcr.io/datasette-222320/datasette-new-service]
667b1dc69e5e: Preparing
...
d8ddfcff216f: Pushed
latest: digest: sha256:452daffb2d3d7a8579c2ab39854be285155252c9428b4c1c50caac6a3a269e3f size: 2004
DONE
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ID CREATE_TIME DURATION SOURCE IMAGES STATUS
290f41a4-e29a-443c-a1e5-c54513c6143d 2023-03-08T20:21:03+00:00 39S gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz gcr.io/datasette-222320/datasette-new-service (+1 more) SUCCESS
Deploying container to Cloud Run service [new-service] in project [datasette-222320] region [us-central1]
✓ Deploying new service... Done.
✓ Creating Revision...
✓ Routing traffic...
✓ Setting IAM Policy...
Done.
Service [new-service] revision [new-service-00001-zon] has been deployed and is serving 100 percent of traffic.
Service URL: https://new-service-j7hipcg4aq-uc.a.run.app
```
https://new-service-j7hipcg4aq-uc.a.run.app/ was deployed successfully.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460906741,https://api.github.com/repos/simonw/datasette/issues/1999,1460906741,IC_kwDOBm6k_c5XE6r1,9599,2023-03-08T21:34:08Z,2023-03-08T21:34:08Z,OWNER,"So maybe I can refactor it to look a bit more like this:
https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1602-L1604
One thing that's useful here is that `is_view` is handled early, like this:
https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L466-L472
So if I omit the `is_view` bit from the extracted function I can simplify more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460639749,https://api.github.com/repos/simonw/datasette/issues/2035,1460639749,IC_kwDOBm6k_c5XD5gF,9599,2023-03-08T18:17:31Z,2023-03-08T18:17:31Z,OWNER,"Since we are pre-1.0 it's still OK to implement a feature that disallows `?id__list=` in the URL, but allows `:id__list` in SQL queries to reference the JSON list of parameters.
So I'm going to prototype this as the `:id__list` feature and see how it feels.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460682625,https://api.github.com/repos/simonw/datasette/issues/2035,1460682625,IC_kwDOBm6k_c5XED-B,9599,2023-03-08T18:40:57Z,2023-03-08T18:40:57Z,OWNER,Pushed that prototype to a branch: https://github.com/simonw/datasette/commit/0fe844e9adb006a0138e83102ced1329d9155c59 / https://github.com/simonw/datasette/compare/sql-list-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460618433,https://api.github.com/repos/simonw/datasette/issues/2035,1460618433,IC_kwDOBm6k_c5XD0TB,9599,2023-03-08T18:06:34Z,2023-03-08T18:06:34Z,OWNER,"One way to do this would be to dynamically generate the `where id in (?, ?, ?)` with the correct number of question marks, then feed in a list from `request.args.getlist(""id"")` - but that would require rewriting the SQL query text to add those question marks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460628199,https://api.github.com/repos/simonw/datasette/issues/2035,1460628199,IC_kwDOBm6k_c5XD2rn,9599,2023-03-08T18:11:31Z,2023-03-08T18:11:31Z,OWNER,"One variant on this idea: maybe you have to specify in your query that you want it to be the JSON list version, not the single item (first `?id=` parameter version)? Maybe with syntax like this:
where id in (select value from json_each(:id__list))
Datasette would automatically pass `{""id"": ""11"", ""id__list"": '[""11"", ""32"", ""62""]'}` as arguments to the `db.execute()` method, if the page was called with `?id=11&id=32&id=62`.
This is more explicit, though the syntax is a bit uglier (maybe there's a nicer design for this?). I also worry about `?id__list=` conflicting with this, but I think that's a risk I can take - tell people not to do that, or even block `?id__list=` style parameters entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460621871,https://api.github.com/repos/simonw/datasette/issues/2035,1460621871,IC_kwDOBm6k_c5XD1Iv,9599,2023-03-08T18:08:25Z,2023-03-08T18:09:04Z,OWNER,"My current preferred solution is to lean into SQLite's JSON support.
What if the query page spotted `?id=11&id=32&id=62` and turned that into a JSON string called `:id:` with a value of `[""11"", ""32"", ""62""]`?
Note that this is still a string, not a list. This avoids a nasty problem that occurred in PHP world, where `?id[]=1&id[]=2` would result in an actual PHP array object, which often broke underlying code that had expected `$_GET[""id""]` to be a string, not an array.
So in a query you'd be able to do this:
where id in (select value from json_each(:id))
And then call it with `?id=11&id=32&id=62`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460916405,https://api.github.com/repos/simonw/datasette/issues/1999,1460916405,IC_kwDOBm6k_c5XE9C1,9599,2023-03-08T21:43:27Z,2023-03-08T21:43:27Z,OWNER,"Just noticed that `_json=colname` is not working, and that's because it's handled by the renderer here:
https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/renderer.py#L29-L40
But that's not currently being called by my new code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/issues/2037#issuecomment-1460838797,https://api.github.com/repos/simonw/datasette/issues/2037,1460838797,IC_kwDOBm6k_c5XEqGN,9599,2023-03-08T20:31:15Z,2023-03-08T20:31:15Z,OWNER,"It's this test here:
https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/tests/test_cli.py#L181-L189
Added in:
- #2033 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615891776,
https://github.com/simonw/datasette/issues/2037#issuecomment-1460838109,https://api.github.com/repos/simonw/datasette/issues/2037,1460838109,IC_kwDOBm6k_c5XEp7d,9599,2023-03-08T20:30:36Z,2023-03-08T20:30:36Z,OWNER,Instead of using `isolated_filesystem()` I could use a `tmpdir` fixture instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615891776,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460760116,https://api.github.com/repos/simonw/datasette/issues/1999,1460760116,IC_kwDOBm6k_c5XEW40,9599,2023-03-08T19:48:52Z,2023-03-08T19:48:52Z,OWNER,"I'm trying to get `http://127.0.0.1:8001/fixtures/compound_three_primary_keys?_next=a,d,v` to return the correct results.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460637906,https://api.github.com/repos/simonw/datasette/issues/2035,1460637906,IC_kwDOBm6k_c5XD5DS,9599,2023-03-08T18:16:31Z,2023-03-08T18:16:31Z,OWNER,"I'm pretty sold on this as a feature now. The main question I have is which of these options to implement:
1. `?id=1&?id=2` results in `:id` in the query being `[""1"", ""2""]` - no additional syntax required
2. `:id` in the query continues to reference just the first of those parameters - but `:id__list` (or some other custom syntax) instead gets `[""1"", ""2""]` - or, if the URL is `?id=1` - gets `[""1""]`
Actually on writing these out I realize that option 2 is the ONLY valid option. It's no good building a query that works against a JSON list if the user might pass just a single ID, `?id=1`, resulting in their query breaking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2036#issuecomment-1460810523,https://api.github.com/repos/simonw/datasette/issues/2036,1460810523,IC_kwDOBm6k_c5XEjMb,9599,2023-03-08T20:17:01Z,2023-03-08T20:17:01Z,OWNER,"I'm going to solve this by using the service name in that `image_id` instead:
```python
image_id = f""gcr.io/{project}/{service_name}""
```
This is a nasty bug, so I'm going to backport it to a `0.64.2` release as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460907148,https://api.github.com/repos/simonw/datasette/issues/1999,1460907148,IC_kwDOBm6k_c5XE6yM,9599,2023-03-08T21:34:30Z,2023-03-08T21:34:30Z,OWNER,"I'm going to hold off on that refactor until later, when I have tests to show me if the refactor works or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469,https://api.github.com/repos/simonw/datasette/issues/1999,1460905469,IC_kwDOBm6k_c5XE6X9,9599,2023-03-08T21:32:56Z,2023-03-08T21:32:56Z,OWNER,"I'd really like to extract this ugly logic out into a helper function:
https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1617-L1684
I copied it in and asked ChatGPT to ""List all of the variable that are either modified or assigned to by the above code"":
> Modified variables:
>
> offset
> where_clauses
> params
> order_by
>
> Assigned variables:
>
> sort_value
> next_by_pk_clauses
> order_by (when sort or sort_desc is not None)
Then I asked which variables were used as inputs, and argued with it a bit about whether it should be counting functions. Eventually got to this:
> My apologies for the oversight. Here are the variables needed as input by the above code, excluding any functions:
>
> _next
> is_view
> sort
> sort_desc
> use_rowid
> pks
> order_by
Note that `use_rowid` is actually defined earlier in terms of two of those other variables: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1540","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1461044477,https://api.github.com/repos/simonw/datasette/issues/1999,1461044477,IC_kwDOBm6k_c5XFcT9,9599,2023-03-08T23:47:26Z,2023-03-08T23:47:26Z,OWNER,"I want to package together all of the extras that are needed for the HTML format. A few options for doing that:
- Introduce `?_extra=_html` where the leading underscore indicates that this is a ""bundle"" of extras, then define a bundle that's everything needed for the HTML renderer
- Have some other mechanism whereby different renderers can request a bundle of extras.
I'm leaning towards the first option. I'll try that and see what it looks like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/issues/2036#issuecomment-1460809643,https://api.github.com/repos/simonw/datasette/issues/2036,1460809643,IC_kwDOBm6k_c5XEi-r,9599,2023-03-08T20:16:10Z,2023-03-08T20:16:10Z,OWNER,"I think the code at fault is here:
https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/datasette/publish/cloudrun.py#L176-L182
That name ends up defaulting to `datasette` - so multiple different projects may end up deploying to the same `image_id`.
What I think happened in the `datasette.io` bug is that this workflow: https://github.com/simonw/simonwillisonblog-backup/blob/bfb573e96d8622ab52b22fdcd54724fe6e59fd24/.github/workflows/backup.yml and this workflow: https://github.com/simonw/datasette.io/blob/4676db5bf4a3fc9f792ee270ec0c59eb902cd2c3/.github/workflows/deploy.yml both happened to run at the exact same time.
And so the image that was pushed to `gcr.io/datasette-222320/datasette:latest` by the `simonw/simonwillisonblog-backup` action was then deployed by the `simonw/datasette.io/` action, which broke the site.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460679434,https://api.github.com/repos/simonw/datasette/issues/2035,1460679434,IC_kwDOBm6k_c5XEDMK,9599,2023-03-08T18:39:35Z,2023-03-08T18:39:35Z,OWNER,"I should consider the existing design of magic parameters here: https://docs.datasette.io/en/stable/sql_queries.html#magic-parameters
- `_actor_*`
- `_header_*`
- `_cookie_`
- `_now_epoch`
- `_now_date_utc`
- `_now_datetime_utc`
- `_random_chars_*`
Should this new `id__list` syntax look more like those magic parameters, or is it OK to use `name__magic` syntax here instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2027#issuecomment-1459455356,https://api.github.com/repos/simonw/datasette/issues/2027,1459455356,IC_kwDOBm6k_c5W_YV8,1350673,2023-03-08T04:42:22Z,2023-03-08T04:42:22Z,NONE,"I managed to make it work by using nginx's 'exact match' (=) combined with 'prefix match'; that is, match explicitly on `/`, and redirect to `/