html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/1249#issuecomment-808998719,https://api.github.com/repos/simonw/datasette/issues/1249,808998719,MDEyOklzc3VlQ29tbWVudDgwODk5ODcxOQ==,9599,simonw,2021-03-29T00:57:13Z,2021-03-29T00:57:13Z,OWNER,"I just shipped Datasette 0.56 - here's the CI run: https://github.com/simonw/datasette/runs/2214701802?check_suite_focus=true

It pushed a new `latest` tag to https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated

    docker pull datasetteproject/datasette:latest

And then:

    docker run datasetteproject/datasette:latest datasette \
      --load-extension=spatialite \
      --get /-/versions.json | jq .sqlite

Outputs:
```json
{
  ""version"": ""3.27.2"",
  ""fts_versions"": [
    ""FTS5"",
    ""FTS4"",
    ""FTS3""
  ],
  ""extensions"": {
    ""json1"": null,
    ""spatialite"": ""5.0.1""
  },
  ""compile_options"": [
    ""COMPILER=gcc-8.3.0"",
    ""ENABLE_COLUMN_METADATA"",
    ""ENABLE_DBSTAT_VTAB"",
    ""ENABLE_FTS3"",
    ""ENABLE_FTS3_PARENTHESIS"",
    ""ENABLE_FTS3_TOKENIZER"",
    ""ENABLE_FTS4"",
    ""ENABLE_FTS5"",
    ""ENABLE_JSON1"",
    ""ENABLE_LOAD_EXTENSION"",
    ""ENABLE_PREUPDATE_HOOK"",
    ""ENABLE_RTREE"",
    ""ENABLE_SESSION"",
    ""ENABLE_STMTVTAB"",
    ""ENABLE_UNLOCK_NOTIFY"",
    ""ENABLE_UPDATE_DELETE_LIMIT"",
    ""HAVE_ISNAN"",
    ""LIKE_DOESNT_MATCH_BLOBS"",
    ""MAX_SCHEMA_RETRY=25"",
    ""MAX_VARIABLE_NUMBER=250000"",
    ""OMIT_LOOKASIDE"",
    ""SECURE_DELETE"",
    ""SOUNDEX"",
    ""TEMP_STORE=1"",
    ""THREADSAFE=1"",
    ""USE_URI""
  ]
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/pull/1031#issuecomment-808989067,https://api.github.com/repos/simonw/datasette/issues/1031,808989067,MDEyOklzc3VlQ29tbWVudDgwODk4OTA2Nw==,9599,simonw,2021-03-29T00:23:41Z,2021-03-29T00:23:41Z,OWNER,"This bug should have been fixed in #1229 - let me know if that's not the case!

Thanks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724369025,Fallback to databases in inspect-data.json when no -i options are passed,
https://github.com/simonw/datasette/pull/1260#issuecomment-808988697,https://api.github.com/repos/simonw/datasette/issues/1260,808988697,MDEyOklzc3VlQ29tbWVudDgwODk4ODY5Nw==,9599,simonw,2021-03-29T00:22:21Z,2021-03-29T00:22:21Z,OWNER,"This is interesting!

I've decided to apply a subset of these - the `if` and `elif` blocks are a deliberate style choice from me, because I find code clearer when it has if/else as opposed to relying on early termination. Likewise the iteration against `.keys()` on dictionaries.

I like the other fixes though, I'm about to land them in a separate commit that credits you.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",831163537,Fix: code quality issues,
https://github.com/simonw/datasette/pull/1229#issuecomment-808987304,https://api.github.com/repos/simonw/datasette/issues/1229,808987304,MDEyOklzc3VlQ29tbWVudDgwODk4NzMwNA==,9599,simonw,2021-03-29T00:17:13Z,2021-03-29T00:17:13Z,OWNER,Thanks for figuring this out!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810507413,ensure immutable databses when starting in configuration directory mode with,
https://github.com/simonw/datasette/pull/1252#issuecomment-808986495,https://api.github.com/repos/simonw/datasette/issues/1252,808986495,MDEyOklzc3VlQ29tbWVudDgwODk4NjQ5NQ==,9599,simonw,2021-03-29T00:13:59Z,2021-03-29T00:13:59Z,OWNER,"Neat fix, thank you!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",825217564,Add back styling to lists within table cells (fixes #1141),
https://github.com/simonw/datasette/pull/1279#issuecomment-808986036,https://api.github.com/repos/simonw/datasette/issues/1279,808986036,MDEyOklzc3VlQ29tbWVudDgwODk4NjAzNg==,9599,simonw,2021-03-29T00:11:50Z,2021-03-29T00:11:50Z,OWNER,Thanks for the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842556944,Minor Docs Update. Added `--app` to fly install command.,
https://github.com/simonw/datasette/issues/1280#issuecomment-808983160,https://api.github.com/repos/simonw/datasette/issues/1280,808983160,MDEyOklzc3VlQ29tbWVudDgwODk4MzE2MA==,9599,simonw,2021-03-28T23:59:28Z,2021-03-29T00:10:05Z,OWNER,"Might be easier to do this using https://github.com/coleifer/pysqlite3 rather than try to replace the system `sqlite3` on the Ubuntu GitHub Actions instances.

These instructions should help: https://github.com/coleifer/pysqlite3#building-a-statically-linked-library","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842862708,Ability to run CI against multiple SQLite versions,
https://github.com/simonw/datasette/issues/1276#issuecomment-808981968,https://api.github.com/repos/simonw/datasette/issues/1276,808981968,MDEyOklzc3VlQ29tbWVudDgwODk4MTk2OA==,9599,simonw,2021-03-28T23:52:31Z,2021-03-28T23:52:31Z,OWNER,Testing this on Glitch by adding `https://github.com/simonw/datasette/archive/48d5e0e6ac8975cfd869d4e8c69c64ca0c65e29e.zip` as a dependency. That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,"Invalid SQL: ""no such table: pragma_database_list"" on database page",
https://github.com/simonw/datasette/issues/1276#issuecomment-808979608,https://api.github.com/repos/simonw/datasette/issues/1276,808979608,MDEyOklzc3VlQ29tbWVudDgwODk3OTYwOA==,9599,simonw,2021-03-28T23:38:34Z,2021-03-28T23:38:34Z,OWNER,"Aha! https://www.sqlite.org/pragma.html says:

>  The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,"Invalid SQL: ""no such table: pragma_database_list"" on database page",
https://github.com/simonw/datasette/issues/1276#issuecomment-808979218,https://api.github.com/repos/simonw/datasette/issues/1276,808979218,MDEyOklzc3VlQ29tbWVudDgwODk3OTIxOA==,9599,simonw,2021-03-28T23:35:53Z,2021-03-28T23:36:26Z,OWNER,"Here's where I run that: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L249-L253

That's from when I added the `--crossdb` option in #1232: https://github.com/simonw/datasette/commit/6f41c8a2bef309a66588b2875c3e24d26adb4850#diff-4e20309c969326a0008dc9237f6807f48d55783315fbfc1e7dfa480b550e16f9R249","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,"Invalid SQL: ""no such table: pragma_database_list"" on database page",
https://github.com/simonw/datasette/issues/1276#issuecomment-808979049,https://api.github.com/repos/simonw/datasette/issues/1276,808979049,MDEyOklzc3VlQ29tbWVudDgwODk3OTA0OQ==,9599,simonw,2021-03-28T23:34:38Z,2021-03-28T23:34:38Z,OWNER,"The Glitch server logs showed:

> `ERROR: conn=<sqlite3.Connection object at 0x7f88ebadd8f0>, sql = 'select seq, name, file from pragma_database_list() where seq > 0', params = None: no such table: pragma_database_list`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,"Invalid SQL: ""no such table: pragma_database_list"" on database page",
https://github.com/simonw/datasette/issues/1276#issuecomment-808978808,https://api.github.com/repos/simonw/datasette/issues/1276,808978808,MDEyOklzc3VlQ29tbWVudDgwODk3ODgwOA==,9599,simonw,2021-03-28T23:32:58Z,2021-03-28T23:33:58Z,OWNER,"I just managed to replicate this bug on Glitch: https://nicar-2020.glitch.me/data

> Invalid SQL
> no such table: pragma_database_list

https://nicar-2020.glitch.me/-/versions says:

```json
{
    ""python"": {
        ""version"": ""3.7.10"",
        ""full"": ""3.7.10 (default, Feb 20 2021, 21:21:24) \n[GCC 5.4.0 20160609]""
    },
    ""datasette"": {
        ""version"": ""0.55""
    },
    ""asgi"": ""3.0"",
    ""uvicorn"": ""0.13.4"",
    ""sqlite"": {
        ""version"": ""3.11.0"",
        ""fts_versions"": [
            ""FTS4"",
            ""FTS3""
        ],
        ""extensions"": {
            ""json1"": null
        },
        ""compile_options"": [
            ""ENABLE_COLUMN_METADATA"",
            ""ENABLE_DBSTAT_VTAB"",
            ""ENABLE_FTS3"",
            ""ENABLE_FTS3_PARENTHESIS"",
            ""ENABLE_JSON1"",
            ""ENABLE_LOAD_EXTENSION"",
            ""ENABLE_RTREE"",
            ""ENABLE_UNLOCK_NOTIFY"",
            ""ENABLE_UPDATE_DELETE_LIMIT"",
            ""HAVE_ISNAN"",
            ""LIKE_DOESNT_MATCH_BLOBS"",
            ""MAX_SCHEMA_RETRY=25"",
            ""OMIT_LOOKASIDE"",
            ""SECURE_DELETE"",
            ""SOUNDEX"",
            ""SYSTEM_MALLOC"",
            ""TEMP_STORE=1"",
            ""THREADSAFE=1""
        ]
    }
}
```
That's [SQLite 3.11.0 from 2016-02-15](https://www.sqlite.org/releaselog/3_11_0.html) with no FTS5.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,"Invalid SQL: ""no such table: pragma_database_list"" on database page",
https://github.com/simonw/datasette/issues/1273#issuecomment-808759984,https://api.github.com/repos/simonw/datasette/issues/1273,808759984,MDEyOklzc3VlQ29tbWVudDgwODc1OTk4NA==,9599,simonw,2021-03-27T16:43:17Z,2021-03-27T16:43:17Z,OWNER,That rivers example in the tutorial would work a lot better with a live demo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838382890,Refresh SpatiaLite documentation,
https://github.com/simonw/datasette/issues/1273#issuecomment-808757721,https://api.github.com/repos/simonw/datasette/issues/1273,808757721,MDEyOklzc3VlQ29tbWVudDgwODc1NzcyMQ==,9599,simonw,2021-03-27T16:25:48Z,2021-03-27T16:25:48Z,OWNER,"> This will give you back an additional column of GeoJSON. You can copy and paste GeoJSON from this column into the debugging tool at geojson.io to visualize it on a map.

That should promote `datasette-leaflet-geojson` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838382890,Refresh SpatiaLite documentation,
https://github.com/simonw/datasette/issues/1090#issuecomment-808757659,https://api.github.com/repos/simonw/datasette/issues/1090,808757659,MDEyOklzc3VlQ29tbWVudDgwODc1NzY1OQ==,9599,simonw,2021-03-27T16:25:25Z,2021-03-27T16:25:25Z,OWNER,Related feature request: ability to set default values for canned queries: #1258,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741862364,Custom widgets for canned query forms,
https://github.com/simonw/datasette/issues/1090#issuecomment-808757155,https://api.github.com/repos/simonw/datasette/issues/1090,808757155,MDEyOklzc3VlQ29tbWVudDgwODc1NzE1NQ==,9599,simonw,2021-03-27T16:21:43Z,2021-03-27T16:21:43Z,OWNER,"Idea for these: imitate https://django-sql-dashboard.readthedocs.io/en/latest/widgets.html#custom-widgets and drive them with templates.

So a custom widget type of `textarea` would look for a template called `widgets/textarea.html` - which means users could define brand new custom widgets just by creating their own template files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741862364,Custom widgets for canned query forms,
https://github.com/simonw/datasette/issues/1273#issuecomment-808756921,https://api.github.com/repos/simonw/datasette/issues/1273,808756921,MDEyOklzc3VlQ29tbWVudDgwODc1NjkyMQ==,9599,simonw,2021-03-27T16:19:45Z,2021-03-27T16:26:28Z,OWNER,"I have a better recipe for using spatial indexes now on https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/
```sql
select
  AsGeoJSON(geometry), *
from
  CPAD_2020a_SuperUnits
where
  PARK_NAME like '%mini%' and
  Intersects(GeomFromGeoJSON(:freedraw), geometry) = 1
  and CPAD_2020a_SuperUnits.rowid in (
    select
      rowid
    from
      SpatialIndex
    where
      f_table_name = 'CPAD_2020a_SuperUnits'
      and search_frame = GeomFromGeoJSON(:freedraw)
  )
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838382890,Refresh SpatiaLite documentation,
https://github.com/simonw/datasette/issues/1278#issuecomment-808756366,https://api.github.com/repos/simonw/datasette/issues/1278,808756366,MDEyOklzc3VlQ29tbWVudDgwODc1NjM2Ng==,9599,simonw,2021-03-27T16:15:47Z,2021-03-27T16:15:47Z,OWNER,https://timezones-api.datasette.io/ is now up and running on Cloud Run.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842416110,SpatiaLite timezones demo is broken,
https://github.com/simonw/datasette/issues/1278#issuecomment-808652008,https://api.github.com/repos/simonw/datasette/issues/1278,808652008,MDEyOklzc3VlQ29tbWVudDgwODY1MjAwOA==,9599,simonw,2021-03-27T04:47:17Z,2021-03-27T04:47:17Z,OWNER,"https://github.com/simonw/timezones-api is that project, it's pretty old now. I'll try to get it running on Cloud Run.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842416110,SpatiaLite timezones demo is broken,
https://github.com/simonw/datasette/issues/1258#issuecomment-808651088,https://api.github.com/repos/simonw/datasette/issues/1258,808651088,MDEyOklzc3VlQ29tbWVudDgwODY1MTA4OA==,9599,simonw,2021-03-27T04:41:52Z,2021-03-27T04:42:14Z,OWNER,"Right now they look like this:
```yaml
databases:
  fixtures:
    queries:
      neighborhood_search:
        params:
        - text
```
In addition to being able to specify defaults, I'd also like to add other things in the future - most significantly the ability to specify a different input widget (e.g. textarea v.s. single-line input)

So maybe this looks like:
```yaml
params:
- name: text
  default: """"
- name: age
  widget: number
```","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421,Allow canned query params to specify default values,
https://github.com/simonw/datasette/issues/1258#issuecomment-808650266,https://api.github.com/repos/simonw/datasette/issues/1258,808650266,MDEyOklzc3VlQ29tbWVudDgwODY1MDI2Ng==,9599,simonw,2021-03-27T04:37:07Z,2021-03-27T04:37:07Z,OWNER,I like that idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421,Allow canned query params to specify default values,
https://github.com/simonw/datasette/issues/1249#issuecomment-808649480,https://api.github.com/repos/simonw/datasette/issues/1249,808649480,MDEyOklzc3VlQ29tbWVudDgwODY0OTQ4MA==,9599,simonw,2021-03-27T04:32:10Z,2021-03-27T04:32:10Z,OWNER,I'll close this issue after I ship Datasette 0.56 and confirm that the Dockerfile was correctly built and published to Docker Hub.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-808649322,https://api.github.com/repos/simonw/datasette/issues/1249,808649322,MDEyOklzc3VlQ29tbWVudDgwODY0OTMyMg==,9599,simonw,2021-03-27T04:31:28Z,2021-03-27T04:31:28Z,OWNER,"One last test of that Dockerfile:
```
(datasette) datasette % docker build -f Dockerfile -t datasetteproject/datasette:0.55a --build-arg VERSION=0.55 .
(datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' | jq
{
  ""python"": {
    ""version"": ""3.9.2"",
    ""full"": ""3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]""
  },
  ""datasette"": {
    ""version"": ""0.55""
  },
  ""asgi"": ""3.0"",
  ""uvicorn"": ""0.13.4"",
  ""sqlite"": {
    ""version"": ""3.27.2"",
    ""fts_versions"": [
      ""FTS5"",
      ""FTS4"",
      ""FTS3""
    ],
    ""extensions"": {
      ""json1"": null
    },
    ""compile_options"": [
      ""COMPILER=gcc-8.3.0"",
      ""ENABLE_COLUMN_METADATA"",
      ""ENABLE_DBSTAT_VTAB"",
      ""ENABLE_FTS3"",
      ""ENABLE_FTS3_PARENTHESIS"",
      ""ENABLE_FTS3_TOKENIZER"",
      ""ENABLE_FTS4"",
      ""ENABLE_FTS5"",
      ""ENABLE_JSON1"",
      ""ENABLE_LOAD_EXTENSION"",
      ""ENABLE_PREUPDATE_HOOK"",
      ""ENABLE_RTREE"",
      ""ENABLE_SESSION"",
      ""ENABLE_STMTVTAB"",
      ""ENABLE_UNLOCK_NOTIFY"",
      ""ENABLE_UPDATE_DELETE_LIMIT"",
      ""HAVE_ISNAN"",
      ""LIKE_DOESNT_MATCH_BLOBS"",
      ""MAX_SCHEMA_RETRY=25"",
      ""MAX_VARIABLE_NUMBER=250000"",
      ""OMIT_LOOKASIDE"",
      ""SECURE_DELETE"",
      ""SOUNDEX"",
      ""TEMP_STORE=1"",
      ""THREADSAFE=1"",
      ""USE_URI""
    ]
  }
}
(datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' --load-extension=spatialite | jq
{
  ""python"": {
    ""version"": ""3.9.2"",
    ""full"": ""3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]""
  },
  ""datasette"": {
    ""version"": ""0.55""
  },
  ""asgi"": ""3.0"",
  ""uvicorn"": ""0.13.4"",
  ""sqlite"": {
    ""version"": ""3.27.2"",
    ""fts_versions"": [
      ""FTS5"",
      ""FTS4"",
      ""FTS3""
    ],
    ""extensions"": {
      ""json1"": null,
      ""spatialite"": ""5.0.1""
    },
    ""compile_options"": [
      ""COMPILER=gcc-8.3.0"",
      ""ENABLE_COLUMN_METADATA"",
      ""ENABLE_DBSTAT_VTAB"",
      ""ENABLE_FTS3"",
      ""ENABLE_FTS3_PARENTHESIS"",
      ""ENABLE_FTS3_TOKENIZER"",
      ""ENABLE_FTS4"",
      ""ENABLE_FTS5"",
      ""ENABLE_JSON1"",
      ""ENABLE_LOAD_EXTENSION"",
      ""ENABLE_PREUPDATE_HOOK"",
      ""ENABLE_RTREE"",
      ""ENABLE_SESSION"",
      ""ENABLE_STMTVTAB"",
      ""ENABLE_UNLOCK_NOTIFY"",
      ""ENABLE_UPDATE_DELETE_LIMIT"",
      ""HAVE_ISNAN"",
      ""LIKE_DOESNT_MATCH_BLOBS"",
      ""MAX_SCHEMA_RETRY=25"",
      ""MAX_VARIABLE_NUMBER=250000"",
      ""OMIT_LOOKASIDE"",
      ""SECURE_DELETE"",
      ""SOUNDEX"",
      ""TEMP_STORE=1"",
      ""THREADSAFE=1"",
      ""USE_URI""
    ]
  }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1272#issuecomment-808648974,https://api.github.com/repos/simonw/datasette/issues/1272,808648974,MDEyOklzc3VlQ29tbWVudDgwODY0ODk3NA==,9599,simonw,2021-03-27T04:29:42Z,2021-03-27T04:29:42Z,OWNER,"I'm skipping this for the moment because the new Dockerfile shape introduced in https://github.com/simonw/datasette/issues/1249#issuecomment-804404544 isn't compatible with this technique, since it installs Datasette from PyPI rather than directly from the repo.

Will need to change that if I want to do this unit tests thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838245338,Unit tests for the Dockerfile,
https://github.com/simonw/datasette/issues/1272#issuecomment-808647937,https://api.github.com/repos/simonw/datasette/issues/1272,808647937,MDEyOklzc3VlQ29tbWVudDgwODY0NzkzNw==,9599,simonw,2021-03-27T04:23:19Z,2021-03-27T04:23:36Z,OWNER,"Part of the challenge here is only running if a Docker daemon is available. I think this pattern works, in `tests/test_dockerfile.py`:
```python
import httpx
import pathlib
import pytest
import subprocess

root = pathlib.Path(__file__).parent.parent

def docker_is_available():
    try:
        client = httpx.Client(
            transport=httpx.HTTPTransport(uds=""/var/run/docker.sock"")
        )
        client.get(""http://docker/info"")
        return True
    except httpx.ConnectError:
        return False


@pytest.fixture
def build_container():
    assert (root / ""Dockerfile"").exists()
    subprocess.check_call([
        ""docker"", ""build"", str(root), ""-t"", ""datasette-dockerfile-test""
    ])


@pytest.mark.skipif(not docker_is_available(),
    reason=""Docker is not available""
)
def test_dockerfile(build_container):
    output = subprocess.check_output([
        ""docker"", ""run"", ""datasette-dockerfile-test"", ""datasette"", ""--get"", ""/_memory?sql=select+1&shape=_array""
    ])
    assert False, ""Implement better assertion here""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838245338,Unit tests for the Dockerfile,
https://github.com/simonw/datasette/issues/1276#issuecomment-808642405,https://api.github.com/repos/simonw/datasette/issues/1276,808642405,MDEyOklzc3VlQ29tbWVudDgwODY0MjQwNQ==,9599,simonw,2021-03-27T03:53:18Z,2021-03-27T03:53:18Z,OWNER,That's really odd. What version of SQLite are you using on the server? You can tell by visiting `https://your-site/-/versions`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,"Invalid SQL: ""no such table: pragma_database_list"" on database page",
https://github.com/simonw/datasette/issues/1277#issuecomment-808641846,https://api.github.com/repos/simonw/datasette/issues/1277,808641846,MDEyOklzc3VlQ29tbWVudDgwODY0MTg0Ng==,9599,simonw,2021-03-27T03:49:34Z,2021-03-27T03:49:34Z,OWNER,"I fixed this already, it's a duplicate of #1239","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842212586,Facet by array breaks if table name contains a space,
https://github.com/simonw/sqlite-utils/issues/252#issuecomment-808302971,https://api.github.com/repos/simonw/sqlite-utils/issues/252,808302971,MDEyOklzc3VlQ29tbWVudDgwODMwMjk3MQ==,9599,simonw,2021-03-26T15:21:38Z,2021-03-26T15:21:38Z,OWNER,Already got that! It's the `--nl` option - works for both importing and exporting data: https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842062949,Support json-line files,
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-807647791,https://api.github.com/repos/simonw/sqlite-utils/issues/251,807647791,MDEyOklzc3VlQ29tbWVudDgwNzY0Nzc5MQ==,9599,simonw,2021-03-25T22:42:48Z,2021-03-25T22:44:31Z,OWNER,"Idea: enhance `lambda` to allow it to return a dictionary of values, which will then be used to populate new columns. Use a `--multicolumn` option to indicate this:

    sqlite-utils convert lambda mydb.db mytable mycolumn \
      --code '{""first_name"": value.split()[0], ""last_name"": value.split()[1]}' \
      --multicolumn --drop

The `--drop` means ""drop the `mycolumn` column after making this change"".

Maybe `--multi` is a better name than `--multicolumn` here, since either way it's going to need additional explanation somewhere.

Would this overlap with #239 at all?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-807642041,https://api.github.com/repos/simonw/sqlite-utils/issues/251,807642041,MDEyOklzc3VlQ29tbWVudDgwNzY0MjA0MQ==,9599,simonw,2021-03-25T22:39:22Z,2021-03-25T22:39:22Z,OWNER,"Here's the full current implementation of that tool: https://github.com/simonw/sqlite-transform/blob/0.5/sqlite_transform/cli.py

My current plan is to make this functionality available as the following:

    sqlite-utils convert jsonsplit mydb.db mytable mycolumn
    sqlite-utils convert parsedatetime mydb.db mytable mycolumn
    sqlite-utils convert parsedate mydb.db mytable mycolumn
    sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()'

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/datasette/issues/741#issuecomment-806166575,https://api.github.com/repos/simonw/datasette/issues/741,806166575,MDEyOklzc3VlQ29tbWVudDgwNjE2NjU3NQ==,9599,simonw,2021-03-24T20:30:33Z,2021-03-24T20:30:33Z,OWNER,"`datasette package` is a mostly unmaintained feature at this point - it has a bit of test coverage but I've not made any improvements to it in a few years, and I don't use it for my own projects.

I'll make this change to `package` at the same time as I land it for `publish` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/1274#issuecomment-805216038,https://api.github.com/repos/simonw/datasette/issues/1274,805216038,MDEyOklzc3VlQ29tbWVudDgwNTIxNjAzOA==,9599,simonw,2021-03-23T20:14:53Z,2021-03-23T20:14:53Z,OWNER,"Yes this is one of the main reasons I'm planning to switch to encouraging YAML be default instead of JSON (while still supporting JSON) - YAML supports comments and multi-line strings.

See #1153 for YAML by default in the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",839008371,Might there be some way to comment metadata.json?,
https://github.com/simonw/datasette/issues/1153#issuecomment-805109341,https://api.github.com/repos/simonw/datasette/issues/1153,805109341,MDEyOklzc3VlQ29tbWVudDgwNTEwOTM0MQ==,9599,simonw,2021-03-23T17:55:48Z,2021-03-23T18:41:57Z,OWNER,"Beginnings of a UI element for switching between them:
```html
<div style=""border: 1px solid rgb(225, 228, 229);
  background-color: rgb(238, 255, 204);
  padding: 0.3em;
  position: relative;
  top: 3px;
  font-family: courier;"">
<a href=""#"" style=""display: inline-block; padding-left: 0px; padding-right: 2em;"">JSON</a>
<a href=""#"" style=""display: inline-block;"">YAML</a>
</div>
```
<img width=""646"" alt=""Metadata_—_Datasette_documentation"" src=""https://user-images.githubusercontent.com/9599/112194637-51f92500-8bc6-11eb-9662-3faa7ef37538.png"">

That `<pre>` has a padding of 12px, so using 12px padding on the tab links should get them to line up better.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1249#issuecomment-805033155,https://api.github.com/repos/simonw/datasette/issues/1249,805033155,MDEyOklzc3VlQ29tbWVudDgwNTAzMzE1NQ==,9599,simonw,2021-03-23T16:12:13Z,2021-03-23T16:12:13Z,OWNER,"Don't forget to update this bit of the docs: https://docs.datasette.io/en/0.55/spatialite.html#building-spatialite-from-source

> The packaged versions of SpatiaLite usually provide SpatiaLite 4.3.0a. For an example of how to build the most recent unstable version, 4.4.0-RC0 (which includes the powerful [VirtualKNN module](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN)), take a look at the [Datasette Dockerfile](https://github.com/simonw/datasette/blob/master/Dockerfile).

See also #1273","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1270#issuecomment-805058241,https://api.github.com/repos/simonw/datasette/issues/1270,805058241,MDEyOklzc3VlQ29tbWVudDgwNTA1ODI0MQ==,9599,simonw,2021-03-23T16:45:39Z,2021-03-23T16:45:39Z,OWNER,"I managed to build SpatiaLite such that this isn't necessary any more. I'm still interested in pursuing this further though - it feels like it could be a more robust way of implementing timeouts, but I need to prove to myself that it's better (maybe better performance, or handles more edge-cases?). Not sure how to prove that yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),
https://github.com/simonw/datasette/issues/1153#issuecomment-805056806,https://api.github.com/repos/simonw/datasette/issues/1153,805056806,MDEyOklzc3VlQ29tbWVudDgwNTA1NjgwNg==,9599,simonw,2021-03-23T16:43:38Z,2021-03-23T16:43:38Z,OWNER,"I used this code to get that:
```javascript
var jsonVersion = JSON.stringify(window.jsyaml.load(document.querySelector('.highlight-yaml').textContent), null, 4);
div.querySelector('.highlight pre').innerText = jsonVersion;
div.querySelector('.highlight pre').style.whiteSpace = 'pre-wrap'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805055291,https://api.github.com/repos/simonw/datasette/issues/1153,805055291,MDEyOklzc3VlQ29tbWVudDgwNTA1NTI5MQ==,9599,simonw,2021-03-23T16:41:31Z,2021-03-23T16:41:31Z,OWNER,"One downside of doing this conversion in JavaScript: it's much harder to get the same JSON syntax highlighting as that provided by Sphinx:

<img width=""655"" alt=""Metadata_—_Datasette_documentation"" src=""https://user-images.githubusercontent.com/9599/112183909-ef9b2700-8bbb-11eb-882b-aeb387c29e61.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805050163,https://api.github.com/repos/simonw/datasette/issues/1153,805050163,MDEyOklzc3VlQ29tbWVudDgwNTA1MDE2Mw==,9599,simonw,2021-03-23T16:34:35Z,2021-03-23T16:35:32Z,OWNER,"https://docs.datasette.io/en/stable/metadata.html has this example:
```yaml
title: Demonstrating Metadata from YAML
description_html: |-
  <p>This description includes a long HTML string</p>
  <ul>
    <li>YAML is better for embedding HTML strings than JSON!</li>
  </ul>
license: ODbL
license_url: https://opendatacommons.org/licenses/odbl/
databases:
  fixtures:
    tables:
      no_primary_key:
        hidden: true
    queries:
      neighborhood_search:
        sql: |-
          select neighborhood, facet_cities.name, state
          from facetable join facet_cities on facetable.city_id = facet_cities.id
          where neighborhood like '%' || :text || '%' order by neighborhood;
        title: Search neighborhoods
        description_html: |-
          <p>This demonstrates <em>basic</em> LIKE search
```

I ran this in the browser dev tools:
```javascript
var s = document.createElement('script')
s.src = 'https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js'
document.head.appendChild(s)
var yamlExample = document.querySelector('.highlight-yaml').textContent);
console.log(JSON.stringify(window.jsyaml.load(yamlExample), null, 4))
```
And got:
```json
{
    ""title"": ""Demonstrating Metadata from YAML"",
    ""description_html"": ""<p>This description includes a long HTML string</p>\n<ul>\n  <li>YAML is better for embedding HTML strings than JSON!</li>\n</ul>"",
    ""license"": ""ODbL"",
    ""license_url"": ""https://opendatacommons.org/licenses/odbl/"",
    ""databases"": {
        ""fixtures"": {
            ""tables"": {
                ""no_primary_key"": {
                    ""hidden"": true
                }
            },
            ""queries"": {
                ""neighborhood_search"": {
                    ""sql"": ""select neighborhood, facet_cities.name, state\nfrom facetable join facet_cities on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%' order by neighborhood;"",
                    ""title"": ""Search neighborhoods"",
                    ""description_html"": ""<p>This demonstrates <em>basic</em> LIKE search""
                }
            }
        }
    }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805047117,https://api.github.com/repos/simonw/datasette/issues/1153,805047117,MDEyOklzc3VlQ29tbWVudDgwNTA0NzExNw==,9599,simonw,2021-03-23T16:30:15Z,2021-03-23T16:46:06Z,OWNER,"https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js is only 12.5KB zipped, 38KB total - so that's not a bad option.

https://github.com/nodeca/js-yaml","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805042880,https://api.github.com/repos/simonw/datasette/issues/1153,805042880,MDEyOklzc3VlQ29tbWVudDgwNTA0Mjg4MA==,9599,simonw,2021-03-23T16:24:32Z,2021-03-23T16:24:32Z,OWNER,... actually I think I would do that conversion in Python. The client-side YAML parsers all look a little bit heavy to me in terms of additional page weight.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805041522,https://api.github.com/repos/simonw/datasette/issues/1153,805041522,MDEyOklzc3VlQ29tbWVudDgwNTA0MTUyMg==,9599,simonw,2021-03-23T16:22:46Z,2021-03-23T16:22:46Z,OWNER,"That's a good idea. I could do that with JavaScript - loading YAML and converting it to JSON in JavaScript shouldn't be hard, and it's better than JSON-to-YAML because there's only one correct JSON representation of a YAML file whereas you can represent a JSON document in YAML in a bunch of different ways.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/163#issuecomment-804540869,https://api.github.com/repos/simonw/datasette/issues/163,804540869,MDEyOklzc3VlQ29tbWVudDgwNDU0MDg2OQ==,9599,simonw,2021-03-23T02:44:33Z,2021-03-23T02:44:33Z,OWNER,Comments welcome!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",279547886,Document the querystring argument for setting a different time limit,
https://github.com/simonw/datasette/issues/1249#issuecomment-804406675,https://api.github.com/repos/simonw/datasette/issues/1249,804406675,MDEyOklzc3VlQ29tbWVudDgwNDQwNjY3NQ==,9599,simonw,2021-03-22T21:26:27Z,2021-03-22T21:26:27Z,OWNER,(Without the `apt-get update ...` SpatiaLite line it's 125MB),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804404544,https://api.github.com/repos/simonw/datasette/issues/1249,804404544,MDEyOklzc3VlQ29tbWVudDgwNDQwNDU0NA==,9599,simonw,2021-03-22T21:22:56Z,2021-03-22T21:24:24Z,OWNER,"Final version of Dockerfile which installs the specified version from GitHub:

    docker build . -t datasette-spatialite --build-arg VERSION=0.55

```dockerfile
FROM python:3.9.2-slim-buster as build

# Version of Datasette to install, e.g. 0.55
#   docker build . -t datasette --build-arg VERSION=0.55
ARG VERSION

# software-properties-common provides add-apt-repository
# which we need in order to install a more recent release
# of libsqlite3-mod-spatialite from the sid distribution
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
    apt-get remove -y software-properties-common && \
    apt clean && \
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

RUN pip install https://github.com/simonw/datasette/archive/refs/tags/${VERSION}.zip && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r && \
    rm -rf /root/.cache/pip

EXPOSE 8001
CMD [""datasette""]
```
Run against 0.55 this produces an image of 262MB","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804338678,https://api.github.com/repos/simonw/datasette/issues/1249,804338678,MDEyOklzc3VlQ29tbWVudDgwNDMzODY3OA==,9599,simonw,2021-03-22T19:33:43Z,2021-03-22T19:33:43Z,OWNER,"Replacing `rm -rf /var/lib/{apt,dpkg,cache,log}/` with 
```
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg
```
Got the size down to 305MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804318314,https://api.github.com/repos/simonw/datasette/issues/1249,804318314,MDEyOklzc3VlQ29tbWVudDgwNDMxODMxNA==,9599,simonw,2021-03-22T19:04:30Z,2021-03-22T19:04:30Z,OWNER,Considering the image on Docker Hub right now is `383MB` this is actually an improvement.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804317545,https://api.github.com/repos/simonw/datasette/issues/1249,804317545,MDEyOklzc3VlQ29tbWVudDgwNDMxNzU0NQ==,9599,simonw,2021-03-22T19:03:22Z,2021-03-22T19:03:22Z,OWNER,"This Dockerfile:
```dockerfile
FROM python:3.9.2-slim-buster as build

# software-properties-common provides add-apt-repository
RUN apt-get update && \
    apt-get -y install software-properties-common && \
    add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
    apt-get update && \
    apt-get -t sid install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

EXPOSE 8001
CMD [""datasette""]
```
Produces a 344MB image that includes a working SpatiaLite 5.0 module. And weirdly... it doesn't exhibit the hanging bug!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804310353,https://api.github.com/repos/simonw/datasette/issues/1249,804310353,MDEyOklzc3VlQ29tbWVudDgwNDMxMDM1Mw==,9599,simonw,2021-03-22T18:52:12Z,2021-03-22T18:52:12Z,OWNER,"This Dockerfile:
```dockerfile
FROM python:3.9.2-slim-buster as build

# Setup build dependencies
RUN apt update \
    && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
    libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
    && apt clean

RUN wget ""https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz"" && tar xzf sqlite-autoconf-3340100.tar.gz \
    && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
    CFLAGS=""-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1"" \
    && make && make install

RUN wget ""http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz"" && tar zxf freexl-1.0.6.tar.gz \
    && cd freexl-1.0.6 && ./configure && make && make install

RUN wget ""http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz"" && tar zxf libspatialite-5.0.1.tar.gz \
    && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install

RUN wget ""http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz"" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install

RUN wget ""http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz"" && tar zxf spatialite-tools-5.0.0.tar.gz \
    && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install

# Add local code to the image instead of fetching from pypi.
#COPY . /datasette
#RUN pip install /datasette
RUN pip install datasette

FROM python:3.9.2-slim-buster

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=build /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```
Produced a 448MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804309510,https://api.github.com/repos/simonw/datasette/issues/1249,804309510,MDEyOklzc3VlQ29tbWVudDgwNDMwOTUxMA==,9599,simonw,2021-03-22T18:50:50Z,2021-03-22T18:50:50Z,OWNER,"Ideally I'd like to use the Debian stable `python:3.9.2-slim-buster` base image but install SpatiaLite from Debian unstable here: https://packages.debian.org/sid/libspatialite7

This pattern might let me do that: https://github.com/helmesjo/cpp_bash_utils/blob/f031e926249f8e2d7f260f22dc8974c6d5be11fe/docker/images/linux-gcc.dockerfile#L20-L24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804384196,https://api.github.com/repos/simonw/datasette/issues/1249,804384196,MDEyOklzc3VlQ29tbWVudDgwNDM4NDE5Ng==,9599,simonw,2021-03-22T20:48:46Z,2021-03-22T20:48:46Z,OWNER,I think part of the reason it's smaller is that I ran `pip install datasette` instead of using `COPY . /datasette` followed by `pip install /datasette`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804380181,https://api.github.com/repos/simonw/datasette/issues/1249,804380181,MDEyOklzc3VlQ29tbWVudDgwNDM4MDE4MQ==,9599,simonw,2021-03-22T20:42:16Z,2021-03-22T20:42:16Z,OWNER,"Considering the image on Docker Hub is 383MB, I'm happy with getting that down to 262MB. I'm going to stop looking for new optimizations here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804379644,https://api.github.com/repos/simonw/datasette/issues/1249,804379644,MDEyOklzc3VlQ29tbWVudDgwNDM3OTY0NA==,9599,simonw,2021-03-22T20:41:23Z,2021-03-22T20:41:23Z,OWNER,I tried adding `apt-get remove -y software-properties-common &&` to remove `software-properties-common` but it made no difference to the image size.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804372977,https://api.github.com/repos/simonw/datasette/issues/1249,804372977,MDEyOklzc3VlQ29tbWVudDgwNDM3Mjk3Nw==,9599,simonw,2021-03-22T20:30:37Z,2021-03-22T20:30:37Z,OWNER,"I tried copying just the `mod_spatialite.so` file into a second stage build but it failed. So I ran `bash` in a working image and used `ldd` to figure out what it was linked to:
```
root@39683f91e588:/usr/lib/x86_64-linux-gnu# ldd mod_spatialite.so
	linux-vdso.so.1 (0x00007ffd021f4000)
	libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f5c75412000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5c753f0000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c752ac000)
	libminizip.so.1 => /usr/lib/x86_64-linux-gnu/libminizip.so.1 (0x00007f5c750a0000)
	librttopo.so.1 => /usr/lib/x86_64-linux-gnu/librttopo.so.1 (0x00007f5c75028000)
	libfreexl.so.1 => /usr/lib/x86_64-linux-gnu/libfreexl.so.1 (0x00007f5c7501c000)
	libproj.so.19 => /usr/lib/x86_64-linux-gnu/libproj.so.19 (0x00007f5c74ca7000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c74a89000)
	libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f5c74967000)
	libgeos_c.so.1 => /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007f5c7492b000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c74766000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c74760000)
	libicuuc.so.67 => /usr/lib/x86_64-linux-gnu/libicuuc.so.67 (0x00007f5c74575000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5c7454d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5c75d49000)
	libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f5c744c7000)
	libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f5c74439000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5c7426c000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5c74250000)
	libgeos-3.9.0.so => /usr/lib/x86_64-linux-gnu/libgeos-3.9.0.so (0x00007f5c74040000)
	libicudata.so.67 => /usr/lib/x86_64-linux-gnu/libicudata.so.67 (0x00007f5c72527000)
	libwebp.so.6 => /usr/lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f5c724bc000)
	libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f5c7241c000)
	libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f5c7220e000)
	libjpeg.so.62 => /usr/lib/x86_64-linux-gnu/libjpeg.so.62 (0x00007f5c72188000)
	libdeflate.so.0 => /usr/lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007f5c7216c000)
	libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f5c72144000)
	libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f5c72125000)
	librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f5c71f08000)
	libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f5c71eda000)
	libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f5c71ec5000)
	libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f5c71e8d000)
	libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f5c71ce0000)
	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f5c71c93000)
	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f5c71bb3000)
	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f5c71b7f000)
	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f5c71b77000)
	libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f5c71b23000)
	liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f5c71b12000)
	libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f5c7198e000)
	libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f5c71955000)
	libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f5c718d0000)
	libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f5c717b2000)
	libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f5c71683000)
	libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f5c71470000)
	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f5c71461000)
	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f5c71458000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5c7143e000)
	libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f5c71421000)
	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f5c713fe000)
	libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f5c713f4000)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804368372,https://api.github.com/repos/simonw/datasette/issues/1249,804368372,MDEyOklzc3VlQ29tbWVudDgwNDM2ODM3Mg==,9599,simonw,2021-03-22T20:22:43Z,2021-03-22T20:22:43Z,OWNER,"```dockerfile
FROM python:3.9.2-slim-buster as build

# software-properties-common provides add-apt-repository
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

RUN pip install datasette && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r && \
    rm -rf /root/.cache/pip

EXPOSE 8001
CMD [""datasette""]
```
262 MB","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804363687,https://api.github.com/repos/simonw/datasette/issues/1249,804363687,MDEyOklzc3VlQ29tbWVudDgwNDM2MzY4Nw==,9599,simonw,2021-03-22T20:15:00Z,2021-03-22T20:15:00Z,OWNER,"```
RUN pip install datasette && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r
```
That dropped it to 265MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804360701,https://api.github.com/repos/simonw/datasette/issues/1249,804360701,MDEyOklzc3VlQ29tbWVudDgwNDM2MDcwMQ==,9599,simonw,2021-03-22T20:10:07Z,2021-03-22T20:10:07Z,OWNER,Adding `--no-install-recommends` dropped it to 275MB,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804347152,https://api.github.com/repos/simonw/datasette/issues/1249,804347152,MDEyOklzc3VlQ29tbWVudDgwNDM0NzE1Mg==,9599,simonw,2021-03-22T19:47:56Z,2021-03-22T19:48:03Z,OWNER,I wrote a bunch of tips on creating smaller Docker images here: https://simonwillison.net/2018/Nov/19/smaller-python-docker-images/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804344553,https://api.github.com/repos/simonw/datasette/issues/1249,804344553,MDEyOklzc3VlQ29tbWVudDgwNDM0NDU1Mw==,9599,simonw,2021-03-22T19:43:25Z,2021-03-22T19:43:25Z,OWNER,Does `--no-install-recommends` make a difference?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/pull/1271#issuecomment-804299406,https://api.github.com/repos/simonw/datasette/issues/1271,804299406,MDEyOklzc3VlQ29tbWVudDgwNDI5OTQwNg==,9599,simonw,2021-03-22T18:36:14Z,2021-03-22T21:49:27Z,OWNER,"This isn't actually working - the outer code attempts to send an `.interrupt()` call to the connection object via the `connections` thread-local, which doesn't work because it's a thread-local so the connection isn't visible to that code.

Need to figure out how to communicate with that thread properly.

Also a test that fails in this particular case would be a good idea!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,Use SQLite conn.interrupt() instead of sqlite_timelimit(),
https://github.com/simonw/datasette/pull/1271#issuecomment-804265042,https://api.github.com/repos/simonw/datasette/issues/1271,804265042,MDEyOklzc3VlQ29tbWVudDgwNDI2NTA0Mg==,9599,simonw,2021-03-22T17:45:45Z,2021-03-22T17:45:45Z,OWNER,"I can remove this code too: 
https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L190-L192","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,Use SQLite conn.interrupt() instead of sqlite_timelimit(),
https://github.com/simonw/datasette/issues/1249#issuecomment-804263434,https://api.github.com/repos/simonw/datasette/issues/1249,804263434,MDEyOklzc3VlQ29tbWVudDgwNDI2MzQzNA==,9599,simonw,2021-03-22T17:43:25Z,2021-03-22T17:43:25Z,OWNER,I figured out the cause of the hang in #1268 - it was caused by `select count(*) from SpatialIndex` interacting badly with the `set_progress_handler()` mechanism I was using to implement query time limits. #1271 has a replacement for that using `asyncio.wait_for()` and `conn.interrupt()` which should resolve the SpatiaLite issue too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1268#issuecomment-804261915,https://api.github.com/repos/simonw/datasette/issues/1268,804261915,MDEyOklzc3VlQ29tbWVudDgwNDI2MTkxNQ==,9599,simonw,2021-03-22T17:41:12Z,2021-03-22T17:41:12Z,OWNER,"Closing this because I've figured out the root of the problem now, and I have a potential solution.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1269#issuecomment-804261610,https://api.github.com/repos/simonw/datasette/issues/1269,804261610,MDEyOklzc3VlQ29tbWVudDgwNDI2MTYxMA==,9599,simonw,2021-03-22T17:40:41Z,2021-03-22T17:40:41Z,OWNER,"#1270 looks promising, and I don't want to leave open a security hole where someone could potentially hang Datasette with a nasty `count(*)` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837348479,Don't attempt to run count(*) against virtual tables,
https://github.com/simonw/datasette/issues/1270#issuecomment-804255633,https://api.github.com/repos/simonw/datasette/issues/1270,804255633,MDEyOklzc3VlQ29tbWVudDgwNDI1NTYzMw==,9599,simonw,2021-03-22T17:32:02Z,2021-03-22T17:32:08Z,OWNER,Confirmed that the `interrupt()` based cancellation mechanism fixes the SpatiaLite issue in #1268!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),
https://github.com/simonw/datasette/issues/1270#issuecomment-803834784,https://api.github.com/repos/simonw/datasette/issues/1270,803834784,MDEyOklzc3VlQ29tbWVudDgwMzgzNDc4NA==,9599,simonw,2021-03-22T07:31:57Z,2021-03-22T16:22:19Z,OWNER,"I think the implementation for this goes here: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L146-L157

I figured out a similar pattern in `datasette-ripgrep` here: https://github.com/simonw/datasette-ripgrep/blob/0.7/datasette_ripgrep/__init__.py#L63-L71","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),
https://github.com/simonw/datasette/issues/1268#issuecomment-803802957,https://api.github.com/repos/simonw/datasette/issues/1268,803802957,MDEyOklzc3VlQ29tbWVudDgwMzgwMjk1Nw==,9599,simonw,2021-03-22T06:38:14Z,2021-03-22T06:38:14Z,OWNER,"Also worth trying is to change this code:
```python
     n = 1000 
     if ms < 50: 
         n = 1 
```
What happens with `n = 10` instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1249#issuecomment-803700940,https://api.github.com/repos/simonw/datasette/issues/1249,803700940,MDEyOklzc3VlQ29tbWVudDgwMzcwMDk0MA==,9599,simonw,2021-03-22T01:14:24Z,2021-03-22T01:14:24Z,OWNER,I tried that with just `python3-pip` (removing `libsqlite3-mod-spatialite`) and got 435MB.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803700626,https://api.github.com/repos/simonw/datasette/issues/1249,803700626,MDEyOklzc3VlQ29tbWVudDgwMzcwMDYyNg==,9599,simonw,2021-03-22T01:13:04Z,2021-03-22T01:13:04Z,OWNER,"Building a Dockerfile containing just `FROM ubuntu:20.10` gave me `79.5MB`.

Building this one:
```dockerfile
FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && \
    apt install -y python3-pip libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/
```
Resulted in a 515MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1269#issuecomment-803785808,https://api.github.com/repos/simonw/datasette/issues/1269,803785808,MDEyOklzc3VlQ29tbWVudDgwMzc4NTgwOA==,9599,simonw,2021-03-22T06:00:53Z,2021-03-22T06:00:53Z,OWNER,This may not be necessary if using `.interrupt() for SQLite timeouts in #1270 works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837348479,Don't attempt to run count(*) against virtual tables,
https://github.com/simonw/datasette/issues/1268#issuecomment-803784902,https://api.github.com/repos/simonw/datasette/issues/1268,803784902,MDEyOklzc3VlQ29tbWVudDgwMzc4NDkwMg==,9599,simonw,2021-03-22T05:59:06Z,2021-03-22T05:59:06Z,OWNER,"Even if I implement that workaround in #1269 I'm concerned that this could still allow users to deliberately crash Datasette (if it's running SpatiaLite 5.0) by executing `select count(*) from SpatialIndex`.

That `interrupt` timeout mechanism is worth digging into further.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803782705,https://api.github.com/repos/simonw/datasette/issues/1268,803782705,MDEyOklzc3VlQ29tbWVudDgwMzc4MjcwNQ==,9599,simonw,2021-03-22T05:54:19Z,2021-03-22T05:54:19Z,OWNER,"Got two new TILs out of this:

* [Tracing every executed Python statement](https://til.simonwillison.net/python/tracing-every-statement)
* [Running gdb against a Python process in a running Docker container](https://til.simonwillison.net/docker/gdb-python-docker)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803777724,https://api.github.com/repos/simonw/datasette/issues/1268,803777724,MDEyOklzc3VlQ29tbWVudDgwMzc3NzcyNA==,9599,simonw,2021-03-22T05:42:50Z,2021-03-22T05:43:23Z,OWNER,"<img width=""1129"" alt=""tuscany_housenumbers__select___from_sqlite_master_where_sql_like__create_virtual_table__"" src=""https://user-images.githubusercontent.com/9599/111945249-99809380-8a96-11eb-871b-9a1509360bcf.png"">

If I want to avoid counting virtual tables, I need to detect which tables are virtual tables.

The safest way to do this is probably to pull the `sql` for every table and then, in Python, check for values that start with `create virtual table` after converting to lower case, using any number of spaces.

This would catch things like ` CREATE       virtual      TABLE` which might be missed by a SQL `like` query.   ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803775121,https://api.github.com/repos/simonw/datasette/issues/1268,803775121,MDEyOklzc3VlQ29tbWVudDgwMzc3NTEyMQ==,9599,simonw,2021-03-22T05:36:26Z,2021-03-22T05:36:26Z,OWNER,So one fix could be to avoid running counts for anything that turns out to be a virtual table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803774926,https://api.github.com/repos/simonw/datasette/issues/1268,803774926,MDEyOklzc3VlQ29tbWVudDgwMzc3NDkyNg==,9599,simonw,2021-03-22T05:35:56Z,2021-03-22T05:35:56Z,OWNER,That's in this code here: https://github.com/simonw/datasette/blob/c4f1ec7f33fd7d5b93f0f895dafb5351cc3bfc5b/datasette/database.py#L221-L241,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803774518,https://api.github.com/repos/simonw/datasette/issues/1268,803774518,MDEyOklzc3VlQ29tbWVudDgwMzc3NDUxOA==,9599,simonw,2021-03-22T05:34:57Z,2021-03-22T05:34:57Z,OWNER,"... and sure enough, adding this code fixed the problem:
```diff
diff --git a/datasette/database.py b/datasette/database.py
index 3579cce..b466b12 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -224,6 +226,9 @@ class Database:
         # Try to get counts for each table, $limit timeout for each count
         counts = {}
         for table in await self.table_names():
+            if table == ""SpatialIndex"":
+                counts[table] = 0
+                continue
             try:
                 table_count = (
                     await self.execute(
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803773484,https://api.github.com/repos/simonw/datasette/issues/1268,803773484,MDEyOklzc3VlQ29tbWVudDgwMzc3MzQ4NA==,9599,simonw,2021-03-22T05:32:29Z,2021-03-22T05:32:29Z,OWNER,"To figure out which SQL query triggers the problem I added this code to write to a log file:
```python
            with sqlite_timelimit(conn, time_limit_ms):
                try:
                    cursor = conn.cursor()
                    with open(""/tmp/sql.log"", ""ab"", buffering=0) as fp:
                        fp.write((""{}: {}\n"".format(sql, params)).encode(""utf-8""))
                    cursor.execute(sql, params if params is not None else {})
```
I had to use `ab` binary mode because Python doesn't allow `buffering=0` for non-binary file operations.

With the log enabled, I used `docker exec -it 589ae68de943 bash` to attach to the running container and `tail -f /tmp/sql.log` to see the logs. Here's where it broke:

```
select count(*) from [idx_civici_geom_parent]: None
select count(*) from [sqlite_stat1]: None
select count(*) from [sqlite_stat3]: None
select count(*) from [SpatialIndex]: None
```
So attempting to run a `count(*)` against the `SpatialIndex` virtual table is the thing that triggers the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803764919,https://api.github.com/repos/simonw/datasette/issues/1268,803764919,MDEyOklzc3VlQ29tbWVudDgwMzc2NDkxOQ==,9599,simonw,2021-03-22T05:11:11Z,2021-03-22T05:11:11Z,OWNER,"Maybe I could implement SQLite query timeouts using the `interrupt()` method instead of the progress handler hack I'm currently using?

https://stackoverflow.com/questions/43240496/python-sqlite3-how-to-quickly-and-cleanly-interrupt-long-running-query-with-e has some tips.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803764200,https://api.github.com/repos/simonw/datasette/issues/1268,803764200,MDEyOklzc3VlQ29tbWVudDgwMzc2NDIwMA==,9599,simonw,2021-03-22T05:09:13Z,2021-03-22T05:09:13Z,OWNER,"I tried building a container where the `conn.set_progress_handler(handler, n)` line was commented out... and it fixed the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803762969,https://api.github.com/repos/simonw/datasette/issues/1268,803762969,MDEyOklzc3VlQ29tbWVudDgwMzc2Mjk2OQ==,9599,simonw,2021-03-22T05:05:51Z,2021-03-22T05:05:51Z,OWNER,I had to run `docker kill 16197781a7b5` to kill the broken container - Ctrl+C in the Datasette console window didn't do anything.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803762609,https://api.github.com/repos/simonw/datasette/issues/1268,803762609,MDEyOklzc3VlQ29tbWVudDgwMzc2MjYwOQ==,9599,simonw,2021-03-22T05:05:00Z,2021-03-22T05:05:00Z,OWNER,"Using https://til.simonwillison.net/docker/attach-bash-to-running-container - I figured out how to run `gdb`. I had to use `--privileged` here because otherwise `gdb` showed a ""Could not attach to process"" error.
```
docker exec --privileged -it 16197781a7b5 bash
# apt-get install gdb python3-dbg
# gdb /usr/bin/python3 -p 20
```
This paused the process. I tried running this:
```
(gdb) py-bt
Traceback (most recent call first):
  File ""/usr/lib/python3.8/asyncio/base_events.py"", line 1845, in _run_once
    if handle._cancelled:
  File ""/usr/lib/python3.8/asyncio/base_events.py"", line 570, in run_forever
    self._run_once()
  File ""/usr/lib/python3.8/asyncio/base_events.py"", line 603, in run_until_complete
    self.run_forever()
  File ""/usr/local/lib/python3.8/dist-packages/uvicorn/server.py"", line 49, in run
    loop.run_until_complete(self.serve(sockets=sockets))
  File ""/usr/local/lib/python3.8/dist-packages/uvicorn/main.py"", line 386, in run
    server.run()
  File ""/usr/local/lib/python3.8/dist-packages/datasette/cli.py"", line 575, in serve
    uvicorn.run(ds.app(), **uvicorn_kwargs)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/usr/local/bin/datasette"", line 8, in <module>
    sys.exit(cli())
  <built-in method exec of module object at remote 0x7f0981a280e0>
  File ""/usr/lib/python3.8/trace.py"", line 450, in runctx
    exec(cmd, globals, locals)
  File ""/usr/lib/python3.8/trace.py"", line 6632, in main
  File ""/usr/lib/python3.8/trace.py"", line 756, in <module>
    main()
  <built-in method exec of module object at remote 0x7f0981a280e0>
  File ""/usr/lib/python3.8/runpy.py"", line 343, in _run_code
  File ""/usr/lib/python3.8/runpy.py"", line 450, in _run_module_as_main
```
Not sure if that's useful or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803759051,https://api.github.com/repos/simonw/datasette/issues/1268,803759051,MDEyOklzc3VlQ29tbWVudDgwMzc1OTA1MQ==,9599,simonw,2021-03-22T04:55:22Z,2021-03-22T04:55:22Z,OWNER,So I think there's a bug in the way the `set_progress_handler()` mechanism works when used in conjunction with SpatiaLite 5.0 on Linux.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803758793,https://api.github.com/repos/simonw/datasette/issues/1268,803758793,MDEyOklzc3VlQ29tbWVudDgwMzc1ODc5Mw==,9599,simonw,2021-03-22T04:54:32Z,2021-03-22T04:54:32Z,OWNER,"Hitting http://localhost:8001/tuscany_housenumbers triggers the bug. It gets stuck in a loop that looks like this:

<img width=""381"" alt=""datasette_—_root_16197781a7b5____—_com_docker_cli_◂_docker_run_-it_-p_8001_8001_-v___Dropbox_Development_datasette__mnt_datasette-spatialite_latest_bash_—_195×48_and_getIncidentsGit_—_-zsh_—_162×60"" src=""https://user-images.githubusercontent.com/9599/111942115-e0b75600-8a8f-11eb-8b6d-5c37240ce2f8.png"">

Which looks to me like this code: https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L139-L158","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803758182,https://api.github.com/repos/simonw/datasette/issues/1268,803758182,MDEyOklzc3VlQ29tbWVudDgwMzc1ODE4Mg==,9599,simonw,2021-03-22T04:52:15Z,2021-03-22T04:52:15Z,OWNER,Hitting http://localhost:8001/ successfully shows the homepage (after a lot more scrolling).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803757746,https://api.github.com/repos/simonw/datasette/issues/1268,803757746,MDEyOklzc3VlQ29tbWVudDgwMzc1Nzc0Ng==,9599,simonw,2021-03-22T04:50:40Z,2021-03-22T04:51:52Z,OWNER,"Here's a fun debugging trick:

    docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash
    root@16197781a7b5:/# python3 -m trace --trace $(which datasette) \
      -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite \
      --load-extension=spatialite

A huge amount of stuff scrolls past as Datasette starts up, since we are tracing every executed line of Python.

After about a minute it's finished starting and gets to this point:

```
selectors.py(452):             if timeout is None:
selectors.py(454):             elif timeout <= 0:
selectors.py(459):                 timeout = math.ceil(timeout * 1e3) * 1e-3
selectors.py(464):             max_ev = max(len(self._fd_to_key), 1)
selectors.py(466):             ready = []
selectors.py(467):             try:
selectors.py(468):                 fd_event_list = self._selector.poll(timeout, max_ev)
```
Now  I can make some HTTP requests against it.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803756495,https://api.github.com/repos/simonw/datasette/issues/1268,803756495,MDEyOklzc3VlQ29tbWVudDgwMzc1NjQ5NQ==,9599,simonw,2021-03-22T04:46:04Z,2021-03-22T04:46:04Z,OWNER,`gdb` may be able to help debug this: https://www.podoliaka.org/2016/04/10/debugging-cpython-gdb/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1249#issuecomment-803755698,https://api.github.com/repos/simonw/datasette/issues/1249,803755698,MDEyOklzc3VlQ29tbWVudDgwMzc1NTY5OA==,9599,simonw,2021-03-22T04:43:02Z,2021-03-22T04:43:02Z,OWNER,I'll spin off a separate ticket to investigate the hang.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1267#issuecomment-803754226,https://api.github.com/repos/simonw/datasette/issues/1267,803754226,MDEyOklzc3VlQ29tbWVudDgwMzc1NDIyNg==,9599,simonw,2021-03-22T04:37:26Z,2021-03-22T04:37:26Z,OWNER,"Thanks for doing this - I've used alternativeto.net a bunch in the past, it's great to see Datasette listed there.

This does raise some interesting philosophical questions: three years into the project I'm still not entirely sure what Datasette competes with! Could be SQLite desktop packages, could be visualization software like Tableau, could even be something like Airtable (given a few more plugins).

It will be interesting to see how the alternativeto listing evolves, maybe it will help me answer that question!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837208901,Update Datasette alternativeto listening with details,
https://github.com/simonw/datasette/issues/1249#issuecomment-803753388,https://api.github.com/repos/simonw/datasette/issues/1249,803753388,MDEyOklzc3VlQ29tbWVudDgwMzc1MzM4OA==,9599,simonw,2021-03-22T04:34:20Z,2021-03-22T04:35:10Z,OWNER,"Well this is frustrating. I finally found a Dockerfile that worked and installed an Ubuntu pre-compiled SpatiaLite module that would load...
```dockerfile
FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM ubuntu:20.10

RUN apt update && \
    apt install -y python3-pip && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```
(Which produced a 550MB image)

And when I ran Datasette I got that same error where the database listing page hangs!
```
docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite --load-extension=spatialite
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803751068,https://api.github.com/repos/simonw/datasette/issues/1249,803751068,MDEyOklzc3VlQ29tbWVudDgwMzc1MTA2OA==,9599,simonw,2021-03-22T04:26:45Z,2021-03-22T04:26:45Z,OWNER,"Here's why:
```
datasette % docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash                                                      
root@3430352ff378:/# datasette
bash: /usr/local/bin/datasette: /usr/bin/python3: bad interpreter: No such file or directory
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803750617,https://api.github.com/repos/simonw/datasette/issues/1249,803750617,MDEyOklzc3VlQ29tbWVudDgwMzc1MDYxNw==,9599,simonw,2021-03-22T04:25:14Z,2021-03-22T04:25:14Z,OWNER,"Got this error attempting to run Datasette (with or without SpatiaLite):
```
standard_init_linux.go:219: exec user process caused: no such file or directory
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803750399,https://api.github.com/repos/simonw/datasette/issues/1249,803750399,MDEyOklzc3VlQ29tbWVudDgwMzc1MDM5OQ==,9599,simonw,2021-03-22T04:24:25Z,2021-03-22T04:24:25Z,OWNER,"I'll try using `ubuntu:20.10` for everything:
```dockerfile
FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM ubuntu:20.10 as build

RUN apt update && \
    apt install -y python3-pip && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM ubuntu:20.10

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803749831,https://api.github.com/repos/simonw/datasette/issues/1249,803749831,MDEyOklzc3VlQ29tbWVudDgwMzc0OTgzMQ==,9599,simonw,2021-03-22T04:22:35Z,2021-03-22T04:22:35Z,OWNER,"I tried copying just the `mod_spatialite.so` file:
```dockerfile
FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM python:3.9.2-slim as build

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM python:3.9.2-slim

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```
But when I ran Datasette with `--load-extension=spatialite` I got this:
```
  File ""/usr/local/lib/python3.9/site-packages/datasette/database.py"", line 151, in in_thread
    self.ds._prepare_connection(conn, self.name)
  File ""/usr/local/lib/python3.9/site-packages/datasette/app.py"", line 502, in _prepare_connection
    conn.execute(f""SELECT load_extension('{extension}')"")
sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803748469,https://api.github.com/repos/simonw/datasette/issues/1249,803748469,MDEyOklzc3VlQ29tbWVudDgwMzc0ODQ2OQ==,9599,simonw,2021-03-22T04:17:51Z,2021-03-22T04:17:51Z,OWNER,"... except my clever image using SpatiaLite installed for Ubuntu doesn't actually work:

```
datasette % docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
  File ""/usr/local/lib/python3.9/sqlite3/dbapi2.py"", line 27, in <module>
    from _sqlite3 import *
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /usr/lib/x86_64-linux-gnu/libsqlite3.so.0)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803748158,https://api.github.com/repos/simonw/datasette/issues/1249,803748158,MDEyOklzc3VlQ29tbWVudDgwMzc0ODE1OA==,9599,simonw,2021-03-22T04:16:57Z,2021-03-22T04:16:57Z,OWNER,"Which is great, because the image on Docker Hub right now is 383MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803747701,https://api.github.com/repos/simonw/datasette/issues/1249,803747701,MDEyOklzc3VlQ29tbWVudDgwMzc0NzcwMQ==,9599,simonw,2021-03-22T04:15:40Z,2021-03-22T04:15:40Z,OWNER,"Here's a trick: install SpatiaLite in `ubuntu:20.10` and then copy it into the final `python:3.9.2-slim` image.

```dockerfile
FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM python:3.9.2-slim as build

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM python:3.9.2-slim

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```
That produced a 265MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803698983,https://api.github.com/repos/simonw/datasette/issues/1249,803698983,MDEyOklzc3VlQ29tbWVudDgwMzY5ODk4Mw==,9599,simonw,2021-03-22T01:05:36Z,2021-03-22T01:06:23Z,OWNER,"It's pretty big though. I tried this version which avoids copying junk from my laptop in:

```dockerfile
FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean

RUN pip install datasette

EXPOSE 8001
CMD [""datasette""]
```
And got this:
```
datasette % docker images datasette-spatialite    
REPOSITORY             TAG       IMAGE ID       CREATED         SIZE
datasette-spatialite   latest    0796950653c2   2 seconds ago   528MB
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803698168,https://api.github.com/repos/simonw/datasette/issues/1249,803698168,MDEyOklzc3VlQ29tbWVudDgwMzY5ODE2OA==,9599,simonw,2021-03-22T01:02:02Z,2021-03-22T01:02:30Z,OWNER,"This is the shortest Dockerfile that appeared to give me a working SpatiaLite module:
```dockerfile
FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean

# Add local code to the image instead of fetching from pypi.
COPY . /datasette

RUN pip install /datasette

RUN rm -rf /datasette

EXPOSE 8001
CMD [""datasette""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803697546,https://api.github.com/repos/simonw/datasette/issues/1249,803697546,MDEyOklzc3VlQ29tbWVudDgwMzY5NzU0Ng==,9599,simonw,2021-03-22T00:59:47Z,2021-03-22T00:59:47Z,OWNER,"To debug I'm running:

    docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash

This gets me a shell I can use.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803697211,https://api.github.com/repos/simonw/datasette/issues/1249,803697211,MDEyOklzc3VlQ29tbWVudDgwMzY5NzIxMQ==,9599,simonw,2021-03-22T00:58:01Z,2021-03-22T00:58:01Z,OWNER,"I'm messing around with the `Dockerfile` and after each change I'm running:

    docker build . -t datasette-spatialite

And then:

    docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803694661,https://api.github.com/repos/simonw/datasette/issues/1249,803694661,MDEyOklzc3VlQ29tbWVudDgwMzY5NDY2MQ==,9599,simonw,2021-03-22T00:46:49Z,2021-03-22T00:46:49Z,OWNER,Actually for the loadable module I think I need https://packages.ubuntu.com/groovy/libsqlite3-mod-spatialite,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803694436,https://api.github.com/repos/simonw/datasette/issues/1249,803694436,MDEyOklzc3VlQ29tbWVudDgwMzY5NDQzNg==,9599,simonw,2021-03-22T00:46:00Z,2021-03-22T00:46:00Z,OWNER,So I'm going to try `20.10` and see where that gets me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803694359,https://api.github.com/repos/simonw/datasette/issues/1249,803694359,MDEyOklzc3VlQ29tbWVudDgwMzY5NDM1OQ==,9599,simonw,2021-03-22T00:45:47Z,2021-03-22T00:45:47Z,OWNER,https://pythonspeed.com/articles/base-image-python-docker-images/ suggests using `python:3.9-slim-buster` or `ubuntu:20.04` - but 20.04 is focal which still has SpatiaLite `4.3.0a-6build1` - It's `20.10` that has 5.0: https://packages.ubuntu.com/groovy/libspatialite-dev,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803693181,https://api.github.com/repos/simonw/datasette/issues/1249,803693181,MDEyOklzc3VlQ29tbWVudDgwMzY5MzE4MQ==,9599,simonw,2021-03-22T00:41:02Z,2021-03-22T00:41:02Z,OWNER,Debian sid has it too: https://packages.debian.org/sid/libspatialite-dev,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803692673,https://api.github.com/repos/simonw/datasette/issues/1249,803692673,MDEyOklzc3VlQ29tbWVudDgwMzY5MjY3Mw==,9599,simonw,2021-03-22T00:38:42Z,2021-03-22T00:38:42Z,OWNER,Ubuntu Groovy has a package for SpatiaLite 5 - I could try using that instead: https://packages.ubuntu.com/groovy/libspatialite-dev,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803691236,https://api.github.com/repos/simonw/datasette/issues/1249,803691236,MDEyOklzc3VlQ29tbWVudDgwMzY5MTIzNg==,9599,simonw,2021-03-22T00:32:03Z,2021-03-22T00:32:03Z,OWNER,"Here's something odd: when I run `datasette tuscany_housenumbers.sqlite --load-extension=spatialite` on macOS against SpatiaLite installed using Homebrew (which reports `""spatialite"": ""5.0.0""` on the `/-/versions` page) I don't get any weird errors at all, everything works fine.

But when I tried compiling SpatiaLite inside the Docker container I had hanging errors on some pages.

This is using https://www.gaia-gis.it/gaia-sins/knn/tuscany_housenumbers.7z from the SpatiaLite KNN tutorial at https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1259#issuecomment-803674728,https://api.github.com/repos/simonw/datasette/issues/1259,803674728,MDEyOklzc3VlQ29tbWVudDgwMzY3NDcyOA==,9599,simonw,2021-03-21T22:55:31Z,2021-03-21T22:55:31Z,OWNER,CTEs were added in 2014-02-03 SQLite 3.8.3 - so I think it's OK to depend on them for Datasette.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830567275,Research using CTEs for faster facet counts,
https://github.com/simonw/datasette/issues/647#issuecomment-803673225,https://api.github.com/repos/simonw/datasette/issues/647,803673225,MDEyOklzc3VlQ29tbWVudDgwMzY3MzIyNQ==,9599,simonw,2021-03-21T22:44:19Z,2021-03-21T22:44:19Z,OWNER,"Now that I'm looking at refactoring how views work in #878 it's clear that the gnarliest, most convoluted code I need to deal with relates to this old feature.

I'm going to remove it entirely. Any performance enhancement or provides can be achieved just as well by using regular URLs and a caching proxy.

I may provide a 404 handling plugin that attempts to rewrite old URLs that used this mechanism, but I won't do any more than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin,
https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803501756,https://api.github.com/repos/simonw/sqlite-utils/issues/249,803501756,MDEyOklzc3VlQ29tbWVudDgwMzUwMTc1Ng==,9599,simonw,2021-03-21T02:33:45Z,2021-03-21T02:33:45Z,OWNER,"Did you run `enable-fts` before you inserted the data?

If so you'll need to run `populate-fts` after the insert to populate the FTS index.

A better solution may be to add `--create-triggers` to the `enable-fts` command to add triggers that will automatically keep the index updated as you insert new records.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836963850,Full text search possibly broken?,
https://github.com/simonw/datasette/issues/878#issuecomment-803473015,https://api.github.com/repos/simonw/datasette/issues/878,803473015,MDEyOklzc3VlQ29tbWVudDgwMzQ3MzAxNQ==,9599,simonw,2021-03-20T22:33:05Z,2021-03-20T22:33:05Z,OWNER,"Things this mechanism needs to be able to support:

- Returning a default JSON representation
- Defining ""extra"" JSON representations blocks, which can be requested using `?_extra=`
- Returning rendered HTML, based on the default JSON + one or more extras + a template
- Using Datasette output renderers to return e.g. CSV data
- Potentially also supporting streaming output renderers for streaming CSV/TSV/JSON-nl etc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-803472595,https://api.github.com/repos/simonw/datasette/issues/878,803472595,MDEyOklzc3VlQ29tbWVudDgwMzQ3MjU5NQ==,9599,simonw,2021-03-20T22:28:12Z,2021-03-20T22:28:12Z,OWNER,"Another idea I had: a view is a class that takes the `datasette` instance in its constructor, and defines a `__call__` method that accepts a request and returns a response. Except `await __call__` looks like it might be a bit messy, discussion in https://github.com/encode/starlette/issues/886","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-803472278,https://api.github.com/repos/simonw/datasette/issues/878,803472278,MDEyOklzc3VlQ29tbWVudDgwMzQ3MjI3OA==,9599,simonw,2021-03-20T22:25:04Z,2021-03-20T22:25:04Z,OWNER,"I came up with a slightly wild idea for this that would involve pytest-style dependency injection.

Prototype here: https://gist.github.com/simonw/496b24fdad44f6f8b7237fe394a0ced7

Copying from my private notes:

> Use the lazy evaluated DI mechanism to break up table view into different pieces eg for faceting
> 
> Use that to solve JSON vs HTML views
> 
> Oh here's an idea: what if the various components of the table view were each defined  as async functions.... and then executed using asyncio.gather in order to run the SQL queries in parallel? Then try benchmarking with different numbers of threads?
> 
> The async_call_with_arguments function could do this automatically for any awaitable dependencies
> 
> This would give me massively parallel dependency injection
> 
> (I could build an entire framework around this and call it c64)
> 
> Idea: arguments called eg ""count"" are executed and the result passed to the function. If called count_fn then a reference to the not-yet-called function is passed instead 
> 
> I'm not going to completely combine the views mechanism and the render hooks. Instead, the core view will define a bunch of functions used to compose the page and the render hook will have conditional access to those functions - which will otherwise be asyncio.gather executed directly by the HTML page version
> 
> Using asyncio.gather to execute facets and suggest facets in parallel would be VERY interesting 
> 
> suggest facets should be VERY cachable - doesn't matter if it's wrong unlike actual facets themselves
> 
> What if all Datasette views were defined in terms of dependency injection - and those dependency functions could themselves depend on others just like pytest fixtures. Everything would become composable and async stuff could execute in parallel
> 
> FURTHER IDEA: use this for the ?_extra= mechanism as well.
> 
> Any view in Datasette can be defined as a collection of named keys. Each of those keys maps to a function or an async function that accepts as input other named keys, using DI to handle them.
> 
> The HTML view is a defined function. So are the other outputs.
> 
> Default original inputs include “request” and “datasette”.
> 
> So… maybe a view function is a class methods that use DI. One of those methods as an .html() method used for the default page.
> 
> Output formats are a bit more complicated because they are supposed to be defined separately in plugins. They are unified across query, row and table though.
> 
> I’m going to try breaking up the TableView to see what happens.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-803471917,https://api.github.com/repos/simonw/datasette/issues/878,803471917,MDEyOklzc3VlQ29tbWVudDgwMzQ3MTkxNw==,9599,simonw,2021-03-20T22:21:33Z,2021-03-20T22:21:33Z,OWNER,"This has been blocking things for too long.

If this becomes a documented pattern, things like adding a JSON output to https://github.com/dogsheep/dogsheep-beta becomes easier too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/1258#issuecomment-803471702,https://api.github.com/repos/simonw/datasette/issues/1258,803471702,MDEyOklzc3VlQ29tbWVudDgwMzQ3MTcwMg==,9599,simonw,2021-03-20T22:19:39Z,2021-03-20T22:19:39Z,OWNER,"This is a good idea. I avoided this initially because it should be possible to run a canned query with a parameter set to the empty string, but that view could definitely be smart enough to differentiate between `?sql=...&param=` and `?sql=` with no `param` specified at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421,Allow canned query params to specify default values,
https://github.com/simonw/datasette/issues/782#issuecomment-803469623,https://api.github.com/repos/simonw/datasette/issues/782,803469623,MDEyOklzc3VlQ29tbWVudDgwMzQ2OTYyMw==,9599,simonw,2021-03-20T22:01:23Z,2021-03-20T22:01:23Z,OWNER,"I'm going to keep `?_shape=array` working on the assumption that many existing uses of the Datasette API are already using that option, so it would be nice not to break them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1261#issuecomment-803468314,https://api.github.com/repos/simonw/datasette/issues/1261,803468314,MDEyOklzc3VlQ29tbWVudDgwMzQ2ODMxNA==,9599,simonw,2021-03-20T21:48:48Z,2021-03-20T21:48:48Z,OWNER,That's fixed in this release of `datasette-publish-vercel`: https://github.com/simonw/datasette-publish-vercel/releases/tag/0.9.2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832092321,Some links aren't properly URL encoded.,
https://github.com/simonw/datasette/issues/1261#issuecomment-803466868,https://api.github.com/repos/simonw/datasette/issues/1261,803466868,MDEyOklzc3VlQ29tbWVudDgwMzQ2Njg2OA==,9599,simonw,2021-03-20T21:36:06Z,2021-03-20T21:36:06Z,OWNER,"This isn't a Datasette bug - it's a Vercel bug: https://github.com/simonw/datasette-publish-vercel/issues/28

I'm looking at a fix for that now, so watch that issue for updates.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832092321,Some links aren't properly URL encoded.,
https://github.com/simonw/datasette/issues/1266#issuecomment-803466730,https://api.github.com/repos/simonw/datasette/issues/1266,803466730,MDEyOklzc3VlQ29tbWVudDgwMzQ2NjczMA==,9599,simonw,2021-03-20T21:35:00Z,2021-03-20T21:35:00Z,OWNER,https://docs.datasette.io/en/latest/internals.html#returning-a-response-with-asgi-send-send,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836273891,Documentation for Response.asgi_send(send) method,
https://github.com/simonw/datasette/issues/1265#issuecomment-803130332,https://api.github.com/repos/simonw/datasette/issues/1265,803130332,MDEyOklzc3VlQ29tbWVudDgwMzEzMDMzMg==,9599,simonw,2021-03-19T21:03:09Z,2021-03-19T21:03:09Z,OWNER,This is now available in `datasette-auth-passwords` 0.4! https://github.com/simonw/datasette-auth-passwords/releases/tag/0.4,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",836123030,Support for HTTP Basic Authentication,
https://github.com/simonw/datasette/issues/1262#issuecomment-802099264,https://api.github.com/repos/simonw/datasette/issues/1262,802099264,MDEyOklzc3VlQ29tbWVudDgwMjA5OTI2NA==,9599,simonw,2021-03-18T16:43:09Z,2021-03-18T16:43:09Z,OWNER,"I often find myself wanting this too, when I'm exploring a new dataset.

i agree with Bob that this is a good candidate for a plugin. The plugin system isn't quite setup for this yet though - there isn't an obvious mechanism for adding extra sort orders or other interface elements that manipulate the query used by the table view in some way.

I'm going to promote this issue to status of a plugin hook feature request - I have a hunch that a plugin hook that enables `order by random()` could enable a lot of other useful plugin features too.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",834602299,Plugin hook that could support 'order by random()' for table view,
https://github.com/simonw/datasette/issues/1394#issuecomment-880900534,https://api.github.com/repos/simonw/datasette/issues/1394,880900534,MDEyOklzc3VlQ29tbWVudDg4MDkwMDUzNA==,9599,simonw,2021-07-15T17:58:03Z,2021-07-15T17:58:03Z,OWNER,Started a conversation about this on the SQLite forum: https://sqlite.org/forum/forumpost/2d76f2bcf65d256a?t=h,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by,
https://github.com/simonw/datasette/issues/1396#issuecomment-880967052,https://api.github.com/repos/simonw/datasette/issues/1396,880967052,MDEyOklzc3VlQ29tbWVudDg4MDk2NzA1Mg==,9599,simonw,2021-07-15T19:47:25Z,2021-07-15T19:47:25Z,OWNER,Actually I'm going to close this now and re-open it if the problem occurs again in the future.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880326049,https://api.github.com/repos/simonw/datasette/issues/1396,880326049,MDEyOklzc3VlQ29tbWVudDg4MDMyNjA0OQ==,9599,simonw,2021-07-15T01:50:05Z,2021-07-15T01:50:05Z,OWNER,"I think I made a mistake in this commit: https://github.com/simonw/datasette/commit/0486303b60ce2784fd2e2ecdbecf304b7d6e6659

<img width=""770"" alt=""Explicitly_push_version_tag__refs__1281_·_simonw_datasette_0486303"" src=""https://user-images.githubusercontent.com/9599/125715241-dc96d147-191a-4a69-bdf0-1cb092d97330.png"">

It looks like I copied `$VERSION_TAG` from here - but it's not available in the `publish.yml` flow: https://github.com/simonw/datasette/blob/0486303b60ce2784fd2e2ecdbecf304b7d6e6659/.github/workflows/push_docker_tag.yml#L18-L25","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880325362,https://api.github.com/repos/simonw/datasette/issues/1396,880325362,MDEyOklzc3VlQ29tbWVudDg4MDMyNTM2Mg==,9599,simonw,2021-07-15T01:48:11Z,2021-07-15T01:48:11Z,OWNER,In particular these three lines: https://github.com/simonw/datasette/blob/084cfe1e00e1a4c0515390a513aca286eeea20c2/.github/workflows/publish.yml#L117-L119,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880325004,https://api.github.com/repos/simonw/datasette/issues/1396,880325004,MDEyOklzc3VlQ29tbWVudDg4MDMyNTAwNA==,9599,simonw,2021-07-15T01:47:17Z,2021-07-15T01:47:17Z,OWNER,"This is the part of the publish workflow that failed and threw the ""invalid reference format"" error: https://github.com/simonw/datasette/blob/084cfe1e00e1a4c0515390a513aca286eeea20c2/.github/workflows/publish.yml#L100-L119","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880324637,https://api.github.com/repos/simonw/datasette/issues/1396,880324637,MDEyOklzc3VlQ29tbWVudDg4MDMyNDYzNw==,9599,simonw,2021-07-15T01:46:26Z,2021-07-15T01:46:26Z,OWNER,"I manually published the Docker image using https://github.com/simonw/datasette/actions/workflows/push_docker_tag.yml https://github.com/simonw/datasette/runs/3072505126

The 0.58 release shows up on https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated now - BUT the `latest` tag still points to a version from a month ago.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880374156,https://api.github.com/repos/simonw/datasette/issues/1396,880374156,MDEyOklzc3VlQ29tbWVudDg4MDM3NDE1Ng==,9599,simonw,2021-07-15T04:03:18Z,2021-07-15T04:03:18Z,OWNER,"I fixed  `datasette:latest` by running the following on my laptop:
```
docker pull datasetteproject/datasette:0.58
docker tag datasetteproject/datasette:0.58 datasetteproject/datasette:latest
docker login -u datasetteproject -p ...
docker push datasetteproject/datasette:latest
```
Confirmed on https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated that `datasette:latest` and `datasette:0.58` both now have the same digest of `3b5ba478040e`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880372149,https://api.github.com/repos/simonw/datasette/issues/1396,880372149,MDEyOklzc3VlQ29tbWVudDg4MDM3MjE0OQ==,9599,simonw,2021-07-15T03:56:49Z,2021-07-15T03:56:49Z,OWNER,I'm going to leave this open until I next successfully publish a new version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1394#issuecomment-880287483,https://api.github.com/repos/simonw/datasette/issues/1394,880287483,MDEyOklzc3VlQ29tbWVudDg4MDI4NzQ4Mw==,9599,simonw,2021-07-15T00:01:47Z,2021-07-15T00:01:47Z,OWNER,"I wrote this code:
```python
_order_by_re = re.compile(r""(^.*) order by [a-zA-Z_][a-zA-Z0-9_]+( desc)?$"", re.DOTALL)
_order_by_braces_re = re.compile(r""(^.*) order by \[[^\]]+\]( desc)?$"", re.DOTALL)


def strip_order_by(sql):
    for regex in (_order_by_re, _order_by_braces_re):
        match = regex.match(sql)
        if match is not None:
            return match.group(1)
    return sql

@pytest.mark.parametrize(
    ""sql,expected"",
    [
        (""blah"", ""blah""),
        (""select * from foo"", ""select * from foo""),
        (""select * from foo order by bah"", ""select * from foo""),
        (""select * from foo order by bah desc"", ""select * from foo""),
        (""select * from foo order by [select]"", ""select * from foo""),
        (""select * from foo order by [select] desc"", ""select * from foo""),
    ],
)
def test_strip_order_by(sql, expected):
    assert strip_order_by(sql) == expected
```
But it turns out I don't need it! The SQL that is passed to the facet class is created by this code: https://github.com/simonw/datasette/blob/ba11ef27edd6981eeb26d7ecf5aa236707f5f8ce/datasette/views/table.py#L677-L684

And the only place that uses that `sql_no_limit` variable is here: https://github.com/simonw/datasette/blob/ba11ef27edd6981eeb26d7ecf5aa236707f5f8ce/datasette/views/table.py#L733-L745

So I can change that to `sql_no_limit_no_order` and fix the bug that way instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by,
https://github.com/simonw/datasette/issues/1394#issuecomment-880278256,https://api.github.com/repos/simonw/datasette/issues/1394,880278256,MDEyOklzc3VlQ29tbWVudDg4MDI3ODI1Ng==,9599,simonw,2021-07-14T23:35:18Z,2021-07-14T23:35:18Z,OWNER,"The challenge here is that faceting doesn't currently modify the inner SQL at all - it wraps it so that it can work against any SQL statement (though Datasette itself does not yet take advantage of that ability, only offering faceting on table pages).

So just removing the order by wouldn't be appropriate if the inner query looked something like this:

```sql
select * from items order by created desc limit 100
```
Since the intent there would be to return facet counts against only the most recent 100 items.

In SQLite the `limit` has to come after the `order by` though, so the fix here could be as easy as using a regular expression to identify queries that end with `order by COLUMN (desc)?` and stripping off that clause.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880259255,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880259255,MDEyOklzc3VlQ29tbWVudDg4MDI1OTI1NQ==,9599,simonw,2021-07-14T22:48:41Z,2021-07-14T22:48:41Z,OWNER,Should also take advantage of `.mode tabs` to support `sqlite-utils insert blah.db blah blah.csv --tsv --fast`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880257587,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880257587,MDEyOklzc3VlQ29tbWVudDg4MDI1NzU4Nw==,9599,simonw,2021-07-14T22:44:05Z,2021-07-14T22:44:05Z,OWNER,"https://unix.stackexchange.com/a/642364 suggests you can also use this to import from stdin, like so:

    sqlite3 -csv $database_file_name "".import '|cat -' $table_name""

Here the `sqlite3 -csv` is an alternative to using `.mode csv`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880256865,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880256865,MDEyOklzc3VlQ29tbWVudDg4MDI1Njg2NQ==,9599,simonw,2021-07-14T22:42:11Z,2021-07-14T22:42:11Z,OWNER,"Potential workaround for missing `--skip` implementation is that the filename can be a command instead, so maybe it could shell out to `tail -n +1 filename`:

> The source argument is the name of a file to be read or, if it begins with a ""|"" character, specifies a command which will be run to produce the input CSV data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880256058,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880256058,MDEyOklzc3VlQ29tbWVudDg4MDI1NjA1OA==,9599,simonw,2021-07-14T22:40:01Z,2021-07-14T22:40:47Z,OWNER,"Full docs here: https://www.sqlite.org/draft/cli.html#csv

One catch: how this works has changed in recent SQLite versions: https://www.sqlite.org/changes.html

- 2020-12-01 (3.34.0) - ""Table name quoting works correctly for the .import dot-command""
- 2020-05-22 (3.32.0) - ""Add options to the .import command: --csv, --ascii, --skip""
- 2017-08-01 (3.20.0) - "" The "".import"" command ignores an initial UTF-8 BOM.""

The ""skip"" feature is particularly important to understand. https://www.sqlite.org/draft/cli.html#csv says:

> There are two cases to consider: (1) Table ""tab1"" does not previously exist and (2) table ""tab1"" does already exist.
>
> In the first case, when the table does not previously exist, the table is automatically created and the content of the first row of the input CSV file is used to determine the name of all the columns in the table. In other words, if the table does not previously exist, the first row of the CSV file is interpreted to be column names and the actual data starts on the second row of the CSV file.
>
> For the second case, when the table already exists, every row of the CSV file, including the first row, is assumed to be actual content. If the CSV file contains an initial row of column labels, you can cause the .import command to skip that initial row using the ""--skip 1"" option.

But the `--skip 1` option is only available in 3.32.0 and higher.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/datasette/issues/268#issuecomment-880153069,https://api.github.com/repos/simonw/datasette/issues/268,880153069,MDEyOklzc3VlQ29tbWVudDg4MDE1MzA2OQ==,9599,simonw,2021-07-14T19:31:00Z,2021-07-14T19:31:00Z,OWNER,"... though interestingly I can't replicate that error on `latest.datasette.io` - https://latest.datasette.io/fixtures/searchable?_search=park.&_searchmode=raw

That's running https://latest.datasette.io/-/versions SQLite 3.35.4 whereas https://www.niche-museums.com/-/versions is running 3.27.2 (the most recent version available with Vercel) - but there's nothing in the SQLite changelog between those two versions that suggests changes to how the FTS5 parser works. https://www.sqlite.org/changes.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/268#issuecomment-880150755,https://api.github.com/repos/simonw/datasette/issues/268,880150755,MDEyOklzc3VlQ29tbWVudDg4MDE1MDc1NQ==,9599,simonw,2021-07-14T19:26:47Z,2021-07-14T19:29:08Z,OWNER,"> What are the side-effects of turning that on in the query string, or even by default as you suggested? I see that you stated in the docs... ""to ensure they do not cause any confusion for users who are not aware of them"", but I'm not sure what those could be.

Mainly that it's possible to generate SQL queries that crash with an error. This was the example that convinced me to default to escaping:

- https://www.niche-museums.com/browse/museums?_search=park.&_searchmode=raw (returns `fts5: syntax error near "".""`)

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/pull/1393#issuecomment-879309636,https://api.github.com/repos/simonw/datasette/issues/1393,879309636,MDEyOklzc3VlQ29tbWVudDg3OTMwOTYzNg==,9599,simonw,2021-07-13T18:32:25Z,2021-07-13T18:32:25Z,OWNER,Thanks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941412189,Update deploying.rst,
https://github.com/simonw/datasette/pull/1392#issuecomment-879277953,https://api.github.com/repos/simonw/datasette/issues/1392,879277953,MDEyOklzc3VlQ29tbWVudDg3OTI3Nzk1Mw==,9599,simonw,2021-07-13T17:42:31Z,2021-07-13T17:42:31Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941403676,Update deploying.rst,
https://github.com/simonw/datasette/issues/511#issuecomment-877835171,https://api.github.com/repos/simonw/datasette/issues/511,877835171,MDEyOklzc3VlQ29tbWVudDg3NzgzNTE3MQ==,9599,simonw,2021-07-11T17:23:05Z,2021-07-11T17:23:05Z,OWNER,"    == 87 failed, 819 passed, 7 skipped, 29 errors in 2584.85s (0:43:04) ==

https://github.com/simonw/datasette/runs/3038188870?check_suite_focus=true

Full copy of log here: https://gist.github.com/simonw/4b1fdd24496b989fca56bc757be345ad","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877726495,https://api.github.com/repos/simonw/datasette/issues/511,877726495,MDEyOklzc3VlQ29tbWVudDg3NzcyNjQ5NQ==,9599,simonw,2021-07-11T01:32:27Z,2021-07-11T01:32:27Z,OWNER,"I'm using `pytest-xdist` and this:

    pytest -n auto -m ""not serial""

I'll try not using the `-n auto` bit on Windows and see if that helps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877726288,https://api.github.com/repos/simonw/datasette/issues/511,877726288,MDEyOklzc3VlQ29tbWVudDg3NzcyNjI4OA==,9599,simonw,2021-07-11T01:29:41Z,2021-07-11T01:29:41Z,OWNER,"Lots of errors that look like this:
```
2021-07-11T00:40:32.1189321Z E           NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpdr41pgwg\\data.db'
2021-07-11T00:40:32.1190083Z 
2021-07-11T00:40:32.1191128Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:596: NotADirectoryError
2021-07-11T00:40:32.1191999Z ___________________ ERROR at teardown of test_insert_error ____________________
2021-07-11T00:40:32.1192842Z [gw1] win32 -- Python 3.8.10 c:\hostedtoolcache\windows\python\3.8.10\x64\python.exe
2021-07-11T00:40:32.1193387Z 
2021-07-11T00:40:32.1193930Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_'
2021-07-11T00:40:32.1194876Z onerror = <function TemporaryDirectory._rmtree.<locals>.onerror at 0x00000291FCEA93A0>
2021-07-11T00:40:32.1195480Z 
2021-07-11T00:40:32.1195927Z     def _rmtree_unsafe(path, onerror):
2021-07-11T00:40:32.1196435Z         try:
2021-07-11T00:40:32.1196910Z             with os.scandir(path) as scandir_it:
2021-07-11T00:40:32.1197504Z                 entries = list(scandir_it)
2021-07-11T00:40:32.1198002Z         except OSError:
2021-07-11T00:40:32.1198607Z             onerror(os.scandir, path, sys.exc_info())
2021-07-11T00:40:32.1199137Z             entries = []
2021-07-11T00:40:32.1199637Z         for entry in entries:
2021-07-11T00:40:32.1200184Z             fullname = entry.path
2021-07-11T00:40:32.1200692Z             if _rmtree_isdir(entry):
2021-07-11T00:40:32.1201198Z                 try:
2021-07-11T00:40:32.1201643Z                     if entry.is_symlink():
2021-07-11T00:40:32.1202280Z                         # This can only happen if someone replaces
2021-07-11T00:40:32.1202944Z                         # a directory with a symlink after the call to
2021-07-11T00:40:32.1203623Z                         # os.scandir or entry.is_dir above.
2021-07-11T00:40:32.1204303Z                         raise OSError(""Cannot call rmtree on a symbolic link"")
2021-07-11T00:40:32.1204942Z                 except OSError:
2021-07-11T00:40:32.1206416Z                     onerror(os.path.islink, fullname, sys.exc_info())
2021-07-11T00:40:32.1207022Z                     continue
2021-07-11T00:40:32.1207584Z                 _rmtree_unsafe(fullname, onerror)
2021-07-11T00:40:32.1208074Z             else:
2021-07-11T00:40:32.1208496Z                 try:
2021-07-11T00:40:32.1208926Z >                   os.unlink(fullname)
2021-07-11T00:40:32.1210053Z E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1210974Z 
2021-07-11T00:40:32.1211638Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:616: PermissionError
2021-07-11T00:40:32.1212211Z 
2021-07-11T00:40:32.1212846Z During handling of the above exception, another exception occurred:
2021-07-11T00:40:32.1213320Z 
2021-07-11T00:40:32.1213797Z func = <built-in function unlink>
2021-07-11T00:40:32.1214529Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1215763Z exc_info = (<class 'PermissionError'>, PermissionError(13, 'The process cannot access the file because it is being used by another process'), <traceback object at 0x00000291FB4D7040>)
2021-07-11T00:40:32.1217263Z 
2021-07-11T00:40:32.1217777Z     def onerror(func, path, exc_info):
2021-07-11T00:40:32.1218421Z         if issubclass(exc_info[0], PermissionError):
2021-07-11T00:40:32.1219079Z             def resetperms(path):
2021-07-11T00:40:32.1219518Z                 try:
2021-07-11T00:40:32.1219992Z                     _os.chflags(path, 0)
2021-07-11T00:40:32.1220535Z                 except AttributeError:
2021-07-11T00:40:32.1221110Z                     pass
2021-07-11T00:40:32.1221545Z                 _os.chmod(path, 0o700)
2021-07-11T00:40:32.1221984Z     
2021-07-11T00:40:32.1222330Z             try:
2021-07-11T00:40:32.1222768Z                 if path != name:
2021-07-11T00:40:32.1223332Z                     resetperms(_os.path.dirname(path))
2021-07-11T00:40:32.1223963Z                 resetperms(path)
2021-07-11T00:40:32.1224408Z     
2021-07-11T00:40:32.1224749Z                 try:
2021-07-11T00:40:32.1225954Z >                   _os.unlink(path)
2021-07-11T00:40:32.1227032Z E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1227927Z 
2021-07-11T00:40:32.1228646Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:802: PermissionError
2021-07-11T00:40:32.1229200Z 
2021-07-11T00:40:32.1229842Z During handling of the above exception, another exception occurred:
2021-07-11T00:40:32.1230355Z 
2021-07-11T00:40:32.1230783Z     @pytest.fixture
2021-07-11T00:40:32.1231322Z     def canned_write_client():
2021-07-11T00:40:32.1231805Z         with make_app_client(
2021-07-11T00:40:32.1232467Z             extra_databases={""data.db"": ""create table names (name text)""},
2021-07-11T00:40:32.1233104Z             metadata={
2021-07-11T00:40:32.1233535Z                 ""databases"": {
2021-07-11T00:40:32.1233989Z                     ""data"": {
2021-07-11T00:40:32.1234416Z                         ""queries"": {
2021-07-11T00:40:32.1235001Z                             ""canned_read"": {""sql"": ""select * from names""},
2021-07-11T00:40:32.1235527Z                             ""add_name"": {
2021-07-11T00:40:32.1236117Z                                 ""sql"": ""insert into names (name) values (:name)"",
2021-07-11T00:40:32.1236686Z                                 ""write"": True,
2021-07-11T00:40:32.1237317Z                                 ""on_success_redirect"": ""/data/add_name?success"",
2021-07-11T00:40:32.1237882Z                             },
2021-07-11T00:40:32.1238331Z                             ""add_name_specify_id"": {
2021-07-11T00:40:32.1239009Z                                 ""sql"": ""insert into names (rowid, name) values (:rowid, :name)"",
2021-07-11T00:40:32.1239610Z                                 ""write"": True,
2021-07-11T00:40:32.1240259Z                                 ""on_error_redirect"": ""/data/add_name_specify_id?error"",
2021-07-11T00:40:32.1240839Z                             },
2021-07-11T00:40:32.1241320Z                             ""delete_name"": {
2021-07-11T00:40:32.1242504Z                                 ""sql"": ""delete from names where rowid = :rowid"",
2021-07-11T00:40:32.1243127Z                                 ""write"": True,
2021-07-11T00:40:32.1243721Z                                 ""on_success_message"": ""Name deleted"",
2021-07-11T00:40:32.1244282Z                                 ""allow"": {""id"": ""root""},
2021-07-11T00:40:32.1244749Z                             },
2021-07-11T00:40:32.1245959Z                             ""update_name"": {
2021-07-11T00:40:32.1246614Z                                 ""sql"": ""update names set name = :name where rowid = :rowid"",
2021-07-11T00:40:32.1247267Z                                 ""params"": [""rowid"", ""name"", ""extra""],
2021-07-11T00:40:32.1247828Z                                 ""write"": True,
2021-07-11T00:40:32.1248247Z                             },
2021-07-11T00:40:32.1248653Z                         }
2021-07-11T00:40:32.1249166Z                     }
2021-07-11T00:40:32.1249577Z                 }
2021-07-11T00:40:32.1249962Z             },
2021-07-11T00:40:32.1250333Z         ) as client:
2021-07-11T00:40:32.1250822Z >           yield client
2021-07-11T00:40:32.1251078Z 
2021-07-11T00:40:32.1251678Z D:\a\datasette\datasette\tests\test_canned_queries.py:43: 
2021-07-11T00:40:32.1252347Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-07-11T00:40:32.1253040Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\contextlib.py:120: in __exit__
2021-07-11T00:40:32.1253759Z     next(self.gen)
2021-07-11T00:40:32.1254398Z D:\a\datasette\datasette\tests\fixtures.py:156: in make_app_client
2021-07-11T00:40:32.1255098Z     yield TestClient(ds)
2021-07-11T00:40:32.1255796Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:827: in __exit__
2021-07-11T00:40:32.1256510Z     self.cleanup()
2021-07-11T00:40:32.1257200Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:831: in cleanup
2021-07-11T00:40:32.1257961Z     self._rmtree(self.name)
2021-07-11T00:40:32.1258712Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:813: in _rmtree
2021-07-11T00:40:32.1259487Z     _shutil.rmtree(name, onerror=onerror)
2021-07-11T00:40:32.1260280Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:740: in rmtree
2021-07-11T00:40:32.1261039Z     return _rmtree_unsafe(path, onerror)
2021-07-11T00:40:32.1261843Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:618: in _rmtree_unsafe
2021-07-11T00:40:32.1262633Z     onerror(os.unlink, fullname, sys.exc_info())
2021-07-11T00:40:32.1263456Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:805: in onerror
2021-07-11T00:40:32.1264175Z     cls._rmtree(path)
2021-07-11T00:40:32.1264848Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:813: in _rmtree
2021-07-11T00:40:32.1266329Z     _shutil.rmtree(name, onerror=onerror)
2021-07-11T00:40:32.1267082Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:740: in rmtree
2021-07-11T00:40:32.1267858Z     return _rmtree_unsafe(path, onerror)
2021-07-11T00:40:32.1268615Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:599: in _rmtree_unsafe
2021-07-11T00:40:32.1269440Z     onerror(os.scandir, path, sys.exc_info())
2021-07-11T00:40:32.1269979Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-07-11T00:40:32.1270287Z 
2021-07-11T00:40:32.1270947Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1273356Z onerror = <function TemporaryDirectory._rmtree.<locals>.onerror at 0x00000291FCF40E50>
2021-07-11T00:40:32.1273999Z 
2021-07-11T00:40:32.1274493Z     def _rmtree_unsafe(path, onerror):
2021-07-11T00:40:32.1274953Z         try:
2021-07-11T00:40:32.1275461Z >           with os.scandir(path) as scandir_it:
2021-07-11T00:40:32.1276459Z E           NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1277220Z 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877725742,https://api.github.com/repos/simonw/datasette/issues/511,877725742,MDEyOklzc3VlQ29tbWVudDg3NzcyNTc0Mg==,9599,simonw,2021-07-11T01:25:01Z,2021-07-11T01:26:38Z,OWNER,"That's weird. https://github.com/simonw/datasette/runs/3037862798 finished running and came up green - but actually a TON of the tests failed on Windows. Not sure why that didn't fail the whole test suite:

<img width=""1203"" alt=""Ran_Black__refs__1388_·_simonw_datasette_518cb06"" src=""https://user-images.githubusercontent.com/9599/125180192-12257000-e1ac-11eb-8657-d46b7bcdc1b2.png"">

Also the test suite took 50 minutes on Windows!

Here's a copy of the full log file for the tests on Python 3.8 on Windows: https://gist.github.com/simonw/2900ef33693c1bbda09188eb31c8212d","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/1388#issuecomment-877725193,https://api.github.com/repos/simonw/datasette/issues/1388,877725193,MDEyOklzc3VlQ29tbWVudDg3NzcyNTE5Mw==,9599,simonw,2021-07-11T01:18:38Z,2021-07-11T01:18:38Z,OWNER,Wrote up a TIL: https://til.simonwillison.net/nginx/proxy-domain-sockets,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877721003,https://api.github.com/repos/simonw/datasette/issues/1388,877721003,MDEyOklzc3VlQ29tbWVudDg3NzcyMTAwMw==,9599,simonw,2021-07-11T00:21:19Z,2021-07-11T00:21:19Z,OWNER,Documentation: https://docs.datasette.io/en/latest/deploying.html#nginx-proxy-configuration,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/511#issuecomment-877718364,https://api.github.com/repos/simonw/datasette/issues/511,877718364,MDEyOklzc3VlQ29tbWVudDg3NzcxODM2NA==,9599,simonw,2021-07-10T23:54:37Z,2021-07-10T23:54:37Z,OWNER,"Looks like it's not even 10% of the way through, and already a bunch of errors:

<img width=""777"" alt=""Ran_Black__refs__1388_·_simonw_datasette_518cb06"" src=""https://user-images.githubusercontent.com/9599/125179059-81e12e00-e19f-11eb-94d9-0f2d9ce8afad.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877718286,https://api.github.com/repos/simonw/datasette/issues/511,877718286,MDEyOklzc3VlQ29tbWVudDg3NzcxODI4Ng==,9599,simonw,2021-07-10T23:53:29Z,2021-07-10T23:53:29Z,OWNER,"Test suite on Windows seems to run a lot slower:

<img width=""391"" alt=""Ran_Black__refs__1388_·_simonw_datasette_518cb06"" src=""https://user-images.githubusercontent.com/9599/125179033-4e9e9f00-e19f-11eb-8c2e-8df09fafa795.png"">

From https://github.com/simonw/datasette/actions/runs/1018938850 which is still going.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877717791,https://api.github.com/repos/simonw/datasette/issues/511,877717791,MDEyOklzc3VlQ29tbWVudDg3NzcxNzc5MQ==,9599,simonw,2021-07-10T23:45:35Z,2021-07-10T23:45:35Z,OWNER,"> Trying to run on Windows today, I get an error from the utils/asgi.py module.
> 
> It's trying `from os import EX_CANTCREAT` which is Unix-only. I commented this line out, and (so far) it's working.

Good news: that line was removed in #1094.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/pull/557#issuecomment-877717392,https://api.github.com/repos/simonw/datasette/issues/557,877717392,MDEyOklzc3VlQ29tbWVudDg3NzcxNzM5Mg==,9599,simonw,2021-07-10T23:39:48Z,2021-07-10T23:39:48Z,OWNER,Abandoning this - need to switch to using GitHub Actions for this instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",466996584,Get tests running on Windows using Travis CI,
https://github.com/simonw/datasette/issues/1388#issuecomment-877717262,https://api.github.com/repos/simonw/datasette/issues/1388,877717262,MDEyOklzc3VlQ29tbWVudDg3NzcxNzI2Mg==,9599,simonw,2021-07-10T23:37:54Z,2021-07-10T23:37:54Z,OWNER,"> I wonder if `--fd` is worth supporting too?

I'm going to hold off on implementing this until someone asks for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877716993,https://api.github.com/repos/simonw/datasette/issues/1388,877716993,MDEyOklzc3VlQ29tbWVudDg3NzcxNjk5Mw==,9599,simonw,2021-07-10T23:34:02Z,2021-07-10T23:34:02Z,OWNER,"Figured out an example nginx configuration. This in `nginx.conf`:

    daemon off;
    events {
      worker_connections  1024;
    }
    http {
      server {
        listen 8092;
        location / {
          proxy_pass              http://datasette;
          proxy_set_header        X-Real-IP $remote_addr;
          proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        }
      }
      upstream datasette {
        server unix:/tmp/datasette.sock;
      }
    }

Then run `datasette --uds /tmp/datasette.sock`

Then run nginx like this:

    nginx -c ./nginx.conf

Then hits to `http://localhost:8092/` will be proxied to Datasette.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877716359,https://api.github.com/repos/simonw/datasette/issues/1388,877716359,MDEyOklzc3VlQ29tbWVudDg3NzcxNjM1OQ==,9599,simonw,2021-07-10T23:24:58Z,2021-07-10T23:24:58Z,OWNER,"Apparently Windows 10 has Unix domain socket support: https://bugs.python.org/issue33408

> Unix socket (AF_UNIX) is now avalible in Windows 10 (April 2018 Update). Please add Python support for it.
> More details about it on https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/

But it's not clear if this is going to work. That same issue thread (the issue is still open) suggests using `hasattr(socket, 'AF_UNIX'))` to detect support in tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877716156,https://api.github.com/repos/simonw/datasette/issues/1388,877716156,MDEyOklzc3VlQ29tbWVudDg3NzcxNjE1Ng==,9599,simonw,2021-07-10T23:22:21Z,2021-07-10T23:22:21Z,OWNER,"I don't have the Datasette test suite running on Windows yet, but I'd like it to run there some day - so ideally this test would be skipped if Unix domain sockets are not supported by the underlying operating system.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877715654,https://api.github.com/repos/simonw/datasette/issues/1388,877715654,MDEyOklzc3VlQ29tbWVudDg3NzcxNTY1NA==,9599,simonw,2021-07-10T23:15:06Z,2021-07-10T23:15:06Z,OWNER,"I can run tests against it using `httpx`: https://www.python-httpx.org/advanced/#usage_1

> ```pycon
> >>> import httpx
> >>> # Connect to the Docker API via a Unix Socket.
> >>> transport = httpx.HTTPTransport(uds=""/var/run/docker.sock"")
> >>> client = httpx.Client(transport=transport)
> >>> response = client.get(""http://docker/info"")
> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877714698,https://api.github.com/repos/simonw/datasette/issues/1388,877714698,MDEyOklzc3VlQ29tbWVudDg3NzcxNDY5OA==,9599,simonw,2021-07-10T23:01:37Z,2021-07-10T23:01:37Z,OWNER,"Can test this with:
```
curl --unix-socket ${socket} -i ""http://localhost/"" 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1391#issuecomment-877691558,https://api.github.com/repos/simonw/datasette/issues/1391,877691558,MDEyOklzc3VlQ29tbWVudDg3NzY5MTU1OA==,9599,simonw,2021-07-10T19:26:57Z,2021-07-10T19:26:57Z,OWNER,"The `https://latest.datasette.io/fixtures.db` file no longer includes generated columns, which will help avoid confusion such as seen in #1376.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1391#issuecomment-877691427,https://api.github.com/repos/simonw/datasette/issues/1391,877691427,MDEyOklzc3VlQ29tbWVudDg3NzY5MTQyNw==,9599,simonw,2021-07-10T19:26:00Z,2021-07-10T19:26:00Z,OWNER,I had to run the tests locally on my macOS laptop using `pysqlite3` to get a version that supported generated columns - wrote up a TIL about that here: https://til.simonwillison.net/sqlite/pysqlite3-on-macos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1391#issuecomment-877687196,https://api.github.com/repos/simonw/datasette/issues/1391,877687196,MDEyOklzc3VlQ29tbWVudDg3NzY4NzE5Ng==,9599,simonw,2021-07-10T18:58:40Z,2021-07-10T18:58:40Z,OWNER,I can use the `extra_databases` mechanism as demonstrated here: https://github.com/simonw/datasette/blob/9552414e1f968c6fc704031cec349c05e6bc2371/tests/test_canned_queries.py#L8-L12,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1391#issuecomment-877686784,https://api.github.com/repos/simonw/datasette/issues/1391,877686784,MDEyOklzc3VlQ29tbWVudDg3NzY4Njc4NA==,9599,simonw,2021-07-10T18:56:03Z,2021-07-10T18:56:03Z,OWNER,Here's the SQL used to generate the table for the test: https://github.com/simonw/datasette/blob/02b19c7a9afd328f22040ab33b5c1911cd904c7c/tests/fixtures.py#L723-L733,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1391#issuecomment-877682533,https://api.github.com/repos/simonw/datasette/issues/1391,877682533,MDEyOklzc3VlQ29tbWVudDg3NzY4MjUzMw==,9599,simonw,2021-07-10T18:28:05Z,2021-07-10T18:28:05Z,OWNER,"Here's the test in question: https://github.com/simonw/datasette/blob/a6c55afe8c82ead8deb32f90c9324022fd422324/tests/test_api.py#L2033-L2046

Various other places in the test code also need changing - anything that calls `supports_generated_columns()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1389#issuecomment-877681031,https://api.github.com/repos/simonw/datasette/issues/1389,877681031,MDEyOklzc3VlQ29tbWVudDg3NzY4MTAzMQ==,9599,simonw,2021-07-10T18:17:29Z,2021-07-10T18:17:29Z,OWNER,"I don't like `?_searchmode=default` because it suggests ""use the default"" - but it actually over-rides the default that was specified by `""searchmode"": ""raw""` in `metadata.json`.

I'm going with `?_searchmode=escaped` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1390#issuecomment-877310125,https://api.github.com/repos/simonw/datasette/issues/1390,877310125,MDEyOklzc3VlQ29tbWVudDg3NzMxMDEyNQ==,9599,simonw,2021-07-09T16:32:57Z,2021-07-09T16:32:57Z,OWNER,https://docs.datasette.io/en/latest/deploying.html#running-datasette-using-systemd,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940891698,Mention restarting systemd in documentation,
https://github.com/simonw/datasette/issues/1390#issuecomment-877308310,https://api.github.com/repos/simonw/datasette/issues/1390,877308310,MDEyOklzc3VlQ29tbWVudDg3NzMwODMxMA==,9599,simonw,2021-07-09T16:29:48Z,2021-07-09T16:29:48Z,OWNER,    sudo systemctl restart datasette.service,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940891698,Mention restarting systemd in documentation,
https://github.com/simonw/datasette/issues/1389#issuecomment-876620095,https://api.github.com/repos/simonw/datasette/issues/1389,876620095,MDEyOklzc3VlQ29tbWVudDg3NjYyMDA5NQ==,9599,simonw,2021-07-08T17:35:09Z,2021-07-08T17:35:09Z,OWNER,Also came up here: https://github.com/simonw/datasette/issues/268#issuecomment-876616414,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1389#issuecomment-876619531,https://api.github.com/repos/simonw/datasette/issues/1389,876619531,MDEyOklzc3VlQ29tbWVudDg3NjYxOTUzMQ==,9599,simonw,2021-07-08T17:34:16Z,2021-07-08T17:34:16Z,OWNER,If I implement this I'll also set it up so `?_searchmode=default` can be used to over-ride that and go back to the default behaviour.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1389#issuecomment-876619271,https://api.github.com/repos/simonw/datasette/issues/1389,876619271,MDEyOklzc3VlQ29tbWVudDg3NjYxOTI3MQ==,9599,simonw,2021-07-08T17:33:49Z,2021-07-08T17:33:49Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/table.py#L491-L498,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1389#issuecomment-876618847,https://api.github.com/repos/simonw/datasette/issues/1389,876618847,MDEyOklzc3VlQ29tbWVudDg3NjYxODg0Nw==,9599,simonw,2021-07-08T17:33:08Z,2021-07-08T17:33:08Z,OWNER,Relevant documentation: https://docs.datasette.io/en/0.57.1/full_text_search.html#advanced-sqlite-search-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1389#issuecomment-876618582,https://api.github.com/repos/simonw/datasette/issues/1389,876618582,MDEyOklzc3VlQ29tbWVudDg3NjYxODU4Mg==,9599,simonw,2021-07-08T17:32:40Z,2021-07-08T17:32:40Z,OWNER,This makes sense to me since other useful querystring arguments like this can be set as defaults in the `metadata.json` configuration.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/268#issuecomment-876616414,https://api.github.com/repos/simonw/datasette/issues/268,876616414,MDEyOklzc3VlQ29tbWVudDg3NjYxNjQxNA==,9599,simonw,2021-07-08T17:29:04Z,2021-07-08T17:29:04Z,OWNER,"> I had setup a full text search on my instance of Datasette for title data for our public library, and was noticing that some of the features of the SQLite FTS weren't working as expected ... and maybe the issue is in the `escape_fts()` function

That's a deliberate feature (albeit controversial, see #759) - part of the main problem here is that it's easy to construct a SQLite full-text search string which results in a database error. This is a bad user-experience!

You can opt-in to raw SQL queries by appending `?_searchmode=raw` to the page, see https://docs.datasette.io/en/stable/full_text_search.html#advanced-sqlite-search-queries

But maybe there should be an option for turning that on by default without needing the query string?

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/1388#issuecomment-875738149,https://api.github.com/repos/simonw/datasette/issues/1388,875738149,MDEyOklzc3VlQ29tbWVudDg3NTczODE0OQ==,9599,simonw,2021-07-07T16:14:29Z,2021-07-07T16:14:29Z,OWNER,This sounds like a valuable feature for people running Datasette behind a proxy.,"{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-875742910,https://api.github.com/repos/simonw/datasette/issues/1388,875742910,MDEyOklzc3VlQ29tbWVudDg3NTc0MjkxMA==,9599,simonw,2021-07-07T16:20:50Z,2021-07-07T16:23:02Z,OWNER,"I wonder if `--fd` is worth supporting too? Uvicorn documentation says that's useful for running under process managers, I'd want to understand exactly how to use that (and test it) before adding the feature though.

https://www.uvicorn.org/settings/

Docs on how to use a process manager: https://www.uvicorn.org/deployment/#supervisor","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-875741410,https://api.github.com/repos/simonw/datasette/issues/1388,875741410,MDEyOklzc3VlQ29tbWVudDg3NTc0MTQxMA==,9599,simonw,2021-07-07T16:18:50Z,2021-07-07T16:18:50Z,OWNER,"You could actually run Datasette like this today without modifications by running a thin Python script that imports from `datasette.app`, instantiates the ASGI app and passes that to `uvicorn.run` - but I like this as a supported feature too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-875740085,https://api.github.com/repos/simonw/datasette/issues/1388,875740085,MDEyOklzc3VlQ29tbWVudDg3NTc0MDA4NQ==,9599,simonw,2021-07-07T16:17:08Z,2021-07-07T16:17:08Z,OWNER,"Looks pretty easy to implement - here's a hint from Uvicorn source code: https://github.com/encode/uvicorn/blob/b5af1049e63c059dc750a450c807b9768f91e906/uvicorn/main.py#L368

Need to work out a simple pattern for testing this too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1387#issuecomment-873139138,https://api.github.com/repos/simonw/datasette/issues/1387,873139138,MDEyOklzc3VlQ29tbWVudDg3MzEzOTEzOA==,9599,simonw,2021-07-02T17:05:47Z,2021-07-02T17:05:47Z,OWNER,"In this case the proxy is Apache. So there are a couple of potential fixes:

- Configure Apache to pass the original HTTP request `Host:` header through to the proxied application. This should then be documented.
- Add a new optional feature to Datasette called something like `base_host` which, if set, is always used in place of the host in `request.url` when constructing new URLs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873137935,https://api.github.com/repos/simonw/datasette/issues/1387,873137935,MDEyOklzc3VlQ29tbWVudDg3MzEzNzkzNQ==,9599,simonw,2021-07-02T17:03:36Z,2021-07-02T17:03:36Z,OWNER,"And the links to apply a facet value are broken too! https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-3d4a4b7/bib?_facet=bib_level_callnumber

```json
        {
          ""value"": ""g l fiction"",
          ""label"": ""g l fiction"",
          ""count"": 212,
          ""toggle_url"": ""https://127.0.0.1:8010/collection-analysis/current_collection-3d4a4b7/bib.json?_facet=bib_level_callnumber&bib_level_callnumber=g+l+fiction"",
          ""selected"": false
        }
```
Same problem: https://github.com/simonw/datasette/blob/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f/datasette/facets.py#L251-L261","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873136440,https://api.github.com/repos/simonw/datasette/issues/1387,873136440,MDEyOklzc3VlQ29tbWVudDg3MzEzNjQ0MA==,9599,simonw,2021-07-02T17:01:48Z,2021-07-02T17:01:48Z,OWNER,"Here's what's happening: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/table.py#L827-L829

This is being run through `absolute_url()` - defined here: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/app.py#L633-L637

That's because the `next_url` in the JSON needs to be a full URL that a client can retrieve - as  opposed to the other links on that page which are all relative links that start with `/`: https://github.com/simonw/datasette/blob/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f/datasette/templates/_table.html#L11-L15

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873134866,https://api.github.com/repos/simonw/datasette/issues/1387,873134866,MDEyOklzc3VlQ29tbWVudDg3MzEzNDg2Ng==,9599,simonw,2021-07-02T16:58:52Z,2021-07-02T16:58:52Z,OWNER,What's weird here is that the URL itself is correct - it starts with `/collection-analysis/` as expected - but the hostname is wrong.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873156408,https://api.github.com/repos/simonw/datasette/issues/1387,873156408,MDEyOklzc3VlQ29tbWVudDg3MzE1NjQwOA==,9599,simonw,2021-07-02T17:37:30Z,2021-07-02T17:37:30Z,OWNER,Updated documentation is here: https://docs.datasette.io/en/latest/deploying.html#apache-proxy-configuration,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873141222,https://api.github.com/repos/simonw/datasette/issues/1387,873141222,MDEyOklzc3VlQ29tbWVudDg3MzE0MTIyMg==,9599,simonw,2021-07-02T17:09:32Z,2021-07-02T17:09:32Z,OWNER,I'm going to add this to the suggested Apache configuration at https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873140742,https://api.github.com/repos/simonw/datasette/issues/1387,873140742,MDEyOklzc3VlQ29tbWVudDg3MzE0MDc0Mg==,9599,simonw,2021-07-02T17:08:40Z,2021-07-02T17:08:40Z,OWNER,`ProxyPreserveHost On` is the Apache setting - it defaults to `Off`: https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypreservehost ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1101#issuecomment-869812567,https://api.github.com/repos/simonw/datasette/issues/1101,869812567,MDEyOklzc3VlQ29tbWVudDg2OTgxMjU2Nw==,9599,simonw,2021-06-28T16:06:57Z,2021-06-28T16:07:24Z,OWNER,"Relevant blog post: https://simonwillison.net/2021/Jun/25/streaming-large-api-responses/ - including notes on efficiently streaming formats with some kind of separator in between the records (regular JSON).

> Some export formats are friendlier for streaming than others. CSV and TSV are pretty easy to stream, as is newline-delimited JSON.
> 
> Regular JSON requires a bit more thought: you can output a `[` character, then output each row in a stream with a comma suffix, then skip the comma for the last row and output a `]`. Doing that requires peeking ahead (looping two at a time) to verify that you haven't yet reached the end.
> 
> Or... Martin De Wulf [pointed out](https://twitter.com/madewulf/status/1405559088994467844) that you can output the first row, then output every other row with a preceeding comma---which avoids the whole ""iterate two at a time"" problem entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1384#issuecomment-869075395,https://api.github.com/repos/simonw/datasette/issues/1384,869075395,MDEyOklzc3VlQ29tbWVudDg2OTA3NTM5NQ==,9599,simonw,2021-06-26T23:54:21Z,2021-06-26T23:59:21Z,OWNER,(It may well be that implementing #1168 involves a switch to async metadata),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869075368,https://api.github.com/repos/simonw/datasette/issues/1384,869075368,MDEyOklzc3VlQ29tbWVudDg2OTA3NTM2OA==,9599,simonw,2021-06-26T23:53:55Z,2021-06-26T23:53:55Z,OWNER,"Great, let's drop fallback then.

My instinct at the moment is to ship this plugin hook as-is but with a warning that it may change before Datasette 1.0 - then before 1.0 either figure out an async variant or finish the database-backed metadata concept from #1168 and recommend that as an alternative.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869071790,https://api.github.com/repos/simonw/datasette/issues/1384,869071790,MDEyOklzc3VlQ29tbWVudDg2OTA3MTc5MA==,9599,simonw,2021-06-26T23:04:12Z,2021-06-26T23:04:12Z,OWNER,"> Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.
> 
> @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?

Answering my own question: here's how Brandon implements it in his `datasette-live-config` plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160

That's using a completely separate SQLite connection (actually wrapped in `sqlite-utils`) and making blocking synchronous calls to it.

This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.

But... it's weird. Everywhere else in Datasette land uses `await db.execute(...)` - but here's an example where users are encouraged to use blocking calls instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869071435,https://api.github.com/repos/simonw/datasette/issues/1384,869071435,MDEyOklzc3VlQ29tbWVudDg2OTA3MTQzNQ==,9599,simonw,2021-06-26T22:59:26Z,2021-06-26T22:59:26Z,OWNER,"The other alternative is to finish the work to build a `_metadata` internal table, see #1168. The idea there was that if we want to support efficient pagination and search across the metadata for thousands of attached tables powering it with a plugin hook doesn't work well - we don't want to call the hook once for every one of 1,000+ tables just to implement the homepage.

So instead, all metadata for all attached databases would be loaded into an in-memory database called `_metadata`. Plugins that want to modify stored metadata could then do so by directly writing to that table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-869071236,https://api.github.com/repos/simonw/datasette/issues/860,869071236,MDEyOklzc3VlQ29tbWVudDg2OTA3MTIzNg==,9599,simonw,2021-06-26T22:56:28Z,2021-06-26T22:56:28Z,OWNER,This work is continuing in #1384.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869071167,https://api.github.com/repos/simonw/datasette/issues/1384,869071167,MDEyOklzc3VlQ29tbWVudDg2OTA3MTE2Nw==,9599,simonw,2021-06-26T22:55:36Z,2021-06-26T22:55:36Z,OWNER,"Just realized I already have an issue open for this, at #860. I'm going to close that and continue work on this in this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869070941,https://api.github.com/repos/simonw/datasette/issues/1384,869070941,MDEyOklzc3VlQ29tbWVudDg2OTA3MDk0MQ==,9599,simonw,2021-06-26T22:53:34Z,2021-06-26T22:53:34Z,OWNER,"The `await` thing is worrying me a lot - it feels like this plugin hook is massively less useful if it can't make it's own DB queries and generally do asynchronous stuff - but I'd also like not to break every existing plugin that calls `datasette.metadata(...)`.

One solution that could work: introduce a new method, maybe `await datasette.get_metadata(...)`, which uses this plugin hook - and keep the existing `datasette.metadata()` method (which doesn't call the hook) around. This would ensure existing plugins keep on working.

Then, upgrade those plugins separately - with the goal of deprecating and removing `.metadata()` entirely in Datasette 1.0 - having upgraded the plugins in the meantime.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869070348,https://api.github.com/repos/simonw/datasette/issues/1384,869070348,MDEyOklzc3VlQ29tbWVudDg2OTA3MDM0OA==,9599,simonw,2021-06-26T22:46:18Z,2021-06-26T22:46:18Z,OWNER,"Here's where the plugin hook is called, demonstrating the `fallback=` argument: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472

I'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869070076,https://api.github.com/repos/simonw/datasette/issues/1384,869070076,MDEyOklzc3VlQ29tbWVudDg2OTA3MDA3Ng==,9599,simonw,2021-06-26T22:42:21Z,2021-06-26T22:42:21Z,OWNER,"Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.

@brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869069926,https://api.github.com/repos/simonw/datasette/issues/1384,869069926,MDEyOklzc3VlQ29tbWVudDg2OTA2OTkyNg==,9599,simonw,2021-06-26T22:40:15Z,2021-06-26T22:40:53Z,OWNER,"The documentation says:

> **datasette**:     You can use this to access plugin configuration options via `datasette.plugin_config(your_plugin_name)`, or to execute SQL queries.

That's not accurate: since the plugin hook is a regular function, not an awaitable, you can't use it to run `await db.execute(...)` so you can't execute SQL queries.

I can fix this with the await-me-maybe pattern, used for other plugin hooks: https://simonwillison.net/2020/Sep/2/await-me-maybe/

BUT... that requires changing the `ds.metadata()` function to be awaitable, which will affect every existing plugn that uses that documented internal method!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869069768,https://api.github.com/repos/simonw/datasette/issues/1384,869069768,MDEyOklzc3VlQ29tbWVudDg2OTA2OTc2OA==,9599,simonw,2021-06-26T22:37:53Z,2021-06-26T22:37:53Z,OWNER,The documentation doesn't describe the ``fallback`` argument at the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869069655,https://api.github.com/repos/simonw/datasette/issues/1384,869069655,MDEyOklzc3VlQ29tbWVudDg2OTA2OTY1NQ==,9599,simonw,2021-06-26T22:36:14Z,2021-06-26T22:37:37Z,OWNER,"Documentation for the new hook is now live at https://docs.datasette.io/en/latest/plugin_hooks.html#get-metadata-datasette-key-database-table-fallback

Link to the current snapshot of that documentation: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/docs/plugin_hooks.rst#get-metadata-datasette-key-database-table-fallback","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/pull/1368#issuecomment-869068554,https://api.github.com/repos/simonw/datasette/issues/1368,869068554,MDEyOklzc3VlQ29tbWVudDg2OTA2ODU1NA==,9599,simonw,2021-06-26T22:23:57Z,2021-06-26T22:23:57Z,OWNER,The only test failure is Black. I'm going to merge this and then reformat.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881190,https://api.github.com/repos/simonw/sqlite-utils/issues/37,868881190,MDEyOklzc3VlQ29tbWVudDg2ODg4MTE5MA==,9599,simonw,2021-06-25T23:24:28Z,2021-06-25T23:24:28Z,OWNER,Maybe I could release a separate Python package `types-sqlite-utils-numpy` which adds an over-ridden type definition that includes the `numpy` types?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints,
https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881033,https://api.github.com/repos/simonw/sqlite-utils/issues/37,868881033,MDEyOklzc3VlQ29tbWVudDg2ODg4MTAzMw==,9599,simonw,2021-06-25T23:23:49Z,2021-06-25T23:23:49Z,OWNER,"Twitter conversation about how to add types to the `.create_table(columns=)` parameter: https://twitter.com/simonw/status/1408532867592818693

> Anyone know how to write a mypy type definition for this?
>
> {""id"": int, ""name"": str, ""image"": bytes, ""weight"": float}
>
> It's a dict where keys are strings and values are one of int/str/bytes/float (weird API design I know, but I designed this long before I was thinking about mypy)

Looks like this could work:

```python
    def create_table(
        self,
        name,
        columns: Dict[str, Union[Type[int], Type[bytes], Type[str], Type[float]]],
        pk=None,
        foreign_keys=None,
        column_order=None,
        not_null=None,
        defaults=None,
        hash_id=None,
        extracts=None,
    ):
```
Except... that method can optionally also accept `numpy` types if `numpy` is installed. I don't know if it's possible to dynamically change a signature based on an import, since `mypy` is a static type analyzer and doesn't ever execute the code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints,
https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868728092,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868728092,MDEyOklzc3VlQ29tbWVudDg2ODcyODA5Mg==,9599,simonw,2021-06-25T17:39:35Z,2021-06-25T17:39:35Z,OWNER,Here's more about this problem: https://github.com/numpy/numpy/issues/15947,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",929748885,Test against Python 3.10-dev,
https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868134040,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868134040,MDEyOklzc3VlQ29tbWVudDg2ODEzNDA0MA==,9599,simonw,2021-06-25T01:49:44Z,2021-06-25T01:50:33Z,OWNER,"Test failed on 3.10 with `numpy` on macOS:
```
sqlite_utils/__init__.py:1: in <module>
11
    from .db import Database
12
sqlite_utils/db.py:48: in <module>
13
    import numpy as np  # type: ignore
14
../../../hostedtoolcache/Python/3.10.0-beta.3/x64/lib/python3.10/site-packages/numpy/__init__.py:391: in <module>
15
    raise RuntimeError(msg)
16
E   RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, more information is available at https://numpy.org/doc/stable/user/building.html#accelerated-blas-lapack-libraries Otherwise report this to the vendor that provided NumPy.
17
E   RankWarning: Polyfit may be poorly conditioned
18
Error: Process completed with exit code 4.
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",929748885,Test against Python 3.10-dev,
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-868021624,https://api.github.com/repos/simonw/sqlite-utils/issues/290,868021624,MDEyOklzc3VlQ29tbWVudDg2ODAyMTYyNA==,9599,simonw,2021-06-24T23:17:38Z,2021-06-24T23:17:38Z,OWNER,The new documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#executing-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/datasette/issues/1377#issuecomment-867209791,https://api.github.com/repos/simonw/datasette/issues/1377,867209791,MDEyOklzc3VlQ29tbWVudDg2NzIwOTc5MQ==,9599,simonw,2021-06-23T22:51:32Z,2021-06-23T22:51:32Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#skip-csrf-datasette-scope,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920884085,Mechanism for plugins to exclude certain paths from CSRF checks,
https://github.com/simonw/datasette/pull/1368#issuecomment-867102944,https://api.github.com/repos/simonw/datasette/issues/1368,867102944,MDEyOklzc3VlQ29tbWVudDg2NzEwMjk0NA==,9599,simonw,2021-06-23T19:32:01Z,2021-06-23T19:32:01Z,OWNER,"Yes, let's move ahead with getting this into an alpha.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/sqlite-utils/issues/291#issuecomment-866466388,https://api.github.com/repos/simonw/sqlite-utils/issues/291,866466388,MDEyOklzc3VlQ29tbWVudDg2NjQ2NjM4OA==,9599,simonw,2021-06-23T02:10:24Z,2021-06-23T02:10:24Z,OWNER,This already helped me spot a bug in a test: https://github.com/simonw/sqlite-utils/commit/90e211e3e2f36d2ff911ecf1afe4470ff45c7c0d#diff-4e8715c7a425ee52e74b7df4d34efd32e8c92f3e60bd51bc2e1ad5943b82032eL1160-L1161,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",927766296,Adopt flake8,
https://github.com/simonw/sqlite-utils/issues/291#issuecomment-866461926,https://api.github.com/repos/simonw/sqlite-utils/issues/291,866461926,MDEyOklzc3VlQ29tbWVudDg2NjQ2MTkyNg==,9599,simonw,2021-06-23T01:59:57Z,2021-06-23T01:59:57Z,OWNER,"That shouldn't be failing: it's complaining about these: https://github.com/simonw/sqlite-utils/blob/02898bf7af4a4e484ecc8ec852d5fee98463277b/tests/test_register_function.py#L56-L67

But I added `# noqa: F811` to them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",927766296,Adopt flake8,
https://github.com/simonw/sqlite-utils/issues/289#issuecomment-866219755,https://api.github.com/repos/simonw/sqlite-utils/issues/289,866219755,MDEyOklzc3VlQ29tbWVudDg2NjIxOTc1NQ==,9599,simonw,2021-06-22T18:13:26Z,2021-06-22T18:13:26Z,OWNER,Thanks @adamchainz - `mypy` now has a foothold on this project (and runs in CI).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925677191,Mypy fixes for rows_from_file(),
https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866184260,https://api.github.com/repos/simonw/sqlite-utils/issues/267,866184260,MDEyOklzc3VlQ29tbWVudDg2NjE4NDI2MA==,9599,simonw,2021-06-22T17:26:18Z,2021-06-22T17:27:27Z,OWNER,"If an`.update()` method doesn't work because it collides with an existing dictionary method a `.pk` property could still be nice:
```python
for row in db[""sometable""].rows:
    db[""sometable""].update(row.pk, {""modified"": 1})
```
    
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk,
https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866182655,https://api.github.com/repos/simonw/sqlite-utils/issues/267,866182655,MDEyOklzc3VlQ29tbWVudDg2NjE4MjY1NQ==,9599,simonw,2021-06-22T17:24:03Z,2021-06-22T17:24:03Z,OWNER,"I'm re-opening this as a research task because it may be possible to cleanly implement this using a `dict` subclass - some notes on that here: https://treyhunner.com/2019/04/why-you-shouldnt-inherit-from-list-and-dict-in-python/

Since this would just be for adding methods (and maybe a property for returning the primary keys for a row) the usual disadvantages of subclassing `dict` described in that article shouldn't apply.

One catch: dictionaries already have a `.update()` method! So would have to pick another name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk,
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865511810,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865511810,MDEyOklzc3VlQ29tbWVudDg2NTUxMTgxMA==,9599,simonw,2021-06-22T04:07:34Z,2021-06-22T18:26:21Z,OWNER,"That documentation section is pretty weak at the moment - here's the whole thing:

> ### Executing queries
>
> The `db.execute()` and `db.executescript()` methods provide wrappers around `.execute()` and `.executescript()` on the underlying SQLite connection. These wrappers log to the tracer function if one has been registered.
> ```python
> db = Database(memory=True)
> db[""dogs""].insert({""name"": ""Cleo""})
> db.execute(""update dogs set name = 'Cleopaws'"")
> ```
> You can pass parameters as an optional second argument, using either a list or a dictionary. These will be correctly quoted and escaped.
> ```python
> # Using ? and a list:
> db.execute(""update dogs set name = ?"", [""Cleopaws""])
> # Or using :name and a dictionary:
> db.execute(""update dogs set name = :name"", {""name"": ""Cleopaws""})
> ```

- Talks about `.execute()` - I want to talk about `.query()` instead
- Doesn't clarify that `.execute()` returns a `Cursor` - and assumes you know what to do with one
- Doesn't show an example of a `select` query at all
- The ""tracer function"" bit is confusing (should at least link to docs further down)
- For `UPDATE` should show how to access the number of rows modified (probably using `.execute()` there)

It does at least cover the two types of parameters, though that could be bulked out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865510796,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865510796,MDEyOklzc3VlQ29tbWVudDg2NTUxMDc5Ng==,9599,simonw,2021-06-22T04:04:40Z,2021-06-22T04:04:48Z,OWNER,"Still needs documentation, which will involve rewriting the whole [Executing queries](https://sqlite-utils.datasette.io/en/3.11/python-api.html#executing-queries) section.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865497846,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865497846,MDEyOklzc3VlQ29tbWVudDg2NTQ5Nzg0Ng==,9599,simonw,2021-06-22T03:21:38Z,2021-06-22T03:21:38Z,OWNER,"The Python docs say: https://docs.python.org/3/library/sqlite3.html

> To retrieve data after executing a SELECT statement, you can either treat the cursor as an iterator, call the cursor’s `fetchone()` method to retrieve a single matching row, or call `fetchall()` to get a list of the matching rows.

Looking at the C source code, both `fetchmany()` and `fetchall()` work under the hood by assembling a Python list: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L907-L972 - see calls to `PyList_Append()`

So it looks like the most efficient way to iterate over a cursor may well be `for row in cursor:` - which I think calls this C function: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L813-L876","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865495370,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865495370,MDEyOklzc3VlQ29tbWVudDg2NTQ5NTM3MA==,9599,simonw,2021-06-22T03:14:30Z,2021-06-22T03:14:30Z,OWNER,"One small problem with the existing method:
https://github.com/simonw/sqlite-utils/blob/8cedc6a8b29180e68326f6b76f249d5e39e4b591/sqlite_utils/db.py#L362-L365

It returns a full list, but what if the user would rather have a generator they can iterate over without loading the results into memory in one go?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865491922,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865491922,MDEyOklzc3VlQ29tbWVudDg2NTQ5MTkyMg==,9599,simonw,2021-06-22T03:05:35Z,2021-06-22T03:05:35Z,OWNER,"Potential names:

- `db.query(sql)` - it's weird to have both this and `db.execute()` but it is at least short and memorable
- `db.sql(sql)`
- `db.execute_d(sql)` - ugly
- `db.execute_dicts(sql)` - confusing
- `db.execute_sql(sql)` - easily confused with `db.execute(sql)`

I think `db.query(sql)` may be the best option here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/datasette/pull/1368#issuecomment-865160132,https://api.github.com/repos/simonw/datasette/issues/1368,865160132,MDEyOklzc3VlQ29tbWVudDg2NTE2MDEzMg==,9599,simonw,2021-06-21T16:07:06Z,2021-06-21T16:08:48Z,OWNER,"A few tests failed - Black, the test that checks the docs mention the new hook - the most interesting failing test looks like this one:
```
            updated_metadata[""databases""][""fixtures""][""queries""][""magic_parameters""][
                ""allow""
            ] = (allow if ""query"" in permissions else deny)
>           cascade_app_client.ds._metadata = updated_metadata
E           AttributeError: can't set attribute
```
From https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_permissions.py#L439-L467

This test is directly manipulating `_metadata` purely for the purposes of simulating different permissions - I think updating it to manipulate `_local_metadata` instead would fix that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/sqlite-utils/issues/289#issuecomment-864609271,https://api.github.com/repos/simonw/sqlite-utils/issues/289,864609271,MDEyOklzc3VlQ29tbWVudDg2NDYwOTI3MQ==,9599,simonw,2021-06-20T20:42:07Z,2021-06-20T20:42:07Z,OWNER,"Wow, thank you! I didn't know about `typing.cast()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925677191,Mypy fixes for rows_from_file(),
https://github.com/simonw/sqlite-utils/issues/286#issuecomment-864594956,https://api.github.com/repos/simonw/sqlite-utils/issues/286,864594956,MDEyOklzc3VlQ29tbWVudDg2NDU5NDk1Ng==,9599,simonw,2021-06-20T18:38:05Z,2021-06-20T18:38:05Z,OWNER,3.10 is out in Homebrew now (they turn that around so fast): https://formulae.brew.sh/formula/sqlite-utils,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925487946,Add installation instructions,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864419283,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864419283,MDEyOklzc3VlQ29tbWVudDg2NDQxOTI4Mw==,9599,simonw,2021-06-19T15:15:34Z,2021-06-19T15:15:34Z,OWNER,"I think this code is at fault: https://github.com/simonw/sqlite-utils/blob/5b257949d996fe43dc5d218d4308b88796a90740/sqlite_utils/db.py#L1017-L1023

It's using `.pks` which adds `rowid` if it's missing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418795,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864418795,MDEyOklzc3VlQ29tbWVudDg2NDQxODc5NQ==,9599,simonw,2021-06-19T15:11:05Z,2021-06-19T15:11:14Z,OWNER,"Actually I'm going to go with `use_rowid` instead - because the table doesn't inherently use a rowid itself, but you should use one if you want to query it in a way that gives you back a primary key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418188,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864418188,MDEyOklzc3VlQ29tbWVudDg2NDQxODE4OA==,9599,simonw,2021-06-19T15:05:53Z,2021-06-19T15:05:53Z,OWNER,"```python
    @property
    def uses_rowid(self):
        return not any(column for column in self.columns if column.is_pk)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417808,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417808,MDEyOklzc3VlQ29tbWVudDg2NDQxNzgwOA==,9599,simonw,2021-06-19T15:03:00Z,2021-06-19T15:03:00Z,OWNER,I think I like `table.uses_rowid` best - it reads well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417765,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417765,MDEyOklzc3VlQ29tbWVudDg2NDQxNzc2NQ==,9599,simonw,2021-06-19T15:02:42Z,2021-06-19T15:02:42Z,OWNER,"Some options:

- `table.rowid_only`
- `table.rowid_as_pk`
- `table.no_pks`
- `table.no_pk`
- `table.uses_rowid`
- `table.use_rowid`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417493,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417493,MDEyOklzc3VlQ29tbWVudDg2NDQxNzQ5Mw==,9599,simonw,2021-06-19T15:00:43Z,2021-06-19T15:00:43Z,OWNER,"I have to be careful about the language I use here. Here's the official definition: https://www.sqlite.org/rowidtable.html

> A ""rowid table"" is any table in an SQLite schema that
> 
> -   is *not* a [virtual table](https://www.sqlite.org/vtab.html), and
> -   is *not* a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table.
> 
> Most tables in a typical SQLite database schema are rowid tables.
> 
> Rowid tables are distinguished by the fact that they all have a unique, non-NULL, signed 64-bit integer [rowid](https://www.sqlite.org/lang_createtable.html#rowid) that is used as the access key for the data in the underlying [B-tree](https://www.sqlite.org/fileformat2.html#btree) storage engine.

So it's not correct to call a table a ""rowid table"" only if it is missing its own primary keys.

Maybe `table.has_rowid` is the right language to use here? No, that's no good - because tables with their own primary keys usually also have a rowid.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417133,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417133,MDEyOklzc3VlQ29tbWVudDg2NDQxNzEzMw==,9599,simonw,2021-06-19T14:57:36Z,2021-06-19T14:57:36Z,OWNER,"So the logic is:
```python
[column.name for column in self.columns if column.is_pk]
```
I need to decide on a property name. Existing names are documented here: https://sqlite-utils.datasette.io/en/stable/python-api.html#introspection","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417031,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417031,MDEyOklzc3VlQ29tbWVudDg2NDQxNzAzMQ==,9599,simonw,2021-06-19T14:56:45Z,2021-06-19T14:56:45Z,OWNER,"```pycon
>>> db = sqlite_utils.Database(memory=True)
>>> db[""rowid_table""].insert({""name"": ""Cleo""})
<Table rowid_table (name)>
>>> db[""regular_table""].insert({""id"": 1, ""name"": ""Cleo""}, pk=""id"")
<Table regular_table (id, name)>
>>> db[""rowid_table""].pks
['rowid']
>>> db[""regular_table""].pks
['id']
```
But that's because the `.pks` property hides the difference: https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416911,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864416911,MDEyOklzc3VlQ29tbWVudDg2NDQxNjkxMQ==,9599,simonw,2021-06-19T14:55:45Z,2021-06-19T14:55:45Z,OWNER,"https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810

So I can indeed detect a `rowid` table by looking for no `is_pk` columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416785,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864416785,MDEyOklzc3VlQ29tbWVudDg2NDQxNjc4NQ==,9599,simonw,2021-06-19T14:54:41Z,2021-06-19T14:54:41Z,OWNER,"```pycon
>>> db = sqlite_utils.Database(memory=True)
>>> db[""rowid_table""].insert({""name"": ""Cleo""})
<Table rowid_table (name)>
>>> db[""regular_table""].insert({""id"": 1, ""name"": ""Cleo""}, pk=""id"")
<Table regular_table (id, name)>
>>> db[""rowid_table""].pks
['rowid']
>>> db[""regular_table""].pks
['id']
```
I think I need an introspection property for working out if a table is a `rowid` table or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/283#issuecomment-864416086,https://api.github.com/repos/simonw/sqlite-utils/issues/283,864416086,MDEyOklzc3VlQ29tbWVudDg2NDQxNjA4Ng==,9599,simonw,2021-06-19T14:49:06Z,2021-06-19T14:49:13Z,OWNER,"Once again, this is difficult because of the use of a generator here - `rows_from_file()` only yields rows, so there is no obvious mechanism for it to communicate back to the wrapping code that the detected format was CSV or TSV as opposed to JSON.

I'm going to change `rows_from_file()` to return a `(generator, detected_format)` tuple.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925319214,memory: Shouldn't detect types for JSON,
https://github.com/simonw/datasette/issues/1382#issuecomment-864480051,https://api.github.com/repos/simonw/datasette/issues/1382,864480051,MDEyOklzc3VlQ29tbWVudDg2NDQ4MDA1MQ==,9599,simonw,2021-06-20T00:20:06Z,2021-06-20T00:21:02Z,OWNER,"Yes you can - thanks for pointing this out, I've added a comment to the `install.sh` script in the `datasette-csvs` Glitch project:

```bash
pip3 install -U --no-cache-dir -r requirements.txt --user && \
  mkdir -p .data && \
  rm .data/data.db || true && \
  for f in *.csv
    do
      # Add --encoding=latin-1 to the following if your CSVs use a different encoding:
      sqlite-utils insert .data/data.db ${f%.*} $f --csv
    done
```

So if you edit that file in your own project and change the line to this:

      sqlite-utils insert .data/data.db ${f%.*} $f --csv --encoding=iso-8859-1

It should fix this for you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925406964,Datasette with Glitch - is it possible to use CSV with ISO-8859-1 encoding?,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864476167,https://api.github.com/repos/simonw/sqlite-utils/issues/272,864476167,MDEyOklzc3VlQ29tbWVudDg2NDQ3NjE2Nw==,9599,simonw,2021-06-19T23:36:48Z,2021-06-19T23:36:48Z,OWNER,Wrote this up on my blog here: https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/ - with a video demo here: https://www.youtube.com/watch?v=OUjd0rkc678,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864330508,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864330508,MDEyOklzc3VlQ29tbWVudDg2NDMzMDUwOA==,9599,simonw,2021-06-19T00:34:24Z,2021-06-19T00:34:24Z,OWNER,"Got this working:

    % curl 'https://api.github.com/repos/simonw/datasette/issues' | sqlite-utils memory - 'select id from stdin' ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864328927,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864328927,MDEyOklzc3VlQ29tbWVudDg2NDMyODkyNw==,9599,simonw,2021-06-19T00:25:08Z,2021-06-19T00:25:17Z,OWNER,"I tried writing this function with type hints, but eventually gave up:
```python
def rows_from_file(
    fp: BinaryIO,
    format: Optional[Format] = None,
    dialect: Optional[Type[csv.Dialect]] = None,
    encoding: Optional[str] = None,
) -> Generator[dict, None, None]:
    if format == Format.JSON:
        decoded = json.load(fp)
        if isinstance(decoded, dict):
            decoded = [decoded]
        if not isinstance(decoded, list):
            raise RowsFromFileBadJSON(""JSON must be a list or a dictionary"")
        yield from decoded
    elif format == Format.CSV:
        decoded_fp = io.TextIOWrapper(fp, encoding=encoding or ""utf-8-sig"")
        yield from csv.DictReader(decoded_fp)
    elif format == Format.TSV:
        yield from rows_from_file(
            fp, format=Format.CSV, dialect=csv.excel_tab, encoding=encoding
        )
    elif format is None:
        # Detect the format, then call this recursively
        buffered = io.BufferedReader(fp, buffer_size=4096)
        first_bytes = buffered.peek(2048).strip()
        if first_bytes[0] in (b""["", b""{""):
            # TODO: Detect newline-JSON
            yield from rows_from_file(fp, format=Format.JSON)
        else:
            dialect = csv.Sniffer().sniff(first_bytes.decode(encoding, ""ignore""))
            yield from rows_from_file(
                fp, format=Format.CSV, dialect=dialect, encoding=encoding
            )
    else:
        raise RowsFromFileError(""Bad format"")
```
mypy said:
```
sqlite_utils/utils.py:157: error: Argument 1 to ""BufferedReader"" has incompatible type ""BinaryIO""; expected ""RawIOBase""
sqlite_utils/utils.py:163: error: Argument 1 to ""decode"" of ""bytes"" has incompatible type ""Optional[str]""; expected ""str""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/281#issuecomment-864323438,https://api.github.com/repos/simonw/sqlite-utils/issues/281,864323438,MDEyOklzc3VlQ29tbWVudDg2NDMyMzQzOA==,9599,simonw,2021-06-18T23:55:06Z,2021-06-18T23:55:06Z,OWNER,"The `-:json` idea is flawed: Click thinks that's the syntax for an option called `:json`.

I'm going to do `stdin:json` - which means you can't open a file called `stdin` - but you could use `cat stdin | sqlite-utils memory stdin:json ...`  instead which is an OK workaround.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924992318,Mechanism for explicitly stating CSV or JSON or TSV for sqlite-utils memory,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358951,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864358951,MDEyOklzc3VlQ29tbWVudDg2NDM1ODk1MQ==,9599,simonw,2021-06-19T05:30:00Z,2021-06-19T05:30:00Z,OWNER,If this can be fixed it will be in the `transform_sql()` method.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358680,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864358680,MDEyOklzc3VlQ29tbWVudDg2NDM1ODY4MA==,9599,simonw,2021-06-19T05:27:13Z,2021-06-19T05:27:13Z,OWNER,How easy is it to detect a `rowid` table? Is it as simple as `.pks` returning `None`? If so the documentation should mention that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864354627,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864354627,MDEyOklzc3VlQ29tbWVudDg2NDM1NDYyNw==,9599,simonw,2021-06-19T04:42:03Z,2021-06-19T04:42:03Z,OWNER,"Demo:

    curl -s 'https://api.github.com/users/simonw/repos?per_page=100' | \
      sqlite-utils memory - 'select sum(size), sum(stargazers_count) from stdin limit 1'
    [{""sum(size)"": 2042547, ""sum(stargazers_count)"": 6769}]
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data,
https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864350407,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864350407,MDEyOklzc3VlQ29tbWVudDg2NDM1MDQwNw==,9599,simonw,2021-06-19T03:52:20Z,2021-06-19T03:52:20Z,OWNER,I'll have an environment variable for `--detect-types` so users who really want that as the default option can turn it on.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data,
https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864349123,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864349123,MDEyOklzc3VlQ29tbWVudDg2NDM0OTEyMw==,9599,simonw,2021-06-19T03:36:54Z,2021-06-19T03:36:54Z,OWNER,"I may change the default for `sqlite-utils insert` to detect types if I release `sqlite-utils` 4.0, as a backwards-incompatible change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data,
https://github.com/simonw/sqlite-utils/issues/179#issuecomment-864349066,https://api.github.com/repos/simonw/sqlite-utils/issues/179,864349066,MDEyOklzc3VlQ29tbWVudDg2NDM0OTA2Ng==,9599,simonw,2021-06-19T03:36:04Z,2021-06-19T03:36:04Z,OWNER,This work is going to happen in #282.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types,
https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864348954,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864348954,MDEyOklzc3VlQ29tbWVudDg2NDM0ODk1NA==,9599,simonw,2021-06-19T03:34:42Z,2021-06-19T03:35:46Z,OWNER,"I built some prototype code here for something which looks at every row in a CSV import and records the likely types: https://gist.github.com/simonw/465f9356f175d1cf86957947dff501d4

This could be used by the command-line tools to figure out what `table.transform(types=...)` method to use at the end.

This is a different approach to the pure SQL version I tried building in https://github.com/simonw/sqlite-utils/issues/179 - I think this is a better approach though, it's less prone to weird idiosyncrasies of SQLite types, and it's also easy for us to add on to the existing CSV import code in a way that won't require scanning the data twice.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864208476,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864208476,MDEyOklzc3VlQ29tbWVudDg2NDIwODQ3Ng==,9599,simonw,2021-06-18T18:30:08Z,2021-06-18T23:30:19Z,OWNER,"So maybe this is a function which can either be told the format or, if none is provided, it detects one for itself.
```python
def rows_from_file(fp, format=None):
    # ...
    yield from rows
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864207841,MDEyOklzc3VlQ29tbWVudDg2NDIwNzg0MQ==,9599,simonw,2021-06-18T18:28:40Z,2021-06-18T18:28:46Z,OWNER,"```python
def detect_format(fp):
    # ...
    return ""csv"", fp, dialect
    # or
    return ""json"", fp, parsed_data
    # or
    return ""json-nl"", fp, docs
```
The mixed return types here are ugly. In all of these cases what we really want is to return a generator of `{...}` objects. So maybe it returns that instead.
```python
def filepointer_to_documents(fp):
    # ...
    yield from documents
```
I can refactor `sqlite-utils insert` to use this new code too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864206308,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864206308,MDEyOklzc3VlQ29tbWVudDg2NDIwNjMwOA==,9599,simonw,2021-06-18T18:25:04Z,2021-06-18T18:25:04Z,OWNER,"Or... since I'm not using a streaming JSON parser at the moment, if I think something is JSON I can load the entire thing into memory to validate it.

I still need to detect newline-delimited JSON. For that I can consume the first line of the input to see if it's a valid JSON object, then maybe sniff the second line too?

This does mean that if the input is a single line of GIANT JSON it will all be consumed into memory at once, but that's going to happen anyway.

So I need a function which, given a file pointer, consumes from it, detects the type, then returns that type AND a file pointer to the beginning of the file again. I can use `io.BufferedReader` for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864129273,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864129273,MDEyOklzc3VlQ29tbWVudDg2NDEyOTI3Mw==,9599,simonw,2021-06-18T15:47:47Z,2021-06-18T15:47:47Z,OWNER,"Detecting valid JSON is tricky - just because a stream starts with `[` or `{` doesn't mean the entire stream is valid JSON. You need to parse the entire stream to determine that for sure.

One way to solve this would be with a custom state machine. Another would be to use the `ijson` streaming parser - annoyingly it throws the same exception class for invalid JSON for different reasons, but the `e.args[0]` for that exception includes human-readable text about the error - if it's anything other than `parse error: premature EOF` then it probably means the JSON was invalid.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864128489,https://api.github.com/repos/simonw/sqlite-utils/issues/278,864128489,MDEyOklzc3VlQ29tbWVudDg2NDEyODQ4OQ==,9599,simonw,2021-06-18T15:46:24Z,2021-06-18T15:46:24Z,OWNER,A workaround could be to define a bash or zsh alias of some sort.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923697888,"Support db as first parameter before subcommand, or as environment variable",
https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864126781,https://api.github.com/repos/simonw/sqlite-utils/issues/278,864126781,MDEyOklzc3VlQ29tbWVudDg2NDEyNjc4MQ==,9599,simonw,2021-06-18T15:43:19Z,2021-06-18T15:43:19Z,OWNER,"I don't think it's possible to do this without breaking backwards compatibility, unfortunately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923697888,"Support db as first parameter before subcommand, or as environment variable",
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864103005,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864103005,MDEyOklzc3VlQ29tbWVudDg2NDEwMzAwNQ==,9599,simonw,2021-06-18T15:04:15Z,2021-06-18T15:04:15Z,OWNER,"To detect JSON, check to see if the stream starts with `[` or `{` - maybe do something more sophisticated than that.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864101267,https://api.github.com/repos/simonw/sqlite-utils/issues/272,864101267,MDEyOklzc3VlQ29tbWVudDg2NDEwMTI2Nw==,9599,simonw,2021-06-18T15:01:41Z,2021-06-18T15:01:41Z,OWNER,I'll split the remaining work out into separate issues.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-864099764,https://api.github.com/repos/simonw/sqlite-utils/issues/273,864099764,MDEyOklzc3VlQ29tbWVudDg2NDA5OTc2NA==,9599,simonw,2021-06-18T14:59:27Z,2021-06-18T14:59:27Z,OWNER,I'm going to merge this as-is and work on the JSON/TSV support in a separate issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/277#issuecomment-864092515,https://api.github.com/repos/simonw/sqlite-utils/issues/277,864092515,MDEyOklzc3VlQ29tbWVudDg2NDA5MjUxNQ==,9599,simonw,2021-06-18T14:47:57Z,2021-06-18T14:47:57Z,OWNER,This is a neat improvement.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",923612361,add -h support closes #276,
https://github.com/simonw/sqlite-utils/issues/275#issuecomment-862617165,https://api.github.com/repos/simonw/sqlite-utils/issues/275,862617165,MDEyOklzc3VlQ29tbWVudDg2MjYxNzE2NQ==,9599,simonw,2021-06-16T18:34:51Z,2021-06-16T18:34:51Z,OWNER,"Also use this: https://github.com/simonw/datasette/blob/83e9c8bc7585dcc62f200e37c2daefcd669ee05e/codecov.yml

And add a badge, as seen on https://github.com/simonw/asgi-csrf","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922955697,Enable code coverage,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862605436,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862605436,MDEyOklzc3VlQ29tbWVudDg2MjYwNTQzNg==,9599,simonw,2021-06-16T18:19:05Z,2021-06-16T18:19:05Z,OWNER,`--attach` documentation: https://github.com/simonw/sqlite-utils/blob/192dc2c5b73bd836ab8e2e5fed4b36c6ea02f250/docs/cli.rst#joining-in-memory-data-against-existing-databases-using-attach,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-862495803,https://api.github.com/repos/simonw/sqlite-utils/issues/131,862495803,MDEyOklzc3VlQ29tbWVudDg2MjQ5NTgwMw==,9599,simonw,2021-06-16T15:52:33Z,2021-06-16T15:52:33Z,OWNER,I like `-t` or `--type` for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042,sqlite-utils insert: options for column types,
https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862494864,https://api.github.com/repos/simonw/sqlite-utils/issues/267,862494864,MDEyOklzc3VlQ29tbWVudDg2MjQ5NDg2NA==,9599,simonw,2021-06-16T15:51:28Z,2021-06-16T16:26:15Z,OWNER,"I did add a slightly clumsy mechanism recently to help a bit here though: the `pks_and_rows_where()` method: https://sqlite-utils.datasette.io/en/stable/python-api.html#listing-rows-with-their-primary-keys

More details in the issue for that feature: #240

The idea here is that if you want to call update you need the primary key for the row - so you can do this:

```python
for pk, row in db[""sometable""].pks_and_rows_where():
    db[""sometable""].update(pk, {""modified"": 1}"")
```
The `pk` may end up as a single value or a tuple depending on if the table has a compound primary key - but you don't need to worry about that if you use this method as it will return the correct primary key value for you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk,
https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862493179,https://api.github.com/repos/simonw/sqlite-utils/issues/267,862493179,MDEyOklzc3VlQ29tbWVudDg2MjQ5MzE3OQ==,9599,simonw,2021-06-16T15:49:13Z,2021-06-16T15:49:13Z,OWNER,"The big challenge here is that the rows returned  by this library aren't objects, they are Python dictionaries - so adding methods to them isn't possible without changing the type that is returned by these methods.

Part of the philosophy of the library is that it should make it as easy as possible to round-trip between Python dictionaries and SQLite table data, so I don't think adding methods like this is going to fit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk,
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862491721,https://api.github.com/repos/simonw/sqlite-utils/issues/270,862491721,MDEyOklzc3VlQ29tbWVudDg2MjQ5MTcyMQ==,9599,simonw,2021-06-16T15:47:06Z,2021-06-16T15:47:06Z,OWNER,"SQLite doesn't have a JSON column type - it has JSON processing functions, but they operate against TEXT columns - so there's nothing I can do here unfortunately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862491016,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862491016,MDEyOklzc3VlQ29tbWVudDg2MjQ5MTAxNg==,9599,simonw,2021-06-16T15:46:13Z,2021-06-16T15:46:13Z,OWNER,"Columns from data imported from CSV in this way is currently treated as `TEXT`, which means numeric sorts and suchlike won't work as people might expect. It would be good to do automatic type detection here, see #179.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862485408,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862485408,MDEyOklzc3VlQ29tbWVudDg2MjQ4NTQwOA==,9599,simonw,2021-06-16T15:38:58Z,2021-06-16T15:39:28Z,OWNER,"Also `sqlite-utils memory` reflects the existing `sqlite-utils :memory:` mechanism, which is a point in its favour.

And it helps emphasize that the file you are querying will be loaded into memory, so probably don't try this against a 1GB CSV file.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862484557,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862484557,MDEyOklzc3VlQ29tbWVudDg2MjQ4NDU1Nw==,9599,simonw,2021-06-16T15:37:51Z,2021-06-16T15:38:34Z,OWNER,"I wonder if there's a better name for this than `sqlite-utils memory`?

- `sqlite-utils memory hello.csv ""select * from hello""`
- `sqlite-utils mem hello.csv ""select * from hello""`
- `sqlite-utils temp hello.csv ""select * from hello""`
- `sqlite-utils adhoc hello.csv ""select * from hello""`
- `sqlite-utils scratch hello.csv ""select * from hello""`

I think `memory` is best. I don't like the others, except for `scratch` which is OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862479704,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862479704,MDEyOklzc3VlQ29tbWVudDg2MjQ3OTcwNA==,9599,simonw,2021-06-16T15:31:31Z,2021-06-16T15:31:31Z,OWNER,"Plus, could I make this change to `sqlite-utils query` without breaking backwards compatibility? Adding a new `sqlite-utils memory` command is completely safe from that perspective.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862478881,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862478881,MDEyOklzc3VlQ29tbWVudDg2MjQ3ODg4MQ==,9599,simonw,2021-06-16T15:30:24Z,2021-06-16T15:30:24Z,OWNER,"But... `sqlite-utils my.csv ""select * from my""` is a much more compelling initial experience than `sqlite-utils memory my.csv ""select * from my""`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862475685,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862475685,MDEyOklzc3VlQ29tbWVudDg2MjQ3NTY4NQ==,9599,simonw,2021-06-16T15:26:19Z,2021-06-16T15:29:38Z,OWNER,"Here's a radical idea: what if I combined `sqlite-utils memory` into `sqlite-utils query`?

The trick here would be to detect if the arguments passed on the command-line refer to SQLite databases or if they refer to CSV/JSON data that should be imported into temporary tables.

Detecting a SQLite database file is actually really easy - they all start with the same binary string:
```pycon
>>> open(""my.db"", ""rb"").read(100)
b'SQLite format 3\x00...
```
(Need to carefully check that a CSV file with`SQLite format 3` as the first column name doesn't accidentally get interpreted as a SQLite DB though).

So then what would the semantics of `sqlite-utils query` (which is also the default command) be?

- `sqlite-utils mydb.db ""select * from x""`
- `sqlite-utils my.csv ""select * from my""`
- `sqlite-utils mydb.db my.csv ""select * from mydb.x join my on ...""` - this is where it gets weird. We can't import the CSV data directly into `mpdb.db` - it's suppose to go into the in-memory database - so now we need to start using database aliases like `mydb.x` because we passed at least one other file?

The complexity here is definitely in the handling of a combination of SQLite database files and CSV filenames. Also, `sqlite-utils query` doesn't accept multiple filenames at the moment, so that will change.

I'm not 100% sold on this as being better than having a separate `sqlite-utils memory` command, as seen in #273.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862018937,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862018937,MDEyOklzc3VlQ29tbWVudDg2MjAxODkzNw==,9599,simonw,2021-06-16T03:59:28Z,2021-06-16T04:00:05Z,OWNER,"Mainly for debugging purposes it would be useful to be able to save the created in-memory database back to a file again later. This could be done with:

    sqlite-utils memory blah.csv --save saved.db

Can use `.iterdump()` to implement this: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump

Maybe instead (or as-well-as) offer `--dump` which dumps out the SQL from that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862046009,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862046009,MDEyOklzc3VlQ29tbWVudDg2MjA0NjAwOQ==,9599,simonw,2021-06-16T05:15:38Z,2021-06-16T05:15:38Z,OWNER,"I'm going to add a `--encoding` option - it will affect ALL CSV input files, so if you have CSV files with different encodings you'll need to sort that mess out yourself (likely by importing each CSV file separately into a database using `sqlite-utils insert` with different `--encoding` values).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045639,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862045639,MDEyOklzc3VlQ29tbWVudDg2MjA0NTYzOQ==,9599,simonw,2021-06-16T05:14:38Z,2021-06-16T05:14:38Z,OWNER,"Can't share much code though since a bunch of that `insert` stuff is specific to that command - showing progress bars, returning errors on illegal option combinations etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045438,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862045438,MDEyOklzc3VlQ29tbWVudDg2MjA0NTQzOA==,9599,simonw,2021-06-16T05:14:00Z,2021-06-16T05:14:00Z,OWNER,I should probably refactor the CSV/JSON/loading stuff into a function in `utils.py` in order to share some of the implementation with the existing `sqlite-utils insert` code: https://github.com/simonw/sqlite-utils/blob/287cdcae8908916687f2ecccc87c38549d004ac6/sqlite_utils/cli.py#L691-L734,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862043974,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862043974,MDEyOklzc3VlQ29tbWVudDg2MjA0Mzk3NA==,9599,simonw,2021-06-16T05:10:12Z,2021-06-16T05:10:12Z,OWNER,"I can stop promoting `:memory:` here and promote `memory` instead:

https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#L83-L86","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862042110,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862042110,MDEyOklzc3VlQ29tbWVudDg2MjA0MjExMA==,9599,simonw,2021-06-16T05:05:51Z,2021-06-16T05:06:11Z,OWNER,"Initial documentation is here: https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#running-queries-directly-against-csv-data

It only talks about CSV at the moment - needs to be updated to mention JSON too once that is implemented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040906,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862040906,MDEyOklzc3VlQ29tbWVudDg2MjA0MDkwNg==,9599,simonw,2021-06-16T05:02:47Z,2021-06-16T05:02:47Z,OWNER,"Got a prototype working!
```
 % curl -s 'https://fivethirtyeight.datasettes.com/polls/president_approval_polls.csv?_size=max&_stream=1' | sqlite-utils memory - 'select * from t limit 5' --nl 
{""rowid"": ""1"", ""question_id"": ""139304"", ""poll_id"": ""74225"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""568"", ""pollster"": ""YouGov"", ""sponsor_ids"": ""352"", ""sponsors"": ""Economist"", ""display_name"": ""YouGov"", ""pollster_rating_id"": ""391"", ""pollster_rating_name"": ""YouGov"", ""fte_grade"": ""B"", ""sample_size"": ""1500"", ""population"": ""a"", ""population_full"": ""a"", ""methodology"": ""Online"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://docs.cdn.yougov.com/y9zsit5bzd/weeklytrackingreport.pdf"", ""source"": ""538"", ""yes"": ""42.0"", ""no"": ""53.0""}
{""rowid"": ""2"", ""question_id"": ""139305"", ""poll_id"": ""74225"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""568"", ""pollster"": ""YouGov"", ""sponsor_ids"": ""352"", ""sponsors"": ""Economist"", ""display_name"": ""YouGov"", ""pollster_rating_id"": ""391"", ""pollster_rating_name"": ""YouGov"", ""fte_grade"": ""B"", ""sample_size"": ""1155"", ""population"": ""rv"", ""population_full"": ""rv"", ""methodology"": ""Online"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://docs.cdn.yougov.com/y9zsit5bzd/weeklytrackingreport.pdf"", ""source"": ""538"", ""yes"": ""44.0"", ""no"": ""55.0""}
{""rowid"": ""3"", ""question_id"": ""139306"", ""poll_id"": ""74226"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""23"", ""pollster"": ""American Research Group"", ""sponsor_ids"": """", ""sponsors"": """", ""display_name"": ""American Research Group"", ""pollster_rating_id"": ""9"", ""pollster_rating_name"": ""American Research Group"", ""fte_grade"": ""B"", ""sample_size"": ""1100"", ""population"": ""a"", ""population_full"": ""a"", ""methodology"": ""Live Phone"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://americanresearchgroup.com/economy/"", ""source"": ""538"", ""yes"": ""30.0"", ""no"": ""66.0""}
{""rowid"": ""4"", ""question_id"": ""139307"", ""poll_id"": ""74226"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""23"", ""pollster"": ""American Research Group"", ""sponsor_ids"": """", ""sponsors"": """", ""display_name"": ""American Research Group"", ""pollster_rating_id"": ""9"", ""pollster_rating_name"": ""American Research Group"", ""fte_grade"": ""B"", ""sample_size"": ""990"", ""population"": ""rv"", ""population_full"": ""rv"", ""methodology"": ""Live Phone"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://americanresearchgroup.com/economy/"", ""source"": ""538"", ""yes"": ""29.0"", ""no"": ""67.0""}
{""rowid"": ""5"", ""question_id"": ""139298"", ""poll_id"": ""74224"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""1528"", ""pollster"": ""AtlasIntel"", ""sponsor_ids"": """", ""sponsors"": """", ""display_name"": ""AtlasIntel"", ""pollster_rating_id"": ""546"", ""pollster_rating_name"": ""AtlasIntel"", ""fte_grade"": ""B/C"", ""sample_size"": ""5188"", ""population"": ""a"", ""population_full"": ""a"", ""methodology"": ""Online"", ""start_date"": ""1/15/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/19/21 21:52"", ""notes"": """", ""url"": ""https://projects.fivethirtyeight.com/polls/20210119_US_Atlas2.pdf"", ""source"": ""538"", ""yes"": ""44.6"", ""no"": ""53.9""}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040971,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862040971,MDEyOklzc3VlQ29tbWVudDg2MjA0MDk3MQ==,9599,simonw,2021-06-16T05:02:56Z,2021-06-16T05:02:56Z,OWNER,Moving this to a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861989987,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861989987,MDEyOklzc3VlQ29tbWVudDg2MTk4OTk4Nw==,9599,simonw,2021-06-16T02:34:21Z,2021-06-16T02:34:21Z,OWNER,"The documentation already covers this
```
$ sqlite-utils :memory: ""select sqlite_version()""
[{""sqlite_version()"": ""3.29.0""}]
```
https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-and-returning-json

`sqlite-utils memory ""select sqlite_version()""` is a little bit more intuitive than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861987651,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861987651,MDEyOklzc3VlQ29tbWVudDg2MTk4NzY1MQ==,9599,simonw,2021-06-16T02:27:20Z,2021-06-16T02:27:20Z,OWNER,Solution: `sqlite-utils memory -` attempts to detect the input based on if it starts with a `{` or `[` (likely JSON) or if it doesn't use the `csv.Sniffer()` mechanism. Or you can use `sqlite-utils memory -:csv` to specifically indicate the type of input.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861985944,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861985944,MDEyOklzc3VlQ29tbWVudDg2MTk4NTk0NA==,9599,simonw,2021-06-16T02:22:52Z,2021-06-16T02:22:52Z,OWNER,"Another option: allow an optional `:suffix` specifying the type of the file. If this is missing we detect based on the filename.

    sqlite-utils memory somefile:csv ""select * from somefile""

One catch: how to treat `-` for standard input?

    cat blah.csv | sqlite-utils memory - ""select * from stdin""

That's fine for CSV, but what about TSV or JSON or nl-JSON? Maybe this:

    cat blah.csv | sqlite-utils memory -:json ""select * from stdin""

Bit weird though. The alternative would be to support this:

    cat blah.csv | sqlite-utils memory --load-csv -

But that's verbose compared to the version without the long `--load-x` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861984707,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861984707,MDEyOklzc3VlQ29tbWVudDg2MTk4NDcwNw==,9599,simonw,2021-06-16T02:19:48Z,2021-06-16T02:19:48Z,OWNER,"This is going to need to be a separate command, for relatively non-obvious reasons.

    sqlite-utils blah.db ""select * from x""

Is equivalent to this, because `query` is the default sub-command:

    sqlite-utils query blah.db ""select * from x""

But... this means that making the filename optional doesn't actually work - because then this is ambiguous:

    sqlite-utils --load-csv blah.csv ""select * from blah""

So instead, I'm going to add a new sub-command. I'm currently thinking `memory` to reflect that this command operates on an in-memory database:

    sqlite-utils memory --load-csv blah.csv ""select * from blah""

I still think I need to use `--load-csv` rather than `--csv` because one interesting use-case for this is loading in CSV and converting it to JSON, or vice-versa.

Another option: allow multiple arguments which are filenames, and use the extension (or sniff the content) to decide what to do with them:

    sqlite-utils memory blah.csv foo.csv ""select * from foo join blah on ...""

This would require the last positional argument to always be a SQL query, and would treat all other positional arguments as files that should be imported into memory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891835,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861891835,MDEyOklzc3VlQ29tbWVudDg2MTg5MTgzNQ==,9599,simonw,2021-06-15T23:09:31Z,2021-06-15T23:09:31Z,OWNER,`--load-csv` and `--load-json` and `--load-nl` and `--load-tsv` are unambiguous.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891693,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861891693,MDEyOklzc3VlQ29tbWVudDg2MTg5MTY5Mw==,9599,simonw,2021-06-15T23:09:08Z,2021-06-15T23:09:08Z,OWNER,Problem: `--csv` and `--json` and `--nl` are already options for `sqlite-utils query` - need new non-conflicting names.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891272,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861891272,MDEyOklzc3VlQ29tbWVudDg2MTg5MTI3Mg==,9599,simonw,2021-06-15T23:08:02Z,2021-06-15T23:08:02Z,OWNER,"`--csv -` should work though, for reading from stdin. The table can be called `stdin`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891110,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861891110,MDEyOklzc3VlQ29tbWVudDg2MTg5MTExMA==,9599,simonw,2021-06-15T23:07:38Z,2021-06-15T23:07:38Z,OWNER,`--csvt` seems unnecessary to me: if people want to load different CSV files with the same filename (but in different directories) they will get an error unless they rename the files first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861890689,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861890689,MDEyOklzc3VlQ29tbWVudDg2MTg5MDY4OQ==,9599,simonw,2021-06-15T23:06:37Z,2021-06-15T23:06:37Z,OWNER,"How about `--json` and `--nl` and `--tsv` too? Imitating the format options for `sqlite-utils insert`.

And what happens if you provide a filename too? I'm tempted to say that the `--csv` stuff still gets loaded into an in-memory database but it's given a name and can then be joined against using SQLite `memory.blah` syntax.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861889437,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861889437,MDEyOklzc3VlQ29tbWVudDg2MTg4OTQzNw==,9599,simonw,2021-06-15T23:03:26Z,2021-06-15T23:03:26Z,OWNER,Maybe also support `--csvt` as an alternative option which takes two arguments: the CSV path and the name of the table that should be created from it (rather than auto-detecting from the filename).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/269#issuecomment-861103967,https://api.github.com/repos/simonw/sqlite-utils/issues/269,861103967,MDEyOklzc3VlQ29tbWVudDg2MTEwMzk2Nw==,9599,simonw,2021-06-15T01:34:10Z,2021-06-15T01:34:10Z,OWNER,"SQLite doesn't have the concept of a boolean column, so there's not much I can do here: https://www.sqlite.org/datatype3.html#boolean_datatype","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919250621,bool type not supported,
https://github.com/simonw/sqlite-utils/issues/266#issuecomment-861103684,https://api.github.com/repos/simonw/sqlite-utils/issues/266,861103684,MDEyOklzc3VlQ29tbWVudDg2MTEwMzY4NA==,9599,simonw,2021-06-15T01:33:13Z,2021-06-15T01:33:13Z,OWNER,Dupe of #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913135723,"Add some types, enforce with mypy",
https://github.com/simonw/datasette/issues/1377#issuecomment-861089794,https://api.github.com/repos/simonw/datasette/issues/1377,861089794,MDEyOklzc3VlQ29tbWVudDg2MTA4OTc5NA==,9599,simonw,2021-06-15T00:53:29Z,2021-06-15T00:53:29Z,OWNER,"Potential hook names:

- `skip_csrf(scope, datasette)`
- ... I can't think of any other ones I would tolerate to be honest","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920884085,Mechanism for plugins to exclude certain paths from CSRF checks,
https://github.com/simonw/datasette/issues/1377#issuecomment-861087949,https://api.github.com/repos/simonw/datasette/issues/1377,861087949,MDEyOklzc3VlQ29tbWVudDg2MTA4Nzk0OQ==,9599,simonw,2021-06-15T00:49:19Z,2021-06-15T00:49:19Z,OWNER,"The new `skip_if_scope` mechanism in `asgi-csrf` https://github.com/simonw/asgi-csrf/issues/20 is designed to help here.

Now I need to design a plugin hook that allows plugins to have an opinion on whether a specific `scope` should have CSRF skipped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920884085,Mechanism for plugins to exclude certain paths from CSRF checks,
https://github.com/simonw/datasette/issues/1376#issuecomment-860230663,https://api.github.com/repos/simonw/datasette/issues/1376,860230663,MDEyOklzc3VlQ29tbWVudDg2MDIzMDY2Mw==,9599,simonw,2021-06-13T15:39:37Z,2021-06-13T15:39:37Z,OWNER,Actually it looks like there is a PR open already that addresses this: #1296 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919822817,Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns),
https://github.com/simonw/datasette/issues/1375#issuecomment-860230385,https://api.github.com/repos/simonw/datasette/issues/1375,860230385,MDEyOklzc3VlQ29tbWVudDg2MDIzMDM4NQ==,9599,simonw,2021-06-13T15:37:49Z,2021-06-13T15:37:49Z,OWNER,"There is a feature for this at the moment, but it's a little bit hidden: you can use `?_json=col` to tell
Datasette that you would like a specific column to be exported as nested JSON: https://docs.datasette.io/en/stable/json_api.html#special-json-arguments

I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning  string instead of the expected dictionary and breaking your code.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919508498,JSON export dumps JSON fields as TEXT,
https://github.com/simonw/datasette/issues/1376#issuecomment-860229397,https://api.github.com/repos/simonw/datasette/issues/1376,860229397,MDEyOklzc3VlQ29tbWVudDg2MDIyOTM5Nw==,9599,simonw,2021-06-13T15:31:02Z,2021-06-13T15:31:02Z,OWNER,Alternative fix would be to update that section of the documentation - if the container upgrade proves tricky I can fall back on that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919822817,Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns),
https://github.com/simonw/datasette/issues/1376#issuecomment-860229226,https://api.github.com/repos/simonw/datasette/issues/1376,860229226,MDEyOklzc3VlQ29tbWVudDg2MDIyOTIyNg==,9599,simonw,2021-06-13T15:29:45Z,2021-06-13T15:29:45Z,OWNER,"Oh good catch - this is a SQLite version issue.

The `fixtures.db` file used on https://latest.datasette.io/ includes a generated column (for testing purposes) which is a feature added in SQLite 3.31.0 on 2020-01-22.

https://latest.datasette.io/-/versions

But... it looks like the packaged Datasette Docker container doesn't have that SQLite version!

I should fix that. I'm renaming this issue.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919822817,Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns),
https://github.com/simonw/sqlite-utils/issues/271#issuecomment-860142489,https://api.github.com/repos/simonw/sqlite-utils/issues/271,860142489,MDEyOklzc3VlQ29tbWVudDg2MDE0MjQ4OQ==,9599,simonw,2021-06-13T02:53:06Z,2021-06-13T02:53:06Z,OWNER,"Looks like this is the problem: https://github.com/simonw/sqlite-utils/blob/b0f9d1e494c9891ce407e27b0f5c6deeea361d30/sqlite_utils/db.py#L1724-L1742

Note how `set_cols = [col for col in all_columns if col not in pks] ` can potentially return an empty list if ALL of the columns are primary keys - but the next line of code that assigns `sql2` continues regardless, when it should instead be skipped if there are no columns in `set_cols`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919702451,table.upsert_all() fails if input has a single column that should be a primary key,
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-859986489,https://api.github.com/repos/simonw/sqlite-utils/issues/270,859986489,MDEyOklzc3VlQ29tbWVudDg1OTk4NjQ4OQ==,9599,simonw,2021-06-12T02:47:12Z,2021-06-12T02:47:12Z,OWNER,"Can you expand on what you'd like to change here? The library and CLI tool already allow JSON data to be stored in columns:

- https://sqlite-utils.datasette.io/en/stable/cli.html#nested-json-values
- https://sqlite-utils.datasette.io/en/stable/python-api.html#storing-json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON,
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859898736,https://api.github.com/repos/simonw/sqlite-utils/issues/268,859898736,MDEyOklzc3VlQ29tbWVudDg1OTg5ODczNg==,9599,simonw,2021-06-11T20:37:44Z,2021-06-11T20:37:44Z,OWNER,"From the prototype:
```
% sqlite-utils schema 24ways.db
CREATE TABLE [articles] (
               [title] TEXT  ,
   [contents] TEXT  ,
   [year] TEXT  ,
   [author] TEXT  ,
   [author_slug] TEXT  ,
   [published] TEXT  ,
   [url] TEXT  ,
   [topic] TEXT  
        );
CREATE VIRTUAL TABLE ""articles_fts"" USING FTS5 (
                title, author, contents,
                content=""articles""
            );
CREATE TABLE 'articles_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'articles_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE 'articles_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE 'articles_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
% sqlite-utils schema 24ways.db | sqlite3 /tmp/boo.db
Error: near line 15: table 'articles_fts_data' already exists
Error: near line 16: table 'articles_fts_idx' already exists
Error: near line 17: table 'articles_fts_docsize' already exists
Error: near line 18: table 'articles_fts_config' already exists
```
The problem here is that the `CREATE VIRTUAL TABLE ""articles_fts""...` line causes those next four tables to be created - but that means that piping the output of this command into `sqlite3` in order to re-create those tables throws errors.

I don't think this matters. I see this tool as more for introspection than for recreating table structures.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919181559,db.schema property and sqlite-utils schema command,
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859895540,https://api.github.com/repos/simonw/sqlite-utils/issues/268,859895540,MDEyOklzc3VlQ29tbWVudDg1OTg5NTU0MA==,9599,simonw,2021-06-11T20:30:34Z,2021-06-11T20:30:34Z,OWNER,"You can currently see the `sql` on the CLI using:

    % sqlite-utils rows fixtures.db sqlite_master -c name -c sql
    name                                                            sql
    --------------------------------------------------------------  ----------------------------------------------------------------------------------------------------------------------------------------------------------------
    simple_primary_key                                              CREATE TABLE simple_primary_key (
                                                                      id varchar(30) primary key,
                                                                      content text
                                                                    )
    sqlite_autoindex_simple_primary_key_1
    primary_key_multiple_columns                                    CREATE TABLE primary_key_multiple_columns (
                                                                      id varchar(30) primary key,
                                                                      content text,
                                                                      content2 text
                                                                    )
    sqlite_autoindex_primary_key_multiple_columns_1
    primary_key_multiple_columns_explicit_label                     CREATE TABLE primary_key_multiple_columns_explicit_label (
                                                                      id varchar(30) primary key,
                                                                      content text,
                                                                      content2 text
                                                                    )
    sqlite_autoindex_primary_key_multiple_columns_explicit_label_1
    compound_primary_key                                            CREATE TABLE compound_primary_key (
                                                                      pk1 varchar(30),
                                                                      pk2 varchar(30),
                                                                      content text,
                                                                      PRIMARY KEY (pk1, pk2)
                                                                    )
    sqlite_autoindex_compound_primary_key_1
    compound_three_primary_keys                                     CREATE TABLE compound_three_primary_keys (
                                                                      pk1 varchar(30),
                                                                      pk2 varchar(30),
                                                                      pk3 varchar(30),
                                                                      content text,
                                                                      PRIMARY KEY (pk1, pk2, pk3)
                                                                    )
    sqlite_autoindex_compound_three_primary_keys_1
    foreign_key_references                                          CREATE TABLE foreign_key_references (
                                                                      pk varchar(30) primary key,
                                                                      foreign_key_with_label varchar(30),
                                                                      foreign_key_with_no_label varchar(30),
                                                                      FOREIGN KEY (foreign_key_with_label) REFERENCES simple_primary_key(id),
                                                                      FOREIGN KEY (foreign_key_with_no_label) REFERENCES primary_key_multiple_columns(id)
                                                                    )
    sqlite_autoindex_foreign_key_references_1
    sortable                                                        CREATE TABLE sortable (
                                                                      pk1 varchar(30),
                                                                      pk2 varchar(30),
                                                                      content text,
                                                                      sortable integer,
                                                                      sortable_with_nulls real,
                                                                      sortable_with_nulls_2 real,
                                                                      text text,
                                                                      PRIMARY KEY (pk1, pk2)
                                                                    )
    sqlite_autoindex_sortable_1
    no_primary_key                                                  CREATE TABLE no_primary_key (
                                                                      content text,
                                                                      a text,
                                                                      b text,
                                                                      c text
                                                                    )
    123_starts_with_digits                                          CREATE TABLE [123_starts_with_digits] (
                                                                      content text
                                                                    )
    paginated_view                                                  CREATE VIEW paginated_view AS
                                                                        SELECT
                                                                            content,
                                                                            '- ' || content || ' -' AS content_extra
                                                                        FROM no_primary_key
    Table With Space In Name                                        CREATE TABLE ""Table With Space In Name"" (
                                                                      pk varchar(30) primary key,
                                                                      content text
                                                                    )
    sqlite_autoindex_Table With Space In Name_1
    table/with/slashes.csv                                          CREATE TABLE ""table/with/slashes.csv"" (
                                                                      pk varchar(30) primary key,
                                                                      content text
                                                                    )
    sqlite_autoindex_table/with/slashes.csv_1
    complex_foreign_keys                                            CREATE TABLE ""complex_foreign_keys"" (
                                                                      pk varchar(30) primary key,
                                                                      f1 text,
                                                                      f2 text,
                                                                      f3 text,
                                                                      FOREIGN KEY (""f1"") REFERENCES [simple_primary_key](id),
                                                                      FOREIGN KEY (""f2"") REFERENCES [simple_primary_key](id),
                                                                      FOREIGN KEY (""f3"") REFERENCES [simple_primary_key](id)
                                                                    )
    sqlite_autoindex_complex_foreign_keys_1
    custom_foreign_key_label                                        CREATE TABLE ""custom_foreign_key_label"" (
                                                                      pk varchar(30) primary key,
                                                                      foreign_key_with_custom_label text,
                                                                      FOREIGN KEY (""foreign_key_with_custom_label"") REFERENCES [primary_key_multiple_columns_explicit_label](id)
                                                                    )
    sqlite_autoindex_custom_foreign_key_label_1
    units                                                           CREATE TABLE units (
                                                                      pk integer primary key,
                                                                      distance int,
                                                                      frequency int
                                                                    )
    searchable                                                      CREATE TABLE searchable (
                                                                      pk integer primary key,
                                                                      text1 text,
                                                                      text2 text,
                                                                      [name with . and spaces] text
                                                                    )
    searchable_fts                                                  CREATE VIRTUAL TABLE ""searchable_fts""
                                                                        USING FTS3 (text1, text2, [name with . and spaces], content=""searchable"")
    searchable_fts_content                                          CREATE TABLE 'searchable_fts_content'(docid INTEGER PRIMARY KEY, 'c0text1', 'c1text2', 'c2name with . and spaces', 'c3content')
    searchable_fts_segments                                         CREATE TABLE 'searchable_fts_segments'(blockid INTEGER PRIMARY KEY, block BLOB)
    searchable_fts_segdir                                           CREATE TABLE 'searchable_fts_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx))
    sqlite_autoindex_searchable_fts_segdir_1
    select                                                          CREATE TABLE [select] (
                                                                      [group] text,
                                                                      [having] text,
                                                                      [and] text
                                                                    )
    facet_cities                                                    CREATE TABLE facet_cities (
                                                                        id integer primary key,
                                                                        name text
                                                                    )
    simple_view                                                     CREATE VIEW simple_view AS
                                                                        SELECT content, upper(content) AS upper_content FROM simple_primary_key
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919181559,db.schema property and sqlite-utils schema command,
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859894105,https://api.github.com/repos/simonw/sqlite-utils/issues/268,859894105,MDEyOklzc3VlQ29tbWVudDg1OTg5NDEwNQ==,9599,simonw,2021-06-11T20:28:52Z,2021-06-11T20:28:52Z,OWNER,"Out of interest, here are the rows from that table where `sql` is `null`: https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++sqlite_master%0D%0Awhere%0D%0A++sql+is+null

```csv
type,name,tbl_name,rootpage,sql
index,sqlite_autoindex_simple_primary_key_1,simple_primary_key,3,
index,sqlite_autoindex_primary_key_multiple_columns_1,primary_key_multiple_columns,5,
index,sqlite_autoindex_primary_key_multiple_columns_explicit_label_1,primary_key_multiple_columns_explicit_label,7,
index,sqlite_autoindex_compound_primary_key_1,compound_primary_key,9,
index,sqlite_autoindex_compound_three_primary_keys_1,compound_three_primary_keys,11,
index,sqlite_autoindex_foreign_key_references_1,foreign_key_references,14,
index,sqlite_autoindex_sortable_1,sortable,16,
index,sqlite_autoindex_Table With Space In Name_1,Table With Space In Name,20,
index,sqlite_autoindex_table/with/slashes.csv_1,table/with/slashes.csv,22,
index,sqlite_autoindex_complex_foreign_keys_1,complex_foreign_keys,24,
index,sqlite_autoindex_custom_foreign_key_label_1,custom_foreign_key_label,26,
index,sqlite_autoindex_tags_1,tags,31,
index,sqlite_autoindex_searchable_tags_1,searchable_tags,34,
index,sqlite_autoindex_searchable_fts_segdir_1,searchable_fts_segdir,37,
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919181559,db.schema property and sqlite-utils schema command,
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859888469,https://api.github.com/repos/simonw/sqlite-utils/issues/268,859888469,MDEyOklzc3VlQ29tbWVudDg1OTg4ODQ2OQ==,9599,simonw,2021-06-11T20:26:20Z,2021-06-11T20:26:20Z,OWNER,`sqlite-utils schema data.db` could output the same thing to the console.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919181559,db.schema property and sqlite-utils schema command,
https://github.com/simonw/datasette/issues/1371#issuecomment-858099514,https://api.github.com/repos/simonw/datasette/issues/1371,858099514,MDEyOklzc3VlQ29tbWVudDg1ODA5OTUxNA==,9599,simonw,2021-06-09T21:03:49Z,2021-06-09T21:03:49Z,OWNER,I'll release these as an alpha straight away - it makes sense to have plugin hook changes available for people to test as alpha dependencies ASAP.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915455228,Menu plugin hooks should include the request,
https://github.com/simonw/datasette/pull/1370#issuecomment-857139881,https://api.github.com/repos/simonw/datasette/issues/1370,857139881,MDEyOklzc3VlQ29tbWVudDg1NzEzOTg4MQ==,9599,simonw,2021-06-08T20:58:41Z,2021-06-08T20:58:41Z,OWNER,We can remove a bunch of unnecessary `str(path)` calls too - this search finds a bunch of possible candidates: https://ripgrep.datasette.io/-/ripgrep?pattern=str%5C%28.*%28db%7Cpath%29&glob=datasette%2F**%2F*.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",914130834,Ensure db.path is a string before trying to insert into internal database,
https://github.com/simonw/sqlite-utils/issues/266#issuecomment-856231119,https://api.github.com/repos/simonw/sqlite-utils/issues/266,856231119,MDEyOklzc3VlQ29tbWVudDg1NjIzMTExOQ==,9599,simonw,2021-06-07T20:26:05Z,2021-06-07T20:26:05Z,OWNER,"https://github.com/python/cpython/blob/2ab27c4af4ddf7528e1375e77c787c7fbb09b5e6/Lib/typing.py#L2173-L2195

In Python 3.6 or higher can do this:

```python
class Employee(NamedTuple):
    name: str
    id: int
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913135723,"Add some types, enforce with mypy",
https://github.com/simonw/datasette/issues/1365#issuecomment-856212136,https://api.github.com/repos/simonw/datasette/issues/1365,856212136,MDEyOklzc3VlQ29tbWVudDg1NjIxMjEzNg==,9599,simonw,2021-06-07T19:54:04Z,2021-06-07T19:54:04Z,OWNER,"I've hit this one too. I agree, fixing this in Datasette itself is better than fixing it in the tests across multiple other projects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913017577,pathlib.Path breaks internal schema,
https://github.com/simonw/datasette/issues/1369#issuecomment-856208637,https://api.github.com/repos/simonw/datasette/issues/1369,856208637,MDEyOklzc3VlQ29tbWVudDg1NjIwODYzNw==,9599,simonw,2021-06-07T19:47:23Z,2021-06-07T19:47:23Z,OWNER,No point in showing the IDs twice if the blue label doesn't differ from the gray ID,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913900374,Don't show foreign key IDs twice if no label,
https://github.com/simonw/datasette/issues/1367#issuecomment-856160770,https://api.github.com/repos/simonw/datasette/issues/1367,856160770,MDEyOklzc3VlQ29tbWVudDg1NjE2MDc3MA==,9599,simonw,2021-06-07T18:22:33Z,2021-06-07T18:22:33Z,OWNER,Here's why: https://github.com/simonw/datasette/blob/03ec71193b9545536898a4bc7493274fec48bdd7/datasette/static/app.css#L455-L458,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913823889,Navigation menu display bug,
https://github.com/simonw/datasette/issues/1366#issuecomment-856147969,https://api.github.com/repos/simonw/datasette/issues/1366,856147969,MDEyOklzc3VlQ29tbWVudDg1NjE0Nzk2OQ==,9599,simonw,2021-06-07T18:03:03Z,2021-06-07T18:03:03Z,OWNER,"Here's an example of a test that uses it. It's necessary because sometimes fixtures that create temporary directories break in unexpected ways:

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_plugins.py#L658-L666","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913809802,Get rid of this `restore_working_directory` hack entirely,
https://github.com/simonw/datasette/issues/1366#issuecomment-856147450,https://api.github.com/repos/simonw/datasette/issues/1366,856147450,MDEyOklzc3VlQ29tbWVudDg1NjE0NzQ1MA==,9599,simonw,2021-06-07T18:02:13Z,2021-06-07T18:02:13Z,OWNER,"The hack in question is this fixture, which I've been using in an ad-hoc manner to work around errors while running the tests: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/tests/conftest.py#L62-L75

I don't understand the underlying issue well enough to know how to get rid of it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913809802,Get rid of this `restore_working_directory` hack entirely,
https://github.com/simonw/sqlite-utils/issues/266#issuecomment-855611939,https://api.github.com/repos/simonw/sqlite-utils/issues/266,855611939,MDEyOklzc3VlQ29tbWVudDg1NTYxMTkzOQ==,9599,simonw,2021-06-07T06:07:41Z,2021-06-07T06:07:41Z,OWNER,"Looks like this is the way to do this:
```python
Point = typing.NamedTuple(
    ""Point"", [('x', int), ('y', int)]
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913135723,"Add some types, enforce with mypy",
https://github.com/simonw/datasette/issues/1362#issuecomment-855430317,https://api.github.com/repos/simonw/datasette/issues/1362,855430317,MDEyOklzc3VlQ29tbWVudDg1NTQzMDMxNw==,9599,simonw,2021-06-06T17:07:48Z,2021-06-06T17:07:48Z,OWNER,"I guess I can offer a `disable_csp` setting so that people with complex custom templates aren't completely blocked from using them with Datasette, but maybe it would be better not to offer that? Or to offer it as a `datasette-insecure-csp` plugin instead?

I like the idea of very actively encouraging CSP across all Datasette projects, but I'm nervous about making the software unusable for certain edge cases.

Maybe require CSP and wait for someone to complain?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855429111,https://api.github.com/repos/simonw/datasette/issues/1362,855429111,MDEyOklzc3VlQ29tbWVudDg1NTQyOTExMQ==,9599,simonw,2021-06-06T16:59:05Z,2021-06-06T17:00:15Z,OWNER,"Twitter conversation: https://twitter.com/simonw/status/1401565566045806594

@dracos provided some really useful code examples there:

> We generate it here: https://github.com/mysociety/fixmystreet/blob/e9fec4e567e7148ed128816e5770c2963be51af6/perllib/FixMyStreet/Cobrand/Default.pm#L89-L90
And use it e.g. https://github.com/mysociety/fixmystreet/blob/ba6788cd25d8f471a4e3308403607627b4d2f4f6/templates/web/base/common_header_tags.html or https://github.com/mysociety/fixmystreet/blob/cb4f2b96364d151988b5c664888468b25cc62240/templates/web/fixmystreet.com/header/css.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855428601,https://api.github.com/repos/simonw/datasette/issues/1362,855428601,MDEyOklzc3VlQ29tbWVudDg1NTQyODYwMQ==,9599,simonw,2021-06-06T16:55:33Z,2021-06-06T16:55:33Z,OWNER,"> No, because Vary header is about _request_ headers that cause the response to vary, not response headers.

Hah, of course! Thanks for the correction. So the nonce mechanism would actually be pretty great here, especially for the `extra_body_script()` hook.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855427396,https://api.github.com/repos/simonw/datasette/issues/1362,855427396,MDEyOklzc3VlQ29tbWVudDg1NTQyNzM5Ng==,9599,simonw,2021-06-06T16:46:17Z,2021-06-06T16:46:17Z,OWNER,"Mind you, since that plugin hook looks like this:

```python
@hookimpl
def extra_body_script():
    return {
        ""module"": True,
        ""script"": ""console.log('Your JavaScript goes here...')""
    }
```
Having it calculate a sha256 hash wouldn't be difficult.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855426750,https://api.github.com/repos/simonw/datasette/issues/1362,855426750,MDEyOklzc3VlQ29tbWVudDg1NTQyNjc1MA==,9599,simonw,2021-06-06T16:41:30Z,2021-06-06T16:44:49Z,OWNER,"This is from the current `base.html` template: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/base.html#L62-L66

Which includes this: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/_close_open_menus.html#L1-L16

The `body_scripts` bit is for this `extra_body_script` plugin hook, which is the thing that will be the most affected by implementing CSP: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855426516,https://api.github.com/repos/simonw/datasette/issues/1362,855426516,MDEyOklzc3VlQ29tbWVudDg1NTQyNjUxNg==,9599,simonw,2021-06-06T16:39:34Z,2021-06-06T16:39:34Z,OWNER,The reason Datasette uses small inline scripts right now is to avoid the overhead of an extra HTTP request for a JavaScript file - but these are both inherently cachable and perform much better under HTTP/2 so that's likely a false optimization.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855426314,https://api.github.com/repos/simonw/datasette/issues/1362,855426314,MDEyOklzc3VlQ29tbWVudDg1NTQyNjMxNA==,9599,simonw,2021-06-06T16:38:04Z,2021-06-06T16:38:04Z,OWNER,"The other option for inline scripts is the CSP nonce:

    Content-Security-Policy: script-src 'nonce-2726c7f26c'

Then:

    <script nonce=""2726c7f26c"">
      var inline = 1;
    </script>

Since an attacker can't guess what the nonce will be it prevents them from injecting their own script block - this seems easier to make available to plugins than a full hashing mechanism, just make `{{ csp_nonce() }}` available to the template.

That template function can then be smart enough to set a flag which Datasette uses to decide if the `script-src 'nonce-2726c7f26c'` policy should be sent or not.

Presumably this would also require adding `Content-Security-Policy` to the `Vary` header though, which will have a nasty effect on Cloudflare and Fastly and such like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418899,https://api.github.com/repos/simonw/datasette/issues/1362,855418899,MDEyOklzc3VlQ29tbWVudDg1NTQxODg5OQ==,9599,simonw,2021-06-06T15:42:55Z,2021-06-06T15:42:55Z,OWNER,Another consideration: testing that this works correctly could require adoption of a real browser test environment (probably Cypress or maybe Playwright) to execute tests that will fail if CSP is violated.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418698,https://api.github.com/repos/simonw/datasette/issues/1362,855418698,MDEyOklzc3VlQ29tbWVudDg1NTQxODY5OA==,9599,simonw,2021-06-06T15:41:24Z,2021-06-06T15:41:24Z,OWNER,"I think the best way to answer these questions is with some prototyping - of both Datasette and some of the existing JavaScript plugins.

I can start with a `datasette-experimental-csp` plugin that sets the header (and could even run an optional report URI mechanism).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418401,https://api.github.com/repos/simonw/datasette/issues/1362,855418401,MDEyOklzc3VlQ29tbWVudDg1NTQxODQwMQ==,9599,simonw,2021-06-06T15:39:38Z,2021-06-06T15:39:38Z,OWNER,"The security benefit of forcing all JavaScript plugins to be written as CSP-friendly external scripts is very compelling though.

Other plugin-heavy ecosystems such as WordPress have suffered greatly from insecurely written plugins - could this be a huge security win for the Datasette ecosystem generally?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418065,https://api.github.com/repos/simonw/datasette/issues/1362,855418065,MDEyOklzc3VlQ29tbWVudDg1NTQxODA2NQ==,9599,simonw,2021-06-06T15:37:11Z,2021-06-06T15:37:11Z,OWNER,"The easiest way to apply CSP is to remove all inline `<script>` blocks (Datasette has a few) and instead serve JavaScript as separate linked files.

It's possible to keep inline script blocks by calculating a hash of their content and adding a `Content-Security-Policy: script-src 'sha256-B2yPHKaXnvFWtRChIbabYmUBFZdVfKKXHbWtWidDVF8='` to the policy. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/script-src

This could be achieved with some Django template tricks, but it feels very risky - and done carelessly could end up calculating a hash of a reflected XSS attack!

The biggest challenge I see around here involves plugins and custom templates. Adopting CSP would require plugins to avoid using any inline scripts, instead keeping their entire implementations in `.js` files.

That's maybe not a bad thing, but it represents a big commitment. It would need to be adopted before Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1361#issuecomment-855308811,https://api.github.com/repos/simonw/datasette/issues/1361,855308811,MDEyOklzc3VlQ29tbWVudDg1NTMwODgxMQ==,9599,simonw,2021-06-05T23:16:21Z,2021-06-05T23:16:21Z,OWNER,That seems to have fixed it. I'd love to get rid of this `restore_working_directory` hack entirely.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912485040,Intermittent CI failure: restore_working_directory FileNotFoundError,
https://github.com/simonw/datasette/issues/1361#issuecomment-855307292,https://api.github.com/repos/simonw/datasette/issues/1361,855307292,MDEyOklzc3VlQ29tbWVudDg1NTMwNzI5Mg==,9599,simonw,2021-06-05T22:59:35Z,2021-06-05T22:59:35Z,OWNER,"That broke things. Here's how `pytest-cov` fixed a similar issue: https://github.com/pytest-dev/pytest-cov/commit/7ccb1783bf8290447df58deeb41eae74296a6d9b

See also https://github.com/nedbat/coveragepy/issues/881 and https://github.com/pytest-dev/pytest-cov/issues/306 and https://github.com/nedbat/coveragepy/issues/824","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912485040,Intermittent CI failure: restore_working_directory FileNotFoundError,
https://github.com/simonw/datasette/issues/1361#issuecomment-855306497,https://api.github.com/repos/simonw/datasette/issues/1361,855306497,MDEyOklzc3VlQ29tbWVudDg1NTMwNjQ5Nw==,9599,simonw,2021-06-05T22:51:01Z,2021-06-05T22:51:01Z,OWNER,I'm going to try removing that `restore_working_directory` fixture entirely.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912485040,Intermittent CI failure: restore_working_directory FileNotFoundError,
https://github.com/simonw/datasette/issues/1361#issuecomment-855306347,https://api.github.com/repos/simonw/datasette/issues/1361,855306347,MDEyOklzc3VlQ29tbWVudDg1NTMwNjM0Nw==,9599,simonw,2021-06-05T22:49:30Z,2021-06-05T22:49:30Z,OWNER,"Stack Overflow: https://stackoverflow.com/a/49367679/6083

> The answer was that `os.chdir()` had been set to the deleted directory by accident. The directory was missing, but the error happened (it seems) at the attempt to get it with `os.getcwd()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912485040,Intermittent CI failure: restore_working_directory FileNotFoundError,
https://github.com/simonw/datasette/issues/1360#issuecomment-855303776,https://api.github.com/repos/simonw/datasette/issues/1360,855303776,MDEyOklzc3VlQ29tbWVudDg1NTMwMzc3Ng==,9599,simonw,2021-06-05T22:23:23Z,2021-06-05T22:23:23Z,OWNER,"Worth noting that I found this issue myself, and to my knowledge it has not been uncovered by anyone else prior to the patch being released.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912464443,"Security flaw, to be fixed in 0.56.1 and 0.57",
https://github.com/simonw/datasette/issues/1360#issuecomment-855303649,https://api.github.com/repos/simonw/datasette/issues/1360,855303649,MDEyOklzc3VlQ29tbWVudDg1NTMwMzY0OQ==,9599,simonw,2021-06-05T22:22:06Z,2021-06-05T22:22:06Z,OWNER,I've released fixes in both 0.56.1 and 0.57.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912464443,"Security flaw, to be fixed in 0.56.1 and 0.57",
https://github.com/simonw/datasette/issues/1358#issuecomment-855302339,https://api.github.com/repos/simonw/datasette/issues/1358,855302339,MDEyOklzc3VlQ29tbWVudDg1NTMwMjMzOQ==,9599,simonw,2021-06-05T22:08:16Z,2021-06-05T22:08:16Z,OWNER,"Release notes are in, pushing the release now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912418094,Release Datasette 0.57,
https://github.com/simonw/datasette/issues/1358#issuecomment-855302320,https://api.github.com/repos/simonw/datasette/issues/1358,855302320,MDEyOklzc3VlQ29tbWVudDg1NTMwMjMyMA==,9599,simonw,2021-06-05T22:08:06Z,2021-06-05T22:08:06Z,OWNER,See #1360 for the security fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912418094,Release Datasette 0.57,
https://github.com/simonw/datasette/issues/283#issuecomment-855369819,https://api.github.com/repos/simonw/datasette/issues/283,855369819,MDEyOklzc3VlQ29tbWVudDg1NTM2OTgxOQ==,9599,simonw,2021-06-06T09:40:18Z,2021-06-06T09:40:18Z,OWNER,"> One note on using this pragma I got an error on starting datasette `no such table: pragma_database_list`.
> 
> I diagnosed this to an older version of sqlite3 (3.14.2) and upgrading to a newer version (3.34.2) fixed the issue.

That issue is fixed in #1276.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/1358#issuecomment-855288228,https://api.github.com/repos/simonw/datasette/issues/1358,855288228,MDEyOklzc3VlQ29tbWVudDg1NTI4ODIyOA==,9599,simonw,2021-06-05T19:57:18Z,2021-06-05T19:57:18Z,OWNER,"There's also a security fix I need to make, so I'm not going to block this on wrapping up the work on the new Docker image testing from #1344 before putting out this release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912418094,Release Datasette 0.57,
https://github.com/simonw/datasette/pull/1291#issuecomment-855287200,https://api.github.com/repos/simonw/datasette/issues/1291,855287200,MDEyOklzc3VlQ29tbWVudDg1NTI4NzIwMA==,9599,simonw,2021-06-05T19:48:36Z,2021-06-05T19:48:36Z,OWNER,"This is great, thank you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849582643,Update docs: explain allow_download setting,
https://github.com/simonw/datasette/issues/1356#issuecomment-855282466,https://api.github.com/repos/simonw/datasette/issues/1356,855282466,MDEyOklzc3VlQ29tbWVudDg1NTI4MjQ2Ng==,9599,simonw,2021-06-05T19:05:06Z,2021-06-05T19:05:06Z,OWNER,Yeah that's a good point. I avoided making them sub-commands because `datasette serve` already supports the multitude of other arguments they also need... but actually that was just me being lazy - I can easily share arguments between multiple functions like I do in `sqlite-utils` itself: https://github.com/simonw/sqlite-utils/blob/d1a372b3006e6cf7d2017b3ddc484bf5c033e45d/sqlite_utils/cli.py#L46,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1238#issuecomment-855278998,https://api.github.com/repos/simonw/datasette/issues/1238,855278998,MDEyOklzc3VlQ29tbWVudDg1NTI3ODk5OA==,9599,simonw,2021-06-05T18:37:16Z,2021-06-05T18:37:16Z,OWNER,"Alternative idea: populate `request.scope` with a new `route_path` which is the base-url-stripped version, which we then use for other routing operations.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-855278540,https://api.github.com/repos/simonw/datasette/issues/1238,855278540,MDEyOklzc3VlQ29tbWVudDg1NTI3ODU0MA==,9599,simonw,2021-06-05T18:33:25Z,2021-06-05T18:33:25Z,OWNER,"Got the test to pass by ensuring the tests don't accidentally double-rewrite the path.

Ran into a new problem:
```
    @pytest.mark.asyncio
    @pytest.mark.parametrize(
        ""prefix,expected_path"", [(None, ""/asgi-scope""), (""/prefix/"", ""/prefix/asgi-scope"")]
    )
    async def test_client_path(datasette, prefix, expected_path):
        original_base_url = datasette._settings[""base_url""]
        try:
            if prefix is not None:
                datasette._settings[""base_url""] = prefix
            response = await datasette.client.get(""/asgi-scope"")
            path = response.json()[""path""]
>           assert path == expected_path
E           AssertionError: assert '/asgi-scope' == '/prefix/asgi-scope'
E             - /prefix/asgi-scope
E             ? -------
E             + /asgi-scope
```
That test confirms that messing around with the `base_url` doesn't modify the ASGI scope... but the fix I'm using for this issue DOES modify the ASGI scope.

The question raised here is: should the ASGI scope stay unmodified when `base_url` is used?

I think it should. It doesn't make sense to obscure the ""real"" path just to get custom pages to work properly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-855272693,https://api.github.com/repos/simonw/datasette/issues/1238,855272693,MDEyOklzc3VlQ29tbWVudDg1NTI3MjY5Mw==,9599,simonw,2021-06-05T17:45:42Z,2021-06-05T17:45:42Z,OWNER,"Applying this fix worked when I manually tested it:
```diff
         base_url = self.ds.setting(""base_url"")
         if base_url != ""/"" and path.startswith(base_url):
             path = ""/"" + path[len(base_url) :]
+            scope = dict(scope, path=path, raw_path=path.encode(""utf-8""))
         request = Request(scope, receive)
```
But... the test I wrote still failed. My hunch is that this is because deep within the test framework requests go through `ds.client` which may be applying its own changes relevant to `base_url`:

https://github.com/simonw/datasette/blob/6e9b07be92905011211d8df7a872fb7c1f2737b2/datasette/utils/testing.py#L139","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-855270917,https://api.github.com/repos/simonw/datasette/issues/1238,855270917,MDEyOklzc3VlQ29tbWVudDg1NTI3MDkxNw==,9599,simonw,2021-06-05T17:32:29Z,2021-06-05T17:32:29Z,OWNER,"This looks like the cause: https://github.com/simonw/datasette/blob/6e9b07be92905011211d8df7a872fb7c1f2737b2/datasette/app.py#L1087-L1092

Note how `path` is modified... but then we create a new `Request()` that uses the old scope, which has unmodified `scope[""path""]` - and then the code later on looks at `request.scope[""path""]` when deciding if the request matches:

https://github.com/simonw/datasette/blob/afed51b1e36cf275c39e71c7cb262d6c5bdbaa31/datasette/app.py#L1154-L1155","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853535559,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853535559,MDEyOklzc3VlQ29tbWVudDg1MzUzNTU1OQ==,9599,simonw,2021-06-03T03:32:47Z,2021-06-03T03:32:47Z,OWNER,"New design:
```python
def test_xindexes(fresh_db):
    fresh_db.executescript(
        """"""
        create table Gosh (c1 text, c2 text, c3 text);
        create index Gosh_c1 on Gosh(c1);
        create index Gosh_c2c3 on Gosh(c2, c3 desc);
    """"""
    )
    assert fresh_db[""Gosh""].xindexes == [
        XIndex(
            name=""Gosh_c2c3"",
            columns=[
                XIndexColumn(seqno=0, cid=1, name=""c2"", desc=0, coll=""BINARY"", key=1),
                XIndexColumn(seqno=1, cid=2, name=""c3"", desc=1, coll=""BINARY"", key=1),
                XIndexColumn(seqno=2, cid=-1, name=None, desc=0, coll=""BINARY"", key=0),
            ],
        ),
        XIndex(
            name=""Gosh_c1"",
            columns=[
                XIndexColumn(seqno=0, cid=0, name=""c1"", desc=0, coll=""BINARY"", key=1),
                XIndexColumn(seqno=1, cid=-1, name=None, desc=0, coll=""BINARY"", key=0),
            ],
        ),
    ]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853534732,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853534732,MDEyOklzc3VlQ29tbWVudDg1MzUzNDczMg==,9599,simonw,2021-06-03T03:30:10Z,2021-06-03T03:30:10Z,OWNER,"I'm going to return `XIndex(name, columns)`  - where `columns` is a list of `XIndexColumn`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853530348,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853530348,MDEyOklzc3VlQ29tbWVudDg1MzUzMDM0OA==,9599,simonw,2021-06-03T03:16:33Z,2021-06-03T03:16:33Z,OWNER,"In prototyping this out I realize that I actually want to get back the name of each index, then for each of them the detailed list of index columns. Here's the test from my initial prototype:
```python
def test_xindexes(fresh_db):
    fresh_db.executescript(
        """"""
        create table Gosh (c1 text, c2 text, c3 text);
        create index Gosh_c1 on Gosh(c1);
        create index Gosh_c2c3 on Gosh(c2, c3 desc);
    """"""
    )
    assert fresh_db[""Gosh""].xindexes == [
        (
            ""Gosh_c2c3"",
            [
                XIndex(seqno=0, cid=1, name=""c2"", desc=0, coll=""BINARY"", key=1),
                XIndex(seqno=1, cid=2, name=""c3"", desc=1, coll=""BINARY"", key=1),
                XIndex(seqno=2, cid=-1, name=None, desc=0, coll=""BINARY"", key=0),
            ],
        ),
        (
            ""Gosh_c1"",
            [
                XIndex(seqno=0, cid=0, name=""c1"", desc=0, coll=""BINARY"", key=1),
                XIndex(seqno=1, cid=-1, name=None, desc=0, coll=""BINARY"", key=0),
            ],
        ),
    ]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853525036,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853525036,MDEyOklzc3VlQ29tbWVudDg1MzUyNTAzNg==,9599,simonw,2021-06-03T03:02:22Z,2021-06-03T03:02:22Z,OWNER,"This would be a breaking change - and the fact that it returns auxiliary columns isn't particularly useful for most cases - ""Auxiliary columns are additional columns needed to locate the table entry that corresponds to each index entry"".

Instead, I'm going to add a new property `table.xindexes` which exposes this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853567861,https://api.github.com/repos/simonw/sqlite-utils/issues/264,853567861,MDEyOklzc3VlQ29tbWVudDg1MzU2Nzg2MQ==,9599,simonw,2021-06-03T05:12:21Z,2021-06-03T05:12:21Z,OWNER,I think this is more likely to happen in Datasette than in sqlite-utils - see https://github.com/simonw/datasette/issues/1356 for thoughts on this.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907642546,"Supporting additional output formats, like GeoJSON",
https://github.com/simonw/datasette/issues/526#issuecomment-853567413,https://api.github.com/repos/simonw/datasette/issues/526,853567413,MDEyOklzc3VlQ29tbWVudDg1MzU2NzQxMw==,9599,simonw,2021-06-03T05:11:27Z,2021-06-03T05:11:27Z,OWNER,"Another potential way to implement this would be to hold the SQLite connection open and execute the full query there.

I've avoided this in the past due to concerns of resource exhaustion - if multiple requests attempt this at the same time all of the connections in the pool will become tied up and the site will be unable to respond to further requests.

But... now that Datasette has authentication there's the possibility of making this feature only available to specific authenticated users - the `--root` user for example. Which avoids the danger while unlocking a super-useful feature.

Not to mention people who are running Datasette privately on their own laptop, or the proposed `--query` CLI feature in #1356.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/1356#issuecomment-853566337,https://api.github.com/repos/simonw/datasette/issues/1356,853566337,MDEyOklzc3VlQ29tbWVudDg1MzU2NjMzNw==,9599,simonw,2021-06-03T05:08:32Z,2021-06-03T05:08:32Z,OWNER,Also relevant: CSV streaming for canned queries in #526 - even better if we could stream ANY SQL query.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1062#issuecomment-853566011,https://api.github.com/repos/simonw/datasette/issues/1062,853566011,MDEyOklzc3VlQ29tbWVudDg1MzU2NjAxMQ==,9599,simonw,2021-06-03T05:07:42Z,2021-06-03T05:07:42Z,OWNER,Implementing this would make #1356 a whole lot more interesting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,
https://github.com/simonw/datasette/issues/1356#issuecomment-853565850,https://api.github.com/repos/simonw/datasette/issues/1356,853565850,MDEyOklzc3VlQ29tbWVudDg1MzU2NTg1MA==,9599,simonw,2021-06-03T05:07:21Z,2021-06-03T05:07:21Z,OWNER,"A problem with this is that if you're using `--query` you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is `.csv` and plugin formats can't handle full streams, see #1062 and #1177.

So there's not much point implementing this unless we first make plugins able to add custom streaming formats.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853562891,https://api.github.com/repos/simonw/datasette/issues/1356,853562891,MDEyOklzc3VlQ29tbWVudDg1MzU2Mjg5MQ==,9599,simonw,2021-06-03T04:59:40Z,2021-06-03T04:59:40Z,OWNER,"It's weird that `--get` is documented on this page right now: https://docs.datasette.io/en/stable/getting_started.html#datasette-get

If I implement this I should build a new ""Datasette on the command-line"" documentation page to cover both `--get` and `--query`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853560870,https://api.github.com/repos/simonw/datasette/issues/1356,853560870,MDEyOklzc3VlQ29tbWVudDg1MzU2MDg3MA==,9599,simonw,2021-06-03T04:53:47Z,2021-06-03T04:53:56Z,OWNER,"This is also interesting when used in conjunction with the proposed `datasette insert` command from #1163 - Datasette would become a plugin-driven CLI tool for both ingesting and outputting data, as a side-effect of its web features.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853560389,https://api.github.com/repos/simonw/datasette/issues/1356,853560389,MDEyOklzc3VlQ29tbWVudDg1MzU2MDM4OQ==,9599,simonw,2021-06-03T04:52:13Z,2021-06-03T04:52:13Z,OWNER,I should implement #1355 for more efficient `--csv` streaming as part of this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853560237,https://api.github.com/repos/simonw/datasette/issues/1356,853560237,MDEyOklzc3VlQ29tbWVudDg1MzU2MDIzNw==,9599,simonw,2021-06-03T04:51:49Z,2021-06-03T04:51:49Z,OWNER,This feels like a relatively simple feature to implement that unlocks a whole new set of possible uses for Datasette - as described by @eyeseast in https://github.com/simonw/sqlite-utils/issues/264#issue-907642546.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853559915,https://api.github.com/repos/simonw/datasette/issues/1356,853559915,MDEyOklzc3VlQ29tbWVudDg1MzU1OTkxNQ==,9599,simonw,2021-06-03T04:50:52Z,2021-06-03T04:50:52Z,OWNER,"What happens if you pass multiple databases? The `--query` would be executed against the first one. And if you pass `--crossdb` it would be executed against the `/_memory` database and would support cross-database joins.

Key thing here is that output plugins are supported (also plugins that add new SQL functions), making many Datasette plugins usable from the command-line.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853558741,https://api.github.com/repos/simonw/sqlite-utils/issues/264,853558741,MDEyOklzc3VlQ29tbWVudDg1MzU1ODc0MQ==,9599,simonw,2021-06-03T04:47:19Z,2021-06-03T04:47:19Z,OWNER,"This inspired me to re-examine how `--get` works, hence this issue: https://github.com/simonw/datasette/issues/1355","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907642546,"Supporting additional output formats, like GeoJSON",
https://github.com/simonw/datasette/issues/1355#issuecomment-853557439,https://api.github.com/repos/simonw/datasette/issues/1355,853557439,MDEyOklzc3VlQ29tbWVudDg1MzU1NzQzOQ==,9599,simonw,2021-06-03T04:43:14Z,2021-06-03T04:43:14Z,OWNER,"It's using `TestClient` at the moment which is a wrapper around `httpx` (as of ) that uses the `@async_to_sync` decorator to hide the async nature.

https://github.com/simonw/datasette/blob/f78ebdc04537a6102316d6dbbf6c887565806078/datasette/utils/testing.py#L102-L156

Maybe the fix here is to switch the `--get` implementation to using `httpx` directly with https://www.python-httpx.org/async/#streaming-responses","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910088936,datasette --get should efficiently handle streaming CSV,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853554550,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853554550,MDEyOklzc3VlQ29tbWVudDg1MzU1NDU1MA==,9599,simonw,2021-06-03T04:34:38Z,2021-06-03T04:34:38Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#listing-indexes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853553754,https://api.github.com/repos/simonw/sqlite-utils/issues/264,853553754,MDEyOklzc3VlQ29tbWVudDg1MzU1Mzc1NA==,9599,simonw,2021-06-03T04:32:42Z,2021-06-03T04:36:36Z,OWNER,"This is a really interesting thought. I've so far resisted the temptation to add plugins to `sqlite-utils`, partly to avoid overlap with Datasette - but I'm open to discussing it.

There's actually a way for you to do what you're describing using `datasette` on the command-line, though it's a little obscure - also Datasette doesn't yet have a GeoJSON output extension, though it really should have one.

Here's an example using [datasette-yaml](https://datasette.io/plugins/datasette-yaml):

```
datasette /tmp/covid.db --get='/covid/ny_times_us_counties.yaml'
- rowid: 1
  date: '2020-01-21'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
- rowid: 2
  date: '2020-01-22'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
```
It even works with arbitrary SQL queries, though you might have to apply URL encoding to the `--get` string (this seems to work though):
```
datasette /tmp/covid.db --get='/covid.yaml?sql=select * from ny_times_us_counties limit 2' 
- date: '2020-01-21'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
- date: '2020-01-22'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
```
Here's the documentation for `--get`: https://docs.datasette.io/en/latest/getting_started.html#datasette-get","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907642546,"Supporting additional output formats, like GeoJSON",
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853548442,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853548442,MDEyOklzc3VlQ29tbWVudDg1MzU0ODQ0Mg==,9599,simonw,2021-06-03T04:16:49Z,2021-06-03T04:16:49Z,OWNER,Needs to show the table each index applies to.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853547681,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853547681,MDEyOklzc3VlQ29tbWVudDg1MzU0NzY4MQ==,9599,simonw,2021-06-03T04:14:28Z,2021-06-03T04:14:28Z,OWNER,"This looks good:
```
% sqlite-utils indexes /tmp/covid.db -t      
index_name                                                seqno    cid  name                 desc  coll      key
------------------------------------------------------  -------  -----  -----------------  ------  ------  -----
idx_johns_hopkins_csse_daily_reports_combined_key             0     12  combined_key            0  BINARY      1
idx_johns_hopkins_csse_daily_reports_country_or_region        0      1  country_or_region       0  BINARY      1
idx_johns_hopkins_csse_daily_reports_province_or_state        0      2  province_or_state       0  BINARY      1
idx_johns_hopkins_csse_daily_reports_day                      0      0  day                     0  BINARY      1
idx_ny_times_us_counties_date                                 0      0  date                    1  BINARY      1
idx_ny_times_us_counties_fips                                 0      3  fips                    0  BINARY      1
idx_ny_times_us_counties_county                               0      1  county                  0  BINARY      1
idx_ny_times_us_counties_state                                0      2  state                   0  BINARY      1

% sqlite-utils indexes /tmp/covid.db -t --aux
index_name                                                seqno    cid  name                 desc  coll      key
------------------------------------------------------  -------  -----  -----------------  ------  ------  -----
idx_johns_hopkins_csse_daily_reports_combined_key             0     12  combined_key            0  BINARY      1
idx_johns_hopkins_csse_daily_reports_combined_key             1     -1                          0  BINARY      0
idx_johns_hopkins_csse_daily_reports_country_or_region        0      1  country_or_region       0  BINARY      1
idx_johns_hopkins_csse_daily_reports_country_or_region        1     -1                          0  BINARY      0
idx_johns_hopkins_csse_daily_reports_province_or_state        0      2  province_or_state       0  BINARY      1
idx_johns_hopkins_csse_daily_reports_province_or_state        1     -1                          0  BINARY      0
idx_johns_hopkins_csse_daily_reports_day                      0      0  day                     0  BINARY      1
idx_johns_hopkins_csse_daily_reports_day                      1     -1                          0  BINARY      0
idx_ny_times_us_counties_date                                 0      0  date                    1  BINARY      1
idx_ny_times_us_counties_date                                 1     -1                          0  BINARY      0
idx_ny_times_us_counties_fips                                 0      3  fips                    0  BINARY      1
idx_ny_times_us_counties_fips                                 1     -1                          0  BINARY      0
idx_ny_times_us_counties_county                               0      1  county                  0  BINARY      1
idx_ny_times_us_counties_county                               1     -1                          0  BINARY      0
idx_ny_times_us_counties_state                                0      2  state                   0  BINARY      1
idx_ny_times_us_counties_state                                1     -1                          0  BINARY      0
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853546818,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853546818,MDEyOklzc3VlQ29tbWVudDg1MzU0NjgxOA==,9599,simonw,2021-06-03T04:11:46Z,2021-06-03T04:11:46Z,OWNER,"By default I won't return auxiliary columns, but I'll offer a `--aux` option to return them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853545743,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853545743,MDEyOklzc3VlQ29tbWVudDg1MzU0NTc0Mw==,9599,simonw,2021-06-03T04:08:04Z,2021-06-03T04:08:04Z,OWNER,"Figuring out the right queries:

https://covid-19.datasettes.com/covid?sql=select+sqlite_master.name%2C+i.*+from+sqlite_master%0D%0Ajoin+pragma_index_list%28sqlite_master.name%29+i%0D%0Awhere+type+%3D+%27table%27

This query shows all columns across all indexes across all tables:

```sql
select i.name as index_name, xinfo.* from sqlite_master
join pragma_index_list(sqlite_master.name) i
join pragma_index_xinfo(index_name) xinfo
where sqlite_master.type = 'table'
```
https://covid-19.datasettes.com/covid?sql=select+i.name+as+index_name%2C+xinfo.*+from+sqlite_master%0D%0Ajoin+pragma_index_list%28sqlite_master.name%29+i%0D%0Ajoin+pragma_index_xinfo%28index_name%29+xinfo%0D%0Awhere+sqlite_master.type+%3D+%27table%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853544493,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853544493,MDEyOklzc3VlQ29tbWVudDg1MzU0NDQ5Mw==,9599,simonw,2021-06-03T04:03:59Z,2021-06-03T04:03:59Z,OWNER,"Here's how `sqlite-utils triggers` works: https://github.com/simonw/sqlite-utils/blob/9c67cb925253cd5ef54a1fe0496e0ff9caeacfd6/sqlite_utils/cli.py#L1266-L1277

Running it from a SQL query makes it easy to support modifiers like `--csv` and `-t`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853541869,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853541869,MDEyOklzc3VlQ29tbWVudDg1MzU0MTg2OQ==,9599,simonw,2021-06-03T03:54:14Z,2021-06-03T03:54:14Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#xindexes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/datasette/issues/893#issuecomment-852712469,https://api.github.com/repos/simonw/datasette/issues/893,852712469,MDEyOklzc3VlQ29tbWVudDg1MjcxMjQ2OQ==,9599,simonw,2021-06-02T04:29:56Z,2021-06-02T04:29:56Z,OWNER,"I think I fixed this a while ago, though I can't find the commit now. Please re-open if the bug still happens!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",656959584,pip3 install datasette not serving static on linuxbrew.,
https://github.com/simonw/datasette/issues/1127#issuecomment-852712106,https://api.github.com/repos/simonw/datasette/issues/1127,852712106,MDEyOklzc3VlQ29tbWVudDg1MjcxMjEwNg==,9599,simonw,2021-06-02T04:28:55Z,2021-06-02T04:28:55Z,OWNER,This became resizable in #1236.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756818250,Make the custom SQL query text box larger or resizable,
https://github.com/simonw/datasette/pull/1306#issuecomment-852710117,https://api.github.com/repos/simonw/datasette/issues/1306,852710117,MDEyOklzc3VlQ29tbWVudDg1MjcxMDExNw==,9599,simonw,2021-06-02T04:24:33Z,2021-06-02T04:24:33Z,OWNER,I have a test - I'll merge this and then add that test.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",864979486,Avoid error sorting by relationships if related tables are not allowed,
https://github.com/simonw/datasette/pull/1306#issuecomment-852708574,https://api.github.com/repos/simonw/datasette/issues/1306,852708574,MDEyOklzc3VlQ29tbWVudDg1MjcwODU3NA==,9599,simonw,2021-06-02T04:20:18Z,2021-06-02T04:20:18Z,OWNER,https://github.com/simonw/datasette/blob/115332ce76c0e867d9936406aaf4bcee6b1ef3cb/datasette/views/index.py#L77 helps here - looks like we can pass `?_sort=relationships` to trigger the bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",864979486,Avoid error sorting by relationships if related tables are not allowed,
https://github.com/simonw/datasette/pull/1306#issuecomment-852707676,https://api.github.com/repos/simonw/datasette/issues/1306,852707676,MDEyOklzc3VlQ29tbWVudDg1MjcwNzY3Ng==,9599,simonw,2021-06-02T04:17:41Z,2021-06-02T04:17:41Z,OWNER,We need to add a test that illustrates the bug in #1305 so we can prove that this definitely fixes it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",864979486,Avoid error sorting by relationships if related tables are not allowed,
https://github.com/simonw/datasette/issues/619#issuecomment-852706970,https://api.github.com/repos/simonw/datasette/issues/619,852706970,MDEyOklzc3VlQ29tbWVudDg1MjcwNjk3MA==,9599,simonw,2021-06-02T04:15:54Z,2021-06-02T04:15:54Z,OWNER,"<img width=""1009"" alt=""fixtures__select___from__foo__and_Facets_should_not_execute_for__shape_array_object_·_Issue__263_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/120423073-8a338700-c31e-11eb-8c8f-affc24708b32.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-852703357,https://api.github.com/repos/simonw/datasette/issues/619,852703357,MDEyOklzc3VlQ29tbWVudDg1MjcwMzM1Nw==,9599,simonw,2021-06-02T04:08:03Z,2021-06-02T04:08:03Z,OWNER,"A SQL error now looks like this: https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++%5Bfoo%5D

<img width=""1023"" alt=""fixtures__select___from__foo_"" src=""https://user-images.githubusercontent.com/9599/120422579-691e6680-c31d-11eb-99ea-655bdc2a401d.png"">

I'm going to get rid of that ""0 results"" message if an error is shown.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/1257#issuecomment-852686827,https://api.github.com/repos/simonw/datasette/issues/1257,852686827,MDEyOklzc3VlQ29tbWVudDg1MjY4NjgyNw==,9599,simonw,2021-06-02T03:26:27Z,2021-06-02T03:26:27Z,OWNER,I tried and failed to get this fix working for tables with double quotes in their name - I couldn't figure out what the double-quote-in-a-table-name version of this code would look like: https://github.com/simonw/datasette/blob/807de378d08752a0f05bb1b980a0a62620a70520/datasette/utils/__init__.py#L538-L548,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828811618,Table names containing single quotes break things,
https://github.com/simonw/datasette/issues/1257#issuecomment-852681622,https://api.github.com/repos/simonw/datasette/issues/1257,852681622,MDEyOklzc3VlQ29tbWVudDg1MjY4MTYyMg==,9599,simonw,2021-06-02T03:12:18Z,2021-06-02T03:12:18Z,OWNER,"Created a test database like this:
```
% sqlite-utils create-table quote-in-name.db ""this'hasquoteinname"" id integer name text --pk id
% datasette quote-in-name.db -p 8025 --pdb
INFO:     Started server process [86046]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8025 (Press CTRL+C to quit)
> /Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py(530)detect_fts()
-> rows = conn.execute(detect_fts_sql(table)).fetchall()
(Pdb) c
Traceback (most recent call last):
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 1124, in route_path
    response = await view(request, send)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 147, in view
    return await self.dispatch_request(
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 122, in dispatch_request
    return await handler(request, *args, **kwargs)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/index.py"", line 72, in get
    ""fts_table"": await db.fts_table(table),
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 279, in fts_table
    return await self.execute_fn(lambda conn: detect_fts(conn, table))
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 155, in execute_fn
    return await asyncio.get_event_loop().run_in_executor(
  File ""/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/concurrent/futures/thread.py"", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 153, in in_thread
    return fn(conn)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 279, in <lambda>
    return await self.execute_fn(lambda conn: detect_fts(conn, table))
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 530, in detect_fts
    rows = conn.execute(detect_fts_sql(table)).fetchall()
sqlite3.OperationalError: near ""hasquoteinname"": syntax error
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828811618,Table names containing single quotes break things,
https://github.com/simonw/datasette/pull/1352#issuecomment-852673695,https://api.github.com/repos/simonw/datasette/issues/1352,852673695,MDEyOklzc3VlQ29tbWVudDg1MjY3MzY5NQ==,9599,simonw,2021-06-02T02:52:26Z,2021-06-02T02:52:26Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",908276134,Bump black from 21.5b1 to 21.5b2,
https://github.com/simonw/datasette/issues/1353#issuecomment-852238201,https://api.github.com/repos/simonw/datasette/issues/1353,852238201,MDEyOklzc3VlQ29tbWVudDg1MjIzODIwMQ==,9599,simonw,2021-06-01T15:53:57Z,2021-06-01T15:53:57Z,OWNER,I'm going to rename `?_nofacets=1` to `?_nofacet=1` to keep it consistent with the new `?_nocount=1` option (and because I don't like `?_nocounts=1`).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",908446997,?_nocount=1 for opting out of table counts,
https://github.com/simonw/datasette/issues/1349#issuecomment-852237347,https://api.github.com/repos/simonw/datasette/issues/1349,852237347,MDEyOklzc3VlQ29tbWVudDg1MjIzNzM0Nw==,9599,simonw,2021-06-01T15:52:50Z,2021-06-01T15:52:50Z,OWNER,Fixed in https://github.com/simonw/datasette/commit/d1d06ace49606da790a765689b4fbffa4c6deecb,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/263#issuecomment-852256784,https://api.github.com/repos/simonw/datasette/issues/263,852256784,MDEyOklzc3VlQ29tbWVudDg1MjI1Njc4NA==,9599,simonw,2021-06-01T16:20:09Z,2021-06-01T16:20:09Z,OWNER,This is a lot easier to implement now that we have a `?_nofacet=1` option from #1350.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323671577,Facets should not execute for ?shape=array|object,
https://github.com/simonw/datasette/issues/1350#issuecomment-852256454,https://api.github.com/repos/simonw/datasette/issues/1350,852256454,MDEyOklzc3VlQ29tbWVudDg1MjI1NjQ1NA==,9599,simonw,2021-06-01T16:19:38Z,2021-06-01T16:19:38Z,OWNER,I renamed this to `?_nofacet=1` in #1353.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906977719,?_nofacets=1 query string argument for disabling facets and suggested facets,
https://github.com/simonw/datasette/issues/1353#issuecomment-852255951,https://api.github.com/repos/simonw/datasette/issues/1353,852255951,MDEyOklzc3VlQ29tbWVudDg1MjI1NTk1MQ==,9599,simonw,2021-06-01T16:18:54Z,2021-06-01T16:18:54Z,OWNER,Documented here: https://docs.datasette.io/en/latest/json_api.html#special-table-arguments,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",908446997,?_nocount=1 for opting out of table counts,
https://github.com/simonw/datasette/issues/1354#issuecomment-852254298,https://api.github.com/repos/simonw/datasette/issues/1354,852254298,MDEyOklzc3VlQ29tbWVudDg1MjI1NDI5OA==,9599,simonw,2021-06-01T16:16:32Z,2021-06-01T16:16:32Z,OWNER,Running `python update-docs-help.py` helps fix this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",908465747,Update help in tests for latest Click,
https://github.com/simonw/datasette/issues/1351#issuecomment-851133471,https://api.github.com/repos/simonw/datasette/issues/1351,851133471,MDEyOklzc3VlQ29tbWVudDg1MTEzMzQ3MQ==,9599,simonw,2021-05-31T03:02:59Z,2021-05-31T03:02:59Z,OWNER,"Since traces only work with `text/html` and JSON at the moment, the easiest way to do this will be to wrap generated CSV in a HTML page in a textarea if the user specified `?_trace=1`:

https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/tracer.py#L125-L134","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906993731,Get `?_trace=1` working with CSV and streaming CSVs,
https://github.com/simonw/datasette/issues/1349#issuecomment-851133125,https://api.github.com/repos/simonw/datasette/issues/1349,851133125,MDEyOklzc3VlQ29tbWVudDg1MTEzMzEyNQ==,9599,simonw,2021-05-31T03:01:48Z,2021-05-31T03:01:48Z,OWNER,I think it's worth getting `?_trace=1` to work with streaming CSV - this would have helped me spot this issue a long time ago.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-851131999,https://api.github.com/repos/simonw/datasette/issues/1349,851131999,MDEyOklzc3VlQ29tbWVudDg1MTEzMTk5OQ==,9599,simonw,2021-05-31T02:57:25Z,2021-05-31T02:57:25Z,OWNER,"I'm having a really hard time figuring out how to unit test this - ideally I'd monitor which SQL queries are executed using the tracing mechanism, but that's not set up to work with anything other than HTML or JSON outputs: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/tracer.py#L125-L134","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-851129464,https://api.github.com/repos/simonw/datasette/issues/1349,851129464,MDEyOklzc3VlQ29tbWVudDg1MTEyOTQ2NA==,9599,simonw,2021-05-31T02:48:06Z,2021-05-31T02:48:06Z,OWNER,"Actually there is precedent for swapping out `request.scope` for a new scope, as seen here in the routing code: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/app.py#L1117-L1122","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-851129019,https://api.github.com/repos/simonw/datasette/issues/1349,851129019,MDEyOklzc3VlQ29tbWVudDg1MTEyOTAxOQ==,9599,simonw,2021-05-31T02:46:38Z,2021-05-31T02:46:38Z,OWNER,I think the right way to do this is to construct a new `Request` with a modified ASGI scope: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/utils/asgi.py#L88-L99,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-851128524,https://api.github.com/repos/simonw/datasette/issues/1349,851128524,MDEyOklzc3VlQ29tbWVudDg1MTEyODUyNA==,9599,simonw,2021-05-31T02:44:44Z,2021-05-31T02:44:44Z,OWNER,"Now that I have `?_nofacets=1` I can use that to fix this. The challenge is that in this block of code I need to modify the incoming request's query string arguments, which isn't something I've done before: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/base.py#L263-L270
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1345#issuecomment-851127314,https://api.github.com/repos/simonw/datasette/issues/1345,851127314,MDEyOklzc3VlQ29tbWVudDg1MTEyNzMxNA==,9599,simonw,2021-05-31T02:40:26Z,2021-05-31T02:40:44Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet=state&_nocol=state - the state column is not selected but facet by state still works: 

<img width=""1134"" alt=""fixtures__facetable__15_rows_and_Why_Russians_do_not_smile"" src=""https://user-images.githubusercontent.com/9599/120132273-110a2780-c198-11eb-91a5-cafb77a78c97.png"">

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1350#issuecomment-851122934,https://api.github.com/repos/simonw/datasette/issues/1350,851122934,MDEyOklzc3VlQ29tbWVudDg1MTEyMjkzNA==,9599,simonw,2021-05-31T02:23:47Z,2021-05-31T02:23:47Z,OWNER,"I think `?_facets_off=1` is a good design for this.

I considered `?_facet=` with an empty string but that's a bit weird, and it's not clear what that would do if used like `?_facet=state&_facet=`.

So it definitely needs to be a separate named option.

Actually, since I have `?_nocol=` I'm going to call this `?_nofacets=1` instead as that's a little more consistent.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906977719,?_nofacets=1 query string argument for disabling facets and suggested facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851120306,https://api.github.com/repos/simonw/datasette/issues/1345,851120306,MDEyOklzc3VlQ29tbWVudDg1MTEyMDMwNg==,9599,simonw,2021-05-31T02:14:36Z,2021-05-31T02:14:36Z,OWNER,"Yes! This was easier than I thought. I'm going with that solution - where facets are calculated against all columns, ignoring `?_col=` and `?_nocol=` entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851117233,https://api.github.com/repos/simonw/datasette/issues/1345,851117233,MDEyOklzc3VlQ29tbWVudDg1MTExNzIzMw==,9599,simonw,2021-05-31T02:04:35Z,2021-05-31T02:04:35Z,OWNER,"That long-term solution may not be too difficult. The facets are calculated against `sql_no_limit` which is constructed here:

https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/datasette/views/table.py#L659-L665

And used here:

https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/datasette/views/table.py#L706-L718

Crucially, `sql_no_limit` is ONLY used for faceting - nothing else uses it anywhere. So constructing it before constructing `sql` and taking `?_col=` and `?_nocol=` into account may not be a complex change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851116105,https://api.github.com/repos/simonw/datasette/issues/1345,851116105,MDEyOklzc3VlQ29tbWVudDg1MTExNjEwNQ==,9599,simonw,2021-05-31T02:00:44Z,2021-05-31T02:00:44Z,OWNER,"Maybe there's a short-term and longer-term solution for this - where the long-term solution is to use different columns in the faceting selects, while the short-term solution is to disable ""Hide this column"" for certain things.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851115805,https://api.github.com/repos/simonw/datasette/issues/1345,851115805,MDEyOklzc3VlQ29tbWVudDg1MTExNTgwNQ==,9599,simonw,2021-05-31T01:59:39Z,2021-05-31T01:59:39Z,OWNER,"Maybe there's a concept here of the columns that are required by a selected facet? Those can then be included as `data-` attributes on the page, which will then impact which ""Hide this column"" options are available.

I can also use them to provide a better error message than ""no such column: state"" - I can verify that `?_nocol` and `?_col` have not been used to disable the required columns.

There is one other option here: I could still include the columns that are known to be needed for faceting in the faceting SQL queries, but leave them out of the query that is used to return the results! That's actually a pretty tempting (albeit more complex) option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851114806,https://api.github.com/repos/simonw/datasette/issues/1345,851114806,MDEyOklzc3VlQ29tbWVudDg1MTExNDgwNg==,9599,simonw,2021-05-31T01:55:56Z,2021-05-31T01:55:56Z,OWNER,"Two options here:

- Don't provide users with options that will lead to this situation - so no ""Hide this column"" option on pages that are already faceted by that column
- Ignore facet selections for columns which are no longer visible

I think I like the first option more.

I could partially implement that in the `table.js` JavaScript by looking at the `?_facet=` parameters... but that won't cover the case where the facet is happening because of default facets configured in `metadata.yml`.

Instead the JavaScript should look for evidence in the DOM that specific facets are enabled. This could also help me cover other types of faceting, such as `?_facet_array=` or even custom facets provided by plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851113886,https://api.github.com/repos/simonw/datasette/issues/1345,851113886,MDEyOklzc3VlQ29tbWVudDg1MTExMzg4Ng==,9599,simonw,2021-05-31T01:52:27Z,2021-05-31T01:52:27Z,OWNER,"Related issue: visit https://latest.datasette.io/fixtures/facetable?_facet=state and click ""Hide this column"" on the ""state"" cog menu and you get https://latest.datasette.io/fixtures/facetable?_facet=state&_nocol=state which shows an error:

> ### Invalid SQL
> no such column: state
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1349#issuecomment-850778311,https://api.github.com/repos/simonw/datasette/issues/1349,850778311,MDEyOklzc3VlQ29tbWVudDg1MDc3ODMxMQ==,9599,simonw,2021-05-29T06:12:21Z,2021-05-29T06:12:21Z,OWNER,"It's not just facets, I think it's trying to execute suggested facets too!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-850778230,https://api.github.com/repos/simonw/datasette/issues/1349,850778230,MDEyOklzc3VlQ29tbWVudDg1MDc3ODIzMA==,9599,simonw,2021-05-29T06:11:28Z,2021-05-29T06:11:28Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/base.py#L263-L315,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-850778273,https://api.github.com/repos/simonw/datasette/issues/1349,850778273,MDEyOklzc3VlQ29tbWVudDg1MDc3ODI3Mw==,9599,simonw,2021-05-29T06:11:59Z,2021-05-29T06:11:59Z,OWNER,"Related issue: https://github.com/simonw/datasette/issues/263 - ""Facets should not execute for ?shape=array|object""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850772428,https://api.github.com/repos/simonw/sqlite-utils/issues/250,850772428,MDEyOklzc3VlQ29tbWVudDg1MDc3MjQyOA==,9599,simonw,2021-05-29T05:16:02Z,2021-05-29T05:28:57Z,OWNER,"I needed to find some CSV files on my computer with a BOM at the beginning - I figured out this recipe:
```
% rg -U -E none '^(?-u:\xEF\xBB\xBF)' --glob '*.csv' .
```
TIL here: https://til.simonwillison.net/bash/finding-bom-csv-files-with-ripgrep","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838148087,Handle byte order marks (BOMs) in CSV files,
https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850771264,https://api.github.com/repos/simonw/sqlite-utils/issues/250,850771264,MDEyOklzc3VlQ29tbWVudDg1MDc3MTI2NA==,9599,simonw,2021-05-29T05:06:13Z,2021-05-29T05:06:13Z,OWNER,The other option is to check if the file starts with `codecs.BOM_UTF8` - which is `b'\xef\xbb\xbf'`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838148087,Handle byte order marks (BOMs) in CSV files,
https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850771017,https://api.github.com/repos/simonw/sqlite-utils/issues/250,850771017,MDEyOklzc3VlQ29tbWVudDg1MDc3MTAxNw==,9599,simonw,2021-05-29T05:04:28Z,2021-05-29T05:04:28Z,OWNER,"https://stackoverflow.com/a/44573867/6083 says:

> There is no reason to check if a BOM exists or not, utf-8-sig manages that for you and behaves exactly as utf-8 if the BOM does not exist

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838148087,Handle byte order marks (BOMs) in CSV files,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850769067,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850769067,MDEyOklzc3VlQ29tbWVudDg1MDc2OTA2Nw==,9599,simonw,2021-05-29T04:48:10Z,2021-05-29T04:48:10Z,OWNER,"I confirmed and it's possible to have a SQLite column with a hyphen at the start, confirmed using:
```
% sqlite-utils create-table demo.db demo -- id integer name text -blah integer
% sqlite-utils tables --schema demo.db -t
table    schema
-------  ---------------------
demo     CREATE TABLE [demo] (
            [id] INTEGER,
            [name] TEXT,
            [-blah] INTEGER
         )
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850768315,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850768315,MDEyOklzc3VlQ29tbWVudDg1MDc2ODMxNQ==,9599,simonw,2021-05-29T04:39:33Z,2021-05-29T04:39:33Z,OWNER,"This doesn't work:
```
sqlite-utils create-index mydb.db dogs ""-age"" name
```
But this does:
```
sqlite-utils create-index mydb.db dogs -- -age name
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850767210,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850767210,MDEyOklzc3VlQ29tbWVudDg1MDc2NzIxMA==,9599,simonw,2021-05-29T04:26:26Z,2021-05-29T04:28:31Z,OWNER,It's weird having to use `Database.DescIndex` - I'm going to put `DescIndex` in `sqlite_utils.db` directly and let people import it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766552,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850766552,MDEyOklzc3VlQ29tbWVudDg1MDc2NjU1Mg==,9599,simonw,2021-05-29T04:20:40Z,2021-05-29T04:24:01Z,OWNER,"`PRAGMA index_xinfo(table)` DOES return that data:
```
(Pdb) [c[0] for c in fresh_db.execute(""PRAGMA index_xinfo('idx_dogs_age_name')"").description]
['seqno', 'cid', 'name', 'desc', 'coll', 'key']
(Pdb) fresh_db.execute(""PRAGMA index_xinfo('idx_dogs_age_name')"").fetchall()
[(0, 2, 'age', 1, 'BINARY', 1), (1, 0, 'name', 0, 'BINARY', 1), (2, -1, None, 0, 'BINARY', 0)]
```
See https://sqlite.org/pragma.html#pragma_index_xinfo

Example output: https://covid-19.datasettes.com/covid?sql=select+*+from+pragma_index_xinfo%28%27idx_ny_times_us_counties_date%27%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766335,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850766335,MDEyOklzc3VlQ29tbWVudDg1MDc2NjMzNQ==,9599,simonw,2021-05-29T04:18:19Z,2021-05-29T04:18:19Z,OWNER,"Annoyingly the `table.indexes` property won't indicate if an index is in regular or reverse order - because the SQLite `PRAGMA index_info(table)` statement doesn't indicate that either. You have to look at the `sqlite_master` index definition to tell if any of the columns are in reverse order:
```
(Pdb) fresh_db.execute(""select * from sqlite_master where type = 'index'"").fetchall()
[('index', 'idx_dogs_age_name', 'dogs', 3, 'CREATE INDEX [idx_dogs_age_name]\n    ON [dogs] ([age] desc, [name])')]
(Pdb) fresh_db.execute(""PRAGMA index_info('idx_dogs_age_name')"").fetchall()
[(0, 2, 'age'), (1, 0, 'name')]
(Pdb) fresh_db.execute(""PRAGMA index_info('idx_dogs_age_name')"").description
(('seqno', None, None, None, None, None, None), ('cid', None, None, None, None, None, None), ('name', None, None, None, None, None, None))
(Pdb) dogs.indexes
[Index(seq=0, name='idx_dogs_age_name', unique=0, origin='c', partial=0, columns=['age', 'name'])]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765450,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850765450,MDEyOklzc3VlQ29tbWVudDg1MDc2NTQ1MA==,9599,simonw,2021-05-29T04:09:13Z,2021-05-29T04:09:13Z,OWNER,"Decisions: for the Python API I'm going with `db.DescIndex(""column"")` as the way to do this.

For the CLI I'm going to do the ""-age"" thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765291,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850765291,MDEyOklzc3VlQ29tbWVudDg1MDc2NTI5MQ==,9599,simonw,2021-05-29T04:07:48Z,2021-05-29T04:08:21Z,OWNER,"For the CLI version I could say that you can use a `-` prefix to specify reverse direction:
```sh
sqlite-utils create-index mydb.db dogs -age name
```
No, that doesn't work - it could get confused with a command-line flag. I guess you could do this:
```
sqlite-utils create-index mydb.db dogs ""-age"" name
```
This does mean that if any of your column names begin with a hyphen you can't use the CLI to add indexes to them. Is that an acceptable limitation? Users can always use `sqlite-utils mydb.db ""create index ...""` in that case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765050,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850765050,MDEyOklzc3VlQ29tbWVudDg1MDc2NTA1MA==,9599,simonw,2021-05-29T04:05:24Z,2021-05-29T04:05:40Z,OWNER,"Need to solve this for the CLI tool too. Currently that works like this: https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes
```sh
sqlite-utils create-index mydb.db mytable col1 [col2...]
```
Even harder to decide how to add a descending option to this. Maybe like this?
```sh
sqlite-utils create-index mydb.db mytable --direction col1 asc --direction col2 desc
```
It's a bit gross though! We're saying here that if a single one of the columns you are creating an index for is in reverse direction you have to use `--direction` to specify each end every other index.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764700,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850764700,MDEyOklzc3VlQ29tbWVudDg1MDc2NDcwMA==,9599,simonw,2021-05-29T04:02:10Z,2021-05-29T04:02:10Z,OWNER,"I could use `db.desc_index(""age"")` to match SQLite SQL syntax, which uses `desc` and not `descending`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764655,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850764655,MDEyOklzc3VlQ29tbWVudDg1MDc2NDY1NQ==,9599,simonw,2021-05-29T04:01:41Z,2021-05-29T04:01:41Z,OWNER,"Maybe:
```python
db[""dogs""].create_index([db.descending_index(""age""), ""name""])
```
It's a little verbose but it's for a relatively rare activity and it does make it very clear what is going on.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764594,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850764594,MDEyOklzc3VlQ29tbWVudDg1MDc2NDU5NA==,9599,simonw,2021-05-29T04:00:54Z,2021-05-29T04:00:54Z,OWNER,"A few options:
```python
db[""dogs""].create_index([(""age"", ""desc""), ""name""])

db[""dogs""].create_index([desc(""age""), ""name""])

db[""dogs""].create_index([db.desc(""age""), ""name""])
```
The first option uses an optional tuple. The second two use a `desc()` function - the question is where should that live? 

`sqlite_utils.desc(column)` or `db.desc(column)`  are both options.

I don't like using the term `desc()` for ""descending index"" though - it feels like it should mean something more broad.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764253,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850764253,MDEyOklzc3VlQ29tbWVudDg1MDc2NDI1Mw==,9599,simonw,2021-05-29T03:57:54Z,2021-05-29T03:57:54Z,OWNER,"The problem here is differentiating between a column with the name `date desc` and wanting to create a descending index on a column called `date`.

This won't work:
```python
db[""ny_times_us_counties""].create_index([""date desc""], desc=True)
```
Because we need to be able to create compound indexes with columns with different directions - for example:
```sql
create index idx_age_desc_name on dogs (age desc, name)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/datasette/pull/1296#issuecomment-850583584,https://api.github.com/repos/simonw/datasette/issues/1296,850583584,MDEyOklzc3VlQ29tbWVudDg1MDU4MzU4NA==,9599,simonw,2021-05-28T18:06:11Z,2021-05-28T18:06:11Z,OWNER,"> As a bonus, the Docker image becomes smaller

That's a huge surprise to me! And most welcome.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",855446829,Dockerfile: use Ubuntu 20.10 as base,
https://github.com/simonw/datasette/issues/619#issuecomment-850025106,https://api.github.com/repos/simonw/datasette/issues/619,850025106,MDEyOklzc3VlQ29tbWVudDg1MDAyNTEwNg==,9599,simonw,2021-05-28T00:18:30Z,2021-05-28T00:18:30Z,OWNER,"I'm going to return a 400 HTTP status code for ""bad request"", under the assumption that the client sent bad SQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-850019486,https://api.github.com/repos/simonw/datasette/issues/619,850019486,MDEyOklzc3VlQ29tbWVudDg1MDAxOTQ4Ng==,9599,simonw,2021-05-28T00:05:50Z,2021-05-28T00:05:50Z,OWNER,Came up on discussions here: https://github.com/simonw/datasette/discussions/1334,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-850059607,https://api.github.com/repos/simonw/datasette/issues/619,850059607,MDEyOklzc3VlQ29tbWVudDg1MDA1OTYwNw==,9599,simonw,2021-05-28T02:08:58Z,2021-05-28T02:08:58Z,OWNER,Remaining work will happen in #1346,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/pull/1346#issuecomment-850059328,https://api.github.com/repos/simonw/datasette/issues/1346,850059328,MDEyOklzc3VlQ29tbWVudDg1MDA1OTMyOA==,9599,simonw,2021-05-28T02:08:10Z,2021-05-28T02:09:25Z,OWNER,"This is the current test failure, but it actually another problem that we don't have tests in place for errors with different formats and shapes:

```
    def test_magic_parameters_cannot_be_used_in_arbitrary_queries(magic_parameters_client):
        response = magic_parameters_client.get(
            ""/data.json?sql=select+:_header_host&_shape=array""
        )
        assert 400 == response.status
>       assert ""You did not supply a value for binding 1."" == response.json[""error""]
E       TypeError: list indices must be integers or slices, not str
```

The test fails because `response.json` here is the empty list `[]`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904537568,Re-display user's query with an error message if an error occurs,
https://github.com/simonw/datasette/pull/1346#issuecomment-850058851,https://api.github.com/repos/simonw/datasette/issues/1346,850058851,MDEyOklzc3VlQ29tbWVudDg1MDA1ODg1MQ==,9599,simonw,2021-05-28T02:06:46Z,2021-05-28T02:06:46Z,OWNER,"The one test failure here actually illustrates a larger problem: if the user specifies `.json?_shape=array` but an error occurs, what should we do?

Prior to this change we return the following JSON:
```json
{
    ""ok"": false,
    ""error"": ""You did not supply a value for binding 1."",
    ""status"": 500,
    ""title"": null
}
```
But this comes from the `handle_500` higher level code here: https://github.com/simonw/datasette/blob/eae3084b46e2c3931db12cdef79093ad0e644bce/datasette/app.py#L1251-L1263","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904537568,Re-display user's query with an error message if an error occurs,
https://github.com/simonw/datasette/issues/619#issuecomment-850057694,https://api.github.com/repos/simonw/datasette/issues/619,850057694,MDEyOklzc3VlQ29tbWVudDg1MDA1NzY5NA==,9599,simonw,2021-05-28T02:03:05Z,2021-05-28T02:03:05Z,OWNER,"I nearly got this working, but I ran into one last problem: the code path for when an error is raised but the user specified `?_shape=array`. I'll open a draft PR with where I've got to so far.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/1344#issuecomment-849820019,https://api.github.com/repos/simonw/datasette/issues/1344,849820019,MDEyOklzc3VlQ29tbWVudDg0OTgyMDAxOQ==,9599,simonw,2021-05-27T17:44:39Z,2021-05-27T17:52:58Z,OWNER,"This pattern appears to work, executed at the root of a checkout of `datasette`:
```
docker run -it -v `pwd`:/mnt --platform linux/amd64 \
  datasetteproject/datasette:0.57a1 bash -c '
    pip install ""pytest>=5.2.2,<6.3.0"" \
      ""pytest-xdist>=2.2.1,<2.3"" \
      ""pytest-asyncio>=0.10,<0.16"" \
      ""beautifulsoup4>=4.8.1,<4.10.0"" \
      ""black==21.5b1"" \
      ""pytest-timeout>=1.4.2,<1.5"" \
      ""trustme>=0.7,<0.8"" \
    && cd /mnt && pytest'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849815361,https://api.github.com/repos/simonw/datasette/issues/1344,849815361,MDEyOklzc3VlQ29tbWVudDg0OTgxNTM2MQ==,9599,simonw,2021-05-27T17:36:51Z,2021-05-27T17:36:51Z,OWNER,"To run the test suite... need to install a few more dependencies: https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/setup.py#L71-L79

And then mount the `tests/` folder into the container and run `pytest`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849806581,https://api.github.com/repos/simonw/datasette/issues/1344,849806581,MDEyOklzc3VlQ29tbWVudDg0OTgwNjU4MQ==,9599,simonw,2021-05-27T17:22:37Z,2021-05-27T17:34:27Z,OWNER,"```
~ % docker run -it --platform linux/arm/v7 -v `pwd`:/mnt -p 8001:8001 datasetteproject/datasette:0.57a1 python -c 'import platform; print(platform.uname())'
Unable to find image 'datasetteproject/datasette:0.57a1' locally
0.57a1: Pulling from datasetteproject/datasette
ee0a2cc24f29: Pull complete 
ca231153300d: Pull complete 
6073fc53d406: Pull complete 
bb2a6a6421dd: Pull complete 
14c15f441034: Pull complete 
28d113f72ba5: Pull complete 
40516c2c3785: Pull complete 
Digest: sha256:8dd469efa0b34b9e946c4e91496fc7bec702efec9c0f5ec548ccd4030f3263b2
Status: Downloaded newer image for datasetteproject/datasette:0.57a1
uname_result(system='Linux', node='c2e1a05732ed', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='armv7l')
```
Shorter version:
```
~ % docker run -it --platform linux/ppc64le datasetteproject/datasette:0.57a1 python -c 'import platform; print(platform.uname())'
Unable to find image 'datasetteproject/datasette:0.57a1' locally
0.57a1: Pulling from datasetteproject/datasette
c840eb5e9aed: Pull complete 
fc17d8b577ec: Pull complete 
712a0ad995f8: Pull complete 
56101149914c: Pull complete 
e6241dfe9d38: Pull complete 
41ed85039888: Pull complete 
26beeabe9213: Pull complete 
Digest: sha256:8dd469efa0b34b9e946c4e91496fc7bec702efec9c0f5ec548ccd4030f3263b2
Status: Downloaded newer image for datasetteproject/datasette:0.57a1
uname_result(system='Linux', node='0fe85887fa30', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='ppc64le')
~ % 
```

OK! Now I need to figure out how to run the full test suite on each one, then I can call this done.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849805778,https://api.github.com/repos/simonw/datasette/issues/1344,849805778,MDEyOklzc3VlQ29tbWVudDg0OTgwNTc3OA==,9599,simonw,2021-05-27T17:21:11Z,2021-05-27T17:21:22Z,OWNER,"This helps:
```
~ % docker run -it --platform linux/s390x -v `pwd`:/mnt \
  -p 8001:8001 datasetteproject/datasette:0.57a1 \
  python -c 'import platform; print(platform.uname())'
```
`uname_result(system='Linux', node='d14916ca91df', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='s390x')`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849804684,https://api.github.com/repos/simonw/datasette/issues/1344,849804684,MDEyOklzc3VlQ29tbWVudDg0OTgwNDY4NA==,9599,simonw,2021-05-27T17:19:16Z,2021-05-27T17:19:16Z,OWNER,"Having pushed that to Docker Hub the following _seems_ to work on my laptop:

    docker run --platform linux/s390x \
      -v `pwd`:/mnt -p 8001:8001 \
      datasetteproject/datasette:0.57a1 datasette -p 8001 -h 0.0.0.0

I'd like to confirm that this is correctly emulating the architecture and running the correct published image though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849801969,https://api.github.com/repos/simonw/datasette/issues/1344,849801969,MDEyOklzc3VlQ29tbWVudDg0OTgwMTk2OQ==,9599,simonw,2021-05-27T17:14:49Z,2021-05-27T17:14:49Z,OWNER,"Job completed in 13m38s - results are here: https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated&name=0.57a1

<img width=""1286"" alt=""datasetteproject_datasette_Tags"" src=""https://user-images.githubusercontent.com/9599/119868791-5c8caf00-bed4-11eb-9e9f-ebca557621c0.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1337#issuecomment-849725291,https://api.github.com/repos/simonw/datasette/issues/1337,849725291,MDEyOklzc3VlQ29tbWVudDg0OTcyNTI5MQ==,9599,simonw,2021-05-27T15:23:33Z,2021-05-27T15:23:33Z,OWNER,Turns out `path_with_replaced_args()` is already exposed to the `table.html` template context: https://github.com/simonw/datasette/blob/f1c29fd6a184254aa68efadf096bcf21e848f921/datasette/views/table.py#L897-L900,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1342#issuecomment-849724915,https://api.github.com/repos/simonw/datasette/issues/1342,849724915,MDEyOklzc3VlQ29tbWVudDg0OTcyNDkxNQ==,9599,simonw,2021-05-27T15:23:01Z,2021-05-27T15:23:01Z,OWNER,Since they are already exposed (but undocumented) this isn't actually a blocker on #1337 like I thought it was. Should still clean this up in the future though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903902495,Improve `path_with_replaced_args()` and friends and document them,
https://github.com/simonw/datasette/issues/1342#issuecomment-849724108,https://api.github.com/repos/simonw/datasette/issues/1342,849724108,MDEyOklzc3VlQ29tbWVudDg0OTcyNDEwOA==,9599,simonw,2021-05-27T15:21:54Z,2021-05-27T15:21:54Z,OWNER,"Turns out some of these are already being exposed to the template context on an ad-hoc basis.

In `QueryView`: https://github.com/simonw/datasette/blob/2bd9d54b2762c991e11950c22c88c0336158d49b/datasette/views/database.py#L295-L304

In `TableView`: https://github.com/simonw/datasette/blob/f1c29fd6a184254aa68efadf096bcf21e848f921/datasette/views/table.py#L897-L900","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903902495,Improve `path_with_replaced_args()` and friends and document them,
https://github.com/simonw/datasette/issues/1342#issuecomment-849722218,https://api.github.com/repos/simonw/datasette/issues/1342,849722218,MDEyOklzc3VlQ29tbWVudDg0OTcyMjIxOA==,9599,simonw,2021-05-27T15:19:23Z,2021-05-27T15:19:23Z,OWNER,Current implementations: https://github.com/simonw/datasette/blob/51d788114035458d8f860d9ea6d74078e6c0ea0d/datasette/utils/__init__.py#L214-L272,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903902495,Improve `path_with_replaced_args()` and friends and document them,
https://github.com/simonw/datasette/issues/1337#issuecomment-849721280,https://api.github.com/repos/simonw/datasette/issues/1337,849721280,MDEyOklzc3VlQ29tbWVudDg0OTcyMTI4MA==,9599,simonw,2021-05-27T15:18:08Z,2021-05-27T15:18:08Z,OWNER,In order to cleanly implement this I need to expose the `path_with_replaced_args` utility function to Datasette's template engine. This is the first time this will become an exposed (and hence should-by-documented) API and I don't like its shape much.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1344#issuecomment-849797767,https://api.github.com/repos/simonw/datasette/issues/1344,849797767,MDEyOklzc3VlQ29tbWVudDg0OTc5Nzc2Nw==,9599,simonw,2021-05-27T17:08:13Z,2021-05-27T17:08:13Z,OWNER,Related issue: #1272 - being able to run the test suite inside Docker inside GitHub Actions would be incredibly helpful here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849793312,https://api.github.com/repos/simonw/datasette/issues/1344,849793312,MDEyOklzc3VlQ29tbWVudDg0OTc5MzMxMg==,9599,simonw,2021-05-27T17:01:16Z,2021-05-27T17:01:16Z,OWNER,https://github.com/simonw/datasette/runs/2687196350 is a run of https://github.com/simonw/datasette/blob/main/.github/workflows/push_docker_tag.yml to publish the new `0.57a1` alpha to Docker Hub.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849788412,https://api.github.com/repos/simonw/datasette/issues/1344,849788412,MDEyOklzc3VlQ29tbWVudDg0OTc4ODQxMg==,9599,simonw,2021-05-27T16:53:28Z,2021-05-27T16:53:28Z,OWNER,(Should also update https://docs.datasette.io/en/stable/contributing.html#release-process with notes on how this works),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849788159,https://api.github.com/repos/simonw/datasette/issues/1344,849788159,MDEyOklzc3VlQ29tbWVudDg0OTc4ODE1OQ==,9599,simonw,2021-05-27T16:53:04Z,2021-05-27T16:53:04Z,OWNER,I'm going to release a `0.57a1` alpha to Docker Hub using the new buildx mechanism.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/pull/1319#issuecomment-849785098,https://api.github.com/repos/simonw/datasette/issues/1319,849785098,MDEyOklzc3VlQ29tbWVudDg0OTc4NTA5OA==,9599,simonw,2021-05-27T16:48:21Z,2021-05-27T16:48:21Z,OWNER,"OK, since I have a mechanism for deploying alpha versions now I'm going to merge this and use it to publish 0.57a1.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",881219362,Add Docker multi-arch support with Buildx,
https://github.com/simonw/datasette/issues/1343#issuecomment-849784005,https://api.github.com/repos/simonw/datasette/issues/1343,849784005,MDEyOklzc3VlQ29tbWVudDg0OTc4NDAwNQ==,9599,simonw,2021-05-27T16:46:37Z,2021-05-27T16:46:37Z,OWNER,"That worked:
```
 % docker run -it datasetteproject/datasette:0.57a0
Unable to find image 'datasetteproject/datasette:0.57a0' locally
0.57a0: Pulling from datasetteproject/datasette
75646c2fb410: Pull complete 
edf8b70d3cc4: Pull complete 
229daa3ebd94: Pull complete 
65d21ac50a56: Pull complete 
e083ad7cc3ca: Pull complete 
ed4de1431432: Pull complete 
1da5e8b27cca: Pull complete 
Digest: sha256:4ab61095c1d879a91d154ed469227387356a305035da9c74542c55bd500c89db
Status: Downloaded newer image for datasetteproject/datasette:0.57a0
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903978133,Figure out how to publish alpha/beta releases to Docker Hub,
https://github.com/simonw/datasette/issues/1343#issuecomment-849783398,https://api.github.com/repos/simonw/datasette/issues/1343,849783398,MDEyOklzc3VlQ29tbWVudDg0OTc4MzM5OA==,9599,simonw,2021-05-27T16:45:41Z,2021-05-27T16:45:41Z,OWNER,https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated now shows this at the top: https://hub.docker.com/layers/datasetteproject/datasette/0.57a0/images/sha256-4ab61095c1d879a91d154ed469227387356a305035da9c74542c55bd500c89db?context=explore,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903978133,Figure out how to publish alpha/beta releases to Docker Hub,
https://github.com/simonw/datasette/issues/1343#issuecomment-849782098,https://api.github.com/repos/simonw/datasette/issues/1343,849782098,MDEyOklzc3VlQ29tbWVudDg0OTc4MjA5OA==,9599,simonw,2021-05-27T16:43:40Z,2021-05-27T16:43:55Z,OWNER,"https://github.com/simonw/datasette/actions/workflows/push_docker_tag.yml

<img width=""944"" alt=""Actions_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/119864935-01f15400-bed0-11eb-8979-0a5cc1b6bcb4.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903978133,Figure out how to publish alpha/beta releases to Docker Hub,
https://github.com/simonw/datasette/issues/1343#issuecomment-849781747,https://api.github.com/repos/simonw/datasette/issues/1343,849781747,MDEyOklzc3VlQ29tbWVudDg0OTc4MTc0Nw==,9599,simonw,2021-05-27T16:43:04Z,2021-05-27T16:43:04Z,OWNER,I'm going to try pushing an alpha of https://github.com/simonw/datasette/releases/tag/0.57a0 using this workflow I built a few months ago: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/push_docker_tag.yml,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903978133,Figure out how to publish alpha/beta releases to Docker Hub,
https://github.com/simonw/datasette/pull/1319#issuecomment-849780481,https://api.github.com/repos/simonw/datasette/issues/1319,849780481,MDEyOklzc3VlQ29tbWVudDg0OTc4MDQ4MQ==,9599,simonw,2021-05-27T16:41:03Z,2021-05-27T16:41:03Z,OWNER,It looks like all I need to do to ship an alpha version to Docker Hub is NOT point the `latest` tag at it after it goes live: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/publish.yml#L75-L77,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",881219362,Add Docker multi-arch support with Buildx,
https://github.com/simonw/datasette/pull/1319#issuecomment-849778373,https://api.github.com/repos/simonw/datasette/issues/1319,849778373,MDEyOklzc3VlQ29tbWVudDg0OTc3ODM3Mw==,9599,simonw,2021-05-27T16:37:50Z,2021-05-27T16:37:50Z,OWNER,Started [a conversation](https://twitter.com/simonw/status/1397953338407223303) about this on Twitter. Ideally I'd like to push an alpha release to Docker Hub so people can start testing these out before they make it into an official non-alpha release.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",881219362,Add Docker multi-arch support with Buildx,
https://github.com/simonw/datasette/issues/1337#issuecomment-849762735,https://api.github.com/repos/simonw/datasette/issues/1337,849762735,MDEyOklzc3VlQ29tbWVudDg0OTc2MjczNQ==,9599,simonw,2021-05-27T16:14:14Z,2021-05-27T16:14:14Z,OWNER,"Here's a demo showing that once you get above 1,000 results you still get the ""..."" but it no longer links to anything: https://fivethirtyeight.datasettes.com/fivethirtyeight/births%2FUS_births_2000-2014_SSA?_facet=births&_facet_size=max","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1332#issuecomment-849761894,https://api.github.com/repos/simonw/datasette/issues/1332,849761894,MDEyOklzc3VlQ29tbWVudDg0OTc2MTg5NA==,9599,simonw,2021-05-27T16:13:07Z,2021-05-27T16:13:07Z,OWNER,Added `?_facet_size=max` in #1337 and made the `...` truncated note link to that - here's a demo: https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act?_facet=states&_facet=current_agency&_facet=pres_or_congress,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893890496,?_facet_size=X to increase number of facets results on the page,
https://github.com/simonw/datasette/issues/1337#issuecomment-849761508,https://api.github.com/repos/simonw/datasette/issues/1337,849761508,MDEyOklzc3VlQ29tbWVudDg0OTc2MTUwOA==,9599,simonw,2021-05-27T16:12:31Z,2021-05-27T16:12:31Z,OWNER,More exciting demo: https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act?_facet=states&_facet=current_agency&_facet=pres_or_congress,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1337#issuecomment-849759440,https://api.github.com/repos/simonw/datasette/issues/1337,849759440,MDEyOklzc3VlQ29tbWVudDg0OTc1OTQ0MA==,9599,simonw,2021-05-27T16:09:34Z,2021-05-27T16:09:34Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet_size=2&_facet=state&_facet=city_id

![demo-max](https://user-images.githubusercontent.com/9599/119860507-3f071780-becb-11eb-8698-64c8c6112983.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1337#issuecomment-849695117,https://api.github.com/repos/simonw/datasette/issues/1337,849695117,MDEyOklzc3VlQ29tbWVudDg0OTY5NTExNw==,9599,simonw,2021-05-27T14:44:13Z,2021-05-27T14:44:13Z,OWNER,"Here's a much easier way to implement this: implement `?_facet_size=max` (sets it to `max_returned_rows`, default 1000) and then link to that from the ellipsis.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/pull/1319#issuecomment-849311884,https://api.github.com/repos/simonw/datasette/issues/1319,849311884,MDEyOklzc3VlQ29tbWVudDg0OTMxMTg4NA==,9599,simonw,2021-05-27T04:33:51Z,2021-05-27T04:33:51Z,OWNER,This is so cool! I wonder how I can best test it? I have a Raspberry Pi I should dig out to try this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",881219362,Add Docker multi-arch support with Buildx,
https://github.com/simonw/datasette/issues/1341#issuecomment-849310294,https://api.github.com/repos/simonw/datasette/issues/1341,849310294,MDEyOklzc3VlQ29tbWVudDg0OTMxMDI5NA==,9599,simonw,2021-05-27T04:29:04Z,2021-05-27T04:29:04Z,OWNER,For implementation simplicity I'm going to show the item even if all of the columns are visible because `?_col=` has selected all of them - selecting it will still reset the view to having no `?_col=` or `?_nocol=` parameters.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903200328,"""Show all columns"" cog menu item should show if ?_col= is used",
https://github.com/simonw/datasette/issues/615#issuecomment-849309330,https://api.github.com/repos/simonw/datasette/issues/615,849309330,MDEyOklzc3VlQ29tbWVudDg0OTMwOTMzMA==,9599,simonw,2021-05-27T04:26:10Z,2021-05-27T04:26:10Z,OWNER,"Demos:

- https://latest.datasette.io/fixtures/sortable?_nocol=sortable
- https://latest.datasette.io/fixtures/sortable?_col=sortable&_col=text

Also try the column cog menu on that page.

Documentation: https://docs.datasette.io/en/latest/json_api.html#special-table-arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/292#issuecomment-849308907,https://api.github.com/repos/simonw/datasette/issues/292,849308907,MDEyOklzc3VlQ29tbWVudDg0OTMwODkwNw==,9599,simonw,2021-05-27T04:25:01Z,2021-05-27T04:25:01Z,OWNER,Now that `?_col=` and `?_nocol=` are implemented I'm closing this ticket - other customizations can already be handled by defining SQL views or creating canned SQL queries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",326800219,Mechanism for customizing the SQL used to select specific columns in the table view,
https://github.com/simonw/datasette/issues/670#issuecomment-849022714,https://api.github.com/repos/simonw/datasette/issues/670,849022714,MDEyOklzc3VlQ29tbWVudDg0OTAyMjcxNA==,9599,simonw,2021-05-26T18:33:47Z,2021-05-26T18:33:58Z,OWNER,Worth mentioning here: I've been doing a tun of research around running Datasette-like functionality against PostgreSQL in my https://github.com/simonw/django-sql-dashboard project - which will definitely inform the Datasette implementation.,"{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564833696,Prototoype for Datasette on PostgreSQL,
https://github.com/simonw/datasette/issues/1340#issuecomment-847987346,https://api.github.com/repos/simonw/datasette/issues/1340,847987346,MDEyOklzc3VlQ29tbWVudDg0Nzk4NzM0Ng==,9599,simonw,2021-05-25T15:49:55Z,2021-05-25T15:49:55Z,OWNER,"Usual problem with this kind of UI: what would it do on mobile where hover isn't a thing? Plenty of other questions: if it opens a menu, what options might I put in there other than ""Show more like this""?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",901009787,Research: Cell action menu (like column action but for individual cells),
https://github.com/simonw/datasette/issues/1320#issuecomment-847237524,https://api.github.com/repos/simonw/datasette/issues/1320,847237524,MDEyOklzc3VlQ29tbWVudDg0NzIzNzUyNA==,9599,simonw,2021-05-24T18:15:56Z,2021-05-24T18:15:56Z,OWNER,Added some new documentation about that here: https://github.com/simonw/datasette/blob/c0a748e5c3f498fa8c139b420d07dd3dea612379/docs/installation.rst#installing-plugins,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",884952179,Can't use apt-get in Dockerfile when using datasetteproj/datasette as base,
https://github.com/simonw/datasette/issues/1320#issuecomment-847232547,https://api.github.com/repos/simonw/datasette/issues/1320,847232547,MDEyOklzc3VlQ29tbWVudDg0NzIzMjU0Nw==,9599,simonw,2021-05-24T18:06:32Z,2021-05-24T18:06:32Z,OWNER,"I'm testing tweaks to the Dockerfile by editing it and running:
```
docker build . -t datasette-057a0 --build-arg VERSION=0.57a0
```
Then running this to inspect the new image:
```
docker run -it -p 8001:8001 datasette-057a0 bash
```
After changing that line to `rm -rf /var/lib/dpkg/info/*` I can run `apt-get` successfully from inside that bash prompt:
```
# apt-get update
# apt-get install ripgrep
# rg --version
ripgrep 12.1.1
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",884952179,Can't use apt-get in Dockerfile when using datasetteproj/datasette as base,
https://github.com/simonw/datasette/issues/1320#issuecomment-847230866,https://api.github.com/repos/simonw/datasette/issues/1320,847230866,MDEyOklzc3VlQ29tbWVudDg0NzIzMDg2Ng==,9599,simonw,2021-05-24T18:03:29Z,2021-05-24T18:03:29Z,OWNER,"```
root@33c269900821:/# du -h /var/lib/dpkg
6.3M	/var/lib/dpkg/info
24K	/var/lib/dpkg/triggers
4.0K	/var/lib/dpkg/updates
20K	/var/lib/dpkg/alternatives
4.0K	/var/lib/dpkg/parts
6.7M	/var/lib/dpkg
```
So `/var/lib/dpkg/info` is most of the space, I'll try just deleting that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",884952179,Can't use apt-get in Dockerfile when using datasetteproj/datasette as base,
https://github.com/simonw/datasette/issues/1320#issuecomment-847223853,https://api.github.com/repos/simonw/datasette/issues/1320,847223853,MDEyOklzc3VlQ29tbWVudDg0NzIyMzg1Mw==,9599,simonw,2021-05-24T17:51:12Z,2021-05-24T17:51:12Z,OWNER,"This is a good catch, thank you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",884952179,Can't use apt-get in Dockerfile when using datasetteproj/datasette as base,
https://github.com/simonw/datasette/issues/1327#issuecomment-847240470,https://api.github.com/repos/simonw/datasette/issues/1327,847240470,MDEyOklzc3VlQ29tbWVudDg0NzI0MDQ3MA==,9599,simonw,2021-05-24T18:21:44Z,2021-05-24T18:21:44Z,OWNER,"I'm having trouble replicating this - I tried the following `metadata.json` in my local environment and it displayed correctly:
```json
{
    ""title"": ""မြန်မာစာ (""
}
```
What operating system are you using here? Is there a chance that your `metadata.json` file is saved in a text encoding other than utf-8?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",892457208,Support Unicode characters in metadata.json,
https://github.com/simonw/datasette/issues/615#issuecomment-846627322,https://api.github.com/repos/simonw/datasette/issues/615,846627322,MDEyOklzc3VlQ29tbWVudDg0NjYyNzMyMg==,9599,simonw,2021-05-23T21:31:22Z,2021-05-23T21:31:22Z,OWNER,"Would it be useful to allow this mechanism to alias columns, for example supporting one of the following:

- `?_col=airline as name_of_airline`
- `?_col=airline:name_of_airline`

This could be handy for renaming columns to match a specific expected JSON output.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846627163,https://api.github.com/repos/simonw/datasette/issues/615,846627163,MDEyOklzc3VlQ29tbWVudDg0NjYyNzE2Mw==,9599,simonw,2021-05-23T21:30:01Z,2021-05-23T21:30:01Z,OWNER,"Interesting side-effect of this implementation is that you can both control column order and request the same column multiple times:

`/fivethirtyeight/airline-safety%2Fairline-safety?_col=fatal_accidents_00_14&_col=fatalities_00_14&_col=airline&_col=airline`

<img width=""930"" alt=""fivethirtyeight__airline-safety_airline-safety__56_rows"" src=""https://user-images.githubusercontent.com/9599/119277243-58f6e080-bbd3-11eb-98be-69787cd9192d.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846626871,https://api.github.com/repos/simonw/datasette/issues/615,846626871,MDEyOklzc3VlQ29tbWVudDg0NjYyNjg3MQ==,9599,simonw,2021-05-23T21:27:36Z,2021-05-23T21:27:36Z,OWNER,A better interface for this would be a full list of columns each with a checkbox for making it visible on invisible - this could then be used to apply a bulk change (rather than refreshing the interface after every removed column) and it could also be easily designed to work on narrow mobile screens where the cog icon is not visible.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846626567,https://api.github.com/repos/simonw/datasette/issues/615,846626567,MDEyOklzc3VlQ29tbWVudDg0NjYyNjU2Nw==,9599,simonw,2021-05-23T21:25:19Z,2021-05-23T21:25:19Z,OWNER,"Here's a bug: removing the `rowid` column returns an error.

![rowid-bug](https://user-images.githubusercontent.com/9599/119277132-b179ae00-bbd2-11eb-8f2f-e228c7d7c264.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846668419,https://api.github.com/repos/simonw/datasette/issues/615,846668419,MDEyOklzc3VlQ29tbWVudDg0NjY2ODQxOQ==,9599,simonw,2021-05-24T01:14:27Z,2021-05-24T01:14:27Z,OWNER,"I tried allowing the removal of `rowid` and got this exception:
```
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 831, in extra_template
    display_columns, display_rows = await self.display_columns_and_rows(
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 163, in display_columns_and_rows
    pk_path = path_from_row_pks(row, pks, not pks, False)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 82, in path_from_row_pks
    bits = [row[""rowid""]]
IndexError: No item with that key
```
I'm going to disable the removal of `rowid` - or indeed any of the primary keys, since they are needed to construct the row permalink.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846660103,https://api.github.com/repos/simonw/datasette/issues/615,846660103,MDEyOklzc3VlQ29tbWVudDg0NjY2MDEwMw==,9599,simonw,2021-05-24T00:47:00Z,2021-05-24T00:47:00Z,OWNER,"> Here's a bug: removing the `rowid` column returns an error.

Removing the `rowid` column should work. We can continue to show the `Link` column, ensuring users can still navigate to the row page for each row.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846659272,https://api.github.com/repos/simonw/datasette/issues/615,846659272,MDEyOklzc3VlQ29tbWVudDg0NjY1OTI3Mg==,9599,simonw,2021-05-24T00:44:47Z,2021-05-24T00:44:47Z,OWNER,I can't think of any good reasons to support passing the same column twice `?col=airline&_col=airline` so I'm going to de-duplicate and silently ignore the second one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846599732,https://api.github.com/repos/simonw/datasette/issues/615,846599732,MDEyOklzc3VlQ29tbWVudDg0NjU5OTczMg==,9599,simonw,2021-05-23T17:46:45Z,2021-05-23T17:46:45Z,OWNER,I've changed my mind about forbidding `?_col=` and `?_nocol=` from being applied at the same time - I'm going to come up with a simple resolution rule instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846595473,https://api.github.com/repos/simonw/datasette/issues/615,846595473,MDEyOklzc3VlQ29tbWVudDg0NjU5NTQ3Mw==,9599,simonw,2021-05-23T17:13:09Z,2021-05-23T17:13:09Z,OWNER,"Natalie suggests a quick way to implement ""undo"" would be to add a ""Show all columns"" item to that menu which only appears when at least one column is hidden.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846595091,https://api.github.com/repos/simonw/datasette/issues/615,846595091,MDEyOklzc3VlQ29tbWVudDg0NjU5NTA5MQ==,9599,simonw,2021-05-23T17:09:59Z,2021-05-23T17:09:59Z,OWNER,"Here's that prototype of a ""Hide this column"" cog menu button:

![hide](https://user-images.githubusercontent.com/9599/119269916-effd7180-bbae-11eb-92fa-0ef374bda4b8.gif)

Need a way to undo that once you've hidden a column - maybe a list of currently hidden columns that lets  you un-hide them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846592583,https://api.github.com/repos/simonw/datasette/issues/615,846592583,MDEyOklzc3VlQ29tbWVudDg0NjU5MjU4Mw==,9599,simonw,2021-05-23T16:50:45Z,2021-05-24T17:15:48Z,OWNER,"Still needed:

- Unit tests
- Documentation

And maybe a UI mechanism for this? I could at least add a ""remove this column"" item to the cog menu on columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846592392,https://api.github.com/repos/simonw/datasette/issues/615,846592392,MDEyOklzc3VlQ29tbWVudDg0NjU5MjM5Mg==,9599,simonw,2021-05-23T16:49:30Z,2021-05-23T16:49:44Z,OWNER,"I started looking at this again, inspired by #1326. I have a new diff that works against the latest `main` branch.

```diff
diff --git a/datasette/views/table.py b/datasette/views/table.py
index 4879228..f4b2ee2 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -64,6 +64,36 @@ class Row:
 
 
 class RowTableShared(DataView):
+    async def columns_to_select(self, db, table, request):
+        table_columns = await db.table_columns(table)
+        if ""_col"" in request.args and ""_nocol"" in request.args:
+            raise DatasetteError(""Cannot use _col and _nocol at the same time"")
+        if ""_col"" in request.args:
+            new_columns = []
+            for column in request.args.getlist(""_col""):
+                if column not in table_columns:
+                    raise DatasetteError(""_col={} is an invalid column"".format(column))
+                new_columns.append(column)
+            return new_columns
+        elif ""_nocol"" in request.args:
+            # Return all columns EXCEPT these
+            bad_columns = [
+                column
+                for column in request.args.getlist(""_nocol"")
+                if column not in table_columns
+            ]
+            if bad_columns:
+                raise DatasetteError(
+                    ""_nocol={} - invalid columns"".format("", "".join(bad_columns))
+                )
+            return [
+                column
+                for column in table_columns
+                if column not in request.args.getlist(""_nocol"")
+            ]
+        else:
+            return table_columns
+
     async def sortable_columns_for_table(self, database, table, use_rowid):
         db = self.ds.databases[database]
         table_metadata = self.ds.table_metadata(database, table)
@@ -321,18 +351,16 @@ class TableView(RowTableShared):
         )
 
         pks = await db.primary_keys(table)
-        table_column_details = await db.table_column_details(table)
-        table_columns = [column.name for column in table_column_details]
-
-        select_columns = "", "".join(escape_sqlite(t) for t in table_columns)
+        table_columns = await self.columns_to_select(db, table, request)
+        select_clause = "", "".join(escape_sqlite(t) for t in table_columns)
 
         use_rowid = not pks and not is_view
         if use_rowid:
-            select = f""rowid, {select_columns}""
+            select = f""rowid, {select_clause}""
             order_by = ""rowid""
             order_by_pks = ""rowid""
         else:
-            select = select_columns
+            select = select_clause
             order_by_pks = "", "".join([escape_sqlite(pk) for pk in pks])
             order_by = order_by_pks
 
@@ -715,6 +743,8 @@ class TableView(RowTableShared):
                 column = fk[""column""]
                 if column not in columns_to_expand:
                     continue
+                if column not in columns:
+                    continue
                 expanded_columns.append(column)
                 # Gather the values
                 column_index = columns.index(column)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/1326#issuecomment-846493370,https://api.github.com/repos/simonw/datasette/issues/1326,846493370,MDEyOklzc3VlQ29tbWVudDg0NjQ5MzM3MA==,9599,simonw,2021-05-23T02:55:00Z,2021-05-23T02:55:00Z,OWNER,"Currently you can do this with a custom SQL query, but as you've noticed this means you don't get automatic pagination: https://latest.datasette.io/fixtures.json?sql=select+pk%2C+created%2C+state+from+facetable+order+by+pk+limit+10&_shape=array

I have an open issue to allow selecting specific columns on the table page: #615

A workaround you can use today is to define a SQL view that selects just the tables you want. View pages behave exactly like tables in that they provide pagination by default.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",891969037,How to limit fields returned from the JSON API?,
https://github.com/simonw/datasette/issues/1331#issuecomment-846483426,https://api.github.com/repos/simonw/datasette/issues/1331,846483426,MDEyOklzc3VlQ29tbWVudDg0NjQ4MzQyNg==,9599,simonw,2021-05-23T00:57:30Z,2021-05-23T00:57:30Z,OWNER,Here's an alpha release with updated dependencies: https://pypi.org/project/datasette/0.57a0/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893537744,Add support for Jinja2 version 3.0,
https://github.com/simonw/datasette/issues/1331#issuecomment-846482057,https://api.github.com/repos/simonw/datasette/issues/1331,846482057,MDEyOklzc3VlQ29tbWVudDg0NjQ4MjA1Nw==,9599,simonw,2021-05-23T00:39:55Z,2021-05-23T00:39:55Z,OWNER,"> I'm stuck also because datasette wants itsdangerous~=1.1 instead of allowing itsdangerous-2.0.0

Bumped that dependency in b64d87204612a84663616e075f542499a5d82a03","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893537744,Add support for Jinja2 version 3.0,
https://github.com/simonw/datasette/issues/1331#issuecomment-846481937,https://api.github.com/repos/simonw/datasette/issues/1331,846481937,MDEyOklzc3VlQ29tbWVudDg0NjQ4MTkzNw==,9599,simonw,2021-05-23T00:38:28Z,2021-05-23T00:38:28Z,OWNER,I'm going to push an alpha release with the new dependencies shortly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893537744,Add support for Jinja2 version 3.0,
https://github.com/simonw/datasette/issues/1332#issuecomment-846481801,https://api.github.com/repos/simonw/datasette/issues/1332,846481801,MDEyOklzc3VlQ29tbWVudDg0NjQ4MTgwMQ==,9599,simonw,2021-05-23T00:36:20Z,2021-05-23T00:36:20Z,OWNER,"Documentation:

- https://docs.datasette.io/en/latest/facets.html#facets-in-query-strings at the bottom of the section
- https://docs.datasette.io/en/latest/json_api.html#special-table-arguments at the bottom of the list of arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893890496,?_facet_size=X to increase number of facets results on the page,
https://github.com/simonw/datasette/issues/1337#issuecomment-846481896,https://api.github.com/repos/simonw/datasette/issues/1337,846481896,MDEyOklzc3VlQ29tbWVudDg0NjQ4MTg5Ng==,9599,simonw,2021-05-23T00:37:52Z,2021-05-23T00:37:52Z,OWNER,"The tricky thing about this is that ideally I'd like this to link to a paginated set of results - but arbitrary SQL queries do not yet support pagination, see #856.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1332#issuecomment-846479730,https://api.github.com/repos/simonw/datasette/issues/1332,846479730,MDEyOklzc3VlQ29tbWVudDg0NjQ3OTczMA==,9599,simonw,2021-05-23T00:11:31Z,2021-05-23T00:11:31Z,OWNER,"Here's the code in question: https://github.com/simonw/datasette/blob/5e9672c9bb33e41686472db4aa427168f9e67dbe/datasette/facets.py#L187-L210

It has access to the current request object as `self.request`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893890496,?_facet_size=X to increase number of facets results on the page,
https://github.com/simonw/datasette/issues/1332#issuecomment-846479212,https://api.github.com/repos/simonw/datasette/issues/1332,846479212,MDEyOklzc3VlQ29tbWVudDg0NjQ3OTIxMg==,9599,simonw,2021-05-23T00:07:46Z,2021-05-23T00:07:46Z,OWNER,`?_facet_size=50` seems like a very reasonable thing to support - I'll cap it at whatever the `max_returned_rows` setting is (which defaults to 1000).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893890496,?_facet_size=X to increase number of facets results on the page,
https://github.com/simonw/datasette/issues/1332#issuecomment-846479062,https://api.github.com/repos/simonw/datasette/issues/1332,846479062,MDEyOklzc3VlQ29tbWVudDg0NjQ3OTA2Mg==,9599,simonw,2021-05-23T00:06:34Z,2021-05-23T00:06:34Z,OWNER,"A URL parameter to modify that facet size is a really good idea.

I thought I had an issue open for ""..."" linking to more results but I can't find it now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893890496,?_facet_size=X to increase number of facets results on the page,
https://github.com/simonw/sqlite-utils/issues/253#issuecomment-843718859,https://api.github.com/repos/simonw/sqlite-utils/issues/253,843718859,MDEyOklzc3VlQ29tbWVudDg0MzcxODg1OQ==,9599,simonw,2021-05-19T03:31:47Z,2021-05-19T03:31:47Z,OWNER,Fixed: https://simonwillison.net/2020/Sep/23/sqlite-advanced-alter-table/,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",847423559,fixtures.db example error in sql-utils blog post,
https://github.com/simonw/sqlite-utils/issues/253#issuecomment-843716042,https://api.github.com/repos/simonw/sqlite-utils/issues/253,843716042,MDEyOklzc3VlQ29tbWVudDg0MzcxNjA0Mg==,9599,simonw,2021-05-19T03:29:04Z,2021-05-19T03:29:04Z,OWNER,"Thanks for pointing this out - the problem here is that these days the `https://latest.datasette.io/fixtures.db` example database includes a test for generated columns, which are a brand new feature in SQLite. Your local SQLite is an older version.

I'm going to update the blog post to instead use a copy of `fixtures.db` that doesn't use that feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",847423559,fixtures.db example error in sql-utils blog post,
https://github.com/simonw/sqlite-utils/issues/259#issuecomment-843715502,https://api.github.com/repos/simonw/sqlite-utils/issues/259,843715502,MDEyOklzc3VlQ29tbWVudDg0MzcxNTUwMg==,9599,simonw,2021-05-19T03:27:33Z,2021-05-19T03:27:33Z,OWNER,"```
 % sqlite-utils insert test-big.db text big.json 
Error: table text has no column named colun

Try using --alter to add additional columns
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",894948100,Suggest the --alter option if a new column cannot be added,
https://github.com/simonw/sqlite-utils/issues/256#issuecomment-843715308,https://api.github.com/repos/simonw/sqlite-utils/issues/256,843715308,MDEyOklzc3VlQ29tbWVudDg0MzcxNTMwOA==,9599,simonw,2021-05-19T03:26:54Z,2021-05-19T03:26:54Z,OWNER,"I decided to treat this as a usability bug: there's a workaround, which is adding `--alter` - but it's not at all obvious that you should do that. I addressed this in #259.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",861622839,inserting with --nl errors with: sqlite3.OperationalError: table <table> has no column named <column>,
https://github.com/simonw/sqlite-utils/issues/256#issuecomment-843708246,https://api.github.com/repos/simonw/sqlite-utils/issues/256,843708246,MDEyOklzc3VlQ29tbWVudDg0MzcwODI0Ng==,9599,simonw,2021-05-19T03:06:02Z,2021-05-19T03:06:02Z,OWNER,Did you try running this with the `--alter` command-line option? That seems to work when I try it on a sample file here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",861622839,inserting with --nl errors with: sqlite3.OperationalError: table <table> has no column named <column>,
https://github.com/simonw/sqlite-utils/pull/244#issuecomment-843705806,https://api.github.com/repos/simonw/sqlite-utils/issues/244,843705806,MDEyOklzc3VlQ29tbWVudDg0MzcwNTgwNg==,9599,simonw,2021-05-19T02:58:18Z,2021-05-19T02:58:18Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",820468864,Typo in upsert example,
https://github.com/simonw/sqlite-utils/pull/245#issuecomment-843705721,https://api.github.com/repos/simonw/sqlite-utils/issues/245,843705721,MDEyOklzc3VlQ29tbWVudDg0MzcwNTcyMQ==,9599,simonw,2021-05-19T02:58:01Z,2021-05-19T02:58:01Z,OWNER,Thanks very much.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830803173,Correct some typos,
https://github.com/simonw/sqlite-utils/pull/254#issuecomment-843705533,https://api.github.com/repos/simonw/sqlite-utils/issues/254,843705533,MDEyOklzc3VlQ29tbWVudDg0MzcwNTUzMw==,9599,simonw,2021-05-19T02:57:22Z,2021-05-19T02:57:22Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",857280617,Fix incorrect create-table cli description,
https://github.com/simonw/sqlite-utils/pull/258#issuecomment-843702392,https://api.github.com/repos/simonw/sqlite-utils/issues/258,843702392,MDEyOklzc3VlQ29tbWVudDg0MzcwMjM5Mg==,9599,simonw,2021-05-19T02:47:37Z,2021-05-19T02:47:37Z,OWNER,I'm going to merge this and add a test - thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",868191959,Fixing insert from JSON containing strings with non-ascii characters …,
https://github.com/simonw/datasette/issues/856#issuecomment-843291675,https://api.github.com/repos/simonw/datasette/issues/856,843291675,MDEyOklzc3VlQ29tbWVudDg0MzI5MTY3NQ==,9599,simonw,2021-05-18T15:56:45Z,2021-05-18T15:56:45Z,OWNER,"Tables and views get ""stream all rows"" at the moment, so one workaround is to define a SQL view for your query - this only works for queries that don't take any parameters though (although you may be able to define a view and then pass it extra fields using the Datasette table interface, like on https://latest.datasette.io/fixtures/paginated_view?content_extra__contains=9)

I've explored this problem in a bit more detail in https://githu.com/simonw/django-sql-dashboard and I think I have a pattern that could work.

For your canned query, you could implement the pattern yourself by setting up two canned queries that look something like this:

https://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+sha%2C+author_date+from+commits+order+by+rowid+limit+1000

```sql
select rowid, sha, author_date from commits order by rowid limit 1000
```
That gets you the first set of 1,000 results. The important thing here is to order by a unique column, in this case `rowid` - because then subsequent pages can be loaded by a separate canned query that looks like this:
```sql
select rowid, sha, author_date from commits where rowid > :after order by rowid limit 1000
```
https://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+sha%2C+author_date+from+commits+where+rowid+%3E+%3Aafter+order+by+rowid+limit+1000&after=1000

You then need to write code which knows how to generate these queries - start with the first query with no `where` clause (or if you are using `rowid` you can just use the second query and pass it `?after=0` for the first call) - then keep calling the query passing in the last rowid you recieved as the `after` parameter.

Basically this is an implementation of keyset pagination with a smart client. When Datasette grows the ability to do this itself it will work by executing this mechanism inside the Python code, which is how the ""stream all rows"" option for tables works at the moment.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",642296989,Consider pagination of canned queries,
https://github.com/simonw/datasette/issues/1331#issuecomment-842497125,https://api.github.com/repos/simonw/datasette/issues/1331,842497125,MDEyOklzc3VlQ29tbWVudDg0MjQ5NzEyNQ==,9599,simonw,2021-05-17T17:20:15Z,2021-05-17T17:20:15Z,OWNER,I wonder if there are any new 3.0 features we should be taking advantage of here that would justify pinning to 3.0 minimum?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893537744,Add support for Jinja2 version 3.0,
https://github.com/simonw/datasette/issues/1331#issuecomment-842495346,https://api.github.com/repos/simonw/datasette/issues/1331,842495346,MDEyOklzc3VlQ29tbWVudDg0MjQ5NTM0Ng==,9599,simonw,2021-05-17T17:17:19Z,2021-05-17T17:17:19Z,OWNER,The tests are passing on #1324 so this may not require any code changes at all!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893537744,Add support for Jinja2 version 3.0,
https://github.com/simonw/datasette/pull/1324#issuecomment-842494844,https://api.github.com/repos/simonw/datasette/issues/1324,842494844,MDEyOklzc3VlQ29tbWVudDg0MjQ5NDg0NA==,9599,simonw,2021-05-17T17:16:36Z,2021-05-17T17:16:36Z,OWNER,Refs #1331.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",890073940,"Update jinja2 requirement from <2.12.0,>=2.10.3 to >=2.10.3,<3.1.0",
https://github.com/simonw/datasette/issues/1331#issuecomment-842494332,https://api.github.com/repos/simonw/datasette/issues/1331,842494332,MDEyOklzc3VlQ29tbWVudDg0MjQ5NDMzMg==,9599,simonw,2021-05-17T17:15:45Z,2021-05-17T17:16:04Z,OWNER,Datasette uses async templates which evidently have been improved in Jinja 2 3.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893537744,Add support for Jinja2 version 3.0,
https://github.com/simonw/datasette/issues/1131#issuecomment-739414118,https://api.github.com/repos/simonw/datasette/issues/1131,739414118,MDEyOklzc3VlQ29tbWVudDczOTQxNDExOA==,9599,simonw,2020-12-05T20:48:33Z,2020-12-05T20:48:33Z,OWNER,"Oddly enough, I tried fixing this with `sys.stderr.write(""{}\n"".format(e))` - but my Click `CLIRunner` tests failed because `result.stderr` was an empty string. Adding `sys.stderr.flush()` to the code that output errors fixed that issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",757481949,"""datasette inspect"" outputs invalid JSON if an error is logged",
https://github.com/simonw/datasette/issues/398#issuecomment-739357330,https://api.github.com/repos/simonw/datasette/issues/398,739357330,MDEyOklzc3VlQ29tbWVudDczOTM1NzMzMA==,9599,simonw,2020-12-05T19:36:27Z,2020-12-05T19:36:27Z,OWNER,This was fixed in #749 by 88ac538b41a4753c3de9b509c3a0e13077f66182,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",398011658,Ensure downloading a 100+MB SQLite database file works,
https://github.com/simonw/datasette/pull/1128#issuecomment-739355855,https://api.github.com/repos/simonw/datasette/issues/1128,739355855,MDEyOklzc3VlQ29tbWVudDczOTM1NTg1NQ==,9599,simonw,2020-12-05T19:34:57Z,2020-12-05T19:34:57Z,OWNER,Thanks for this!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756867924,Fix startup error on windows,
https://github.com/simonw/datasette/issues/1131#issuecomment-739083673,https://api.github.com/repos/simonw/datasette/issues/1131,739083673,MDEyOklzc3VlQ29tbWVudDczOTA4MzY3Mw==,9599,simonw,2020-12-05T00:02:10Z,2020-12-05T00:02:10Z,OWNER,"https://clig.dev/#the-basics

> **Send messaging to stderr**. Log messages, errors, and so on should all be sent to stderr. This means that when commands are piped together, these messages are displayed to the user and not fed into the next command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",757481949,"""datasette inspect"" outputs invalid JSON if an error is logged",
https://github.com/simonw/datasette/issues/1131#issuecomment-739083472,https://api.github.com/repos/simonw/datasette/issues/1131,739083472,MDEyOklzc3VlQ29tbWVudDczOTA4MzQ3Mg==,9599,simonw,2020-12-05T00:01:12Z,2020-12-05T00:01:12Z,OWNER,Here's why: https://github.com/simonw/datasette/blob/37f87b5e52e7f8ddd1c4ffcf368bd7a62a406a6d/datasette/database.py#L158-L163,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",757481949,"""datasette inspect"" outputs invalid JSON if an error is logged",
https://github.com/simonw/datasette/issues/188#issuecomment-738905376,https://api.github.com/repos/simonw/datasette/issues/188,738905376,MDEyOklzc3VlQ29tbWVudDczODkwNTM3Ng==,9599,simonw,2020-12-04T17:18:34Z,2020-12-04T17:18:34Z,OWNER,This is likely to be covered by plugin hooks: #860 for the metadata and after investigating in #1042 it looks like the existing `prepare_jinja2_environment` hook may already be enough to load templates from the database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",309047460,Ability to bundle metadata and templates inside the SQLite file,
https://github.com/simonw/datasette/issues/111#issuecomment-738904347,https://api.github.com/repos/simonw/datasette/issues/111,738904347,MDEyOklzc3VlQ29tbWVudDczODkwNDM0Nw==,9599,simonw,2020-12-04T17:16:56Z,2020-12-04T17:16:56Z,OWNER,This is STILL a good idea.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata,
https://github.com/simonw/datasette/pull/1130#issuecomment-738897582,https://api.github.com/repos/simonw/datasette/issues/1130,738897582,MDEyOklzc3VlQ29tbWVudDczODg5NzU4Mg==,9599,simonw,2020-12-04T17:03:30Z,2020-12-04T17:03:30Z,OWNER,"I deployed this to https://datasette-issue-1129.vercel.app/ (using `datasette publish vercel fixtures.db --branch 8d4c69c6fb0ef741a19070f5172017ea3522e83c --about_url https://github.com/simonw/datasette/issues/1129 --about datasette/issues/1129 --project datasette-issue-1129`) - weirdly, on Mobile Safari the footer appears just below the visible window:

![RPReplay_Final1607100726](https://user-images.githubusercontent.com/9599/101191950-336b2000-360f-11eb-8f14-ed83bd86515c.gif)

I've seen other problems with fixed footers on Mobile Safari too: at Eventbrite this was a really nasty problem for us to figure out: https://www.eventbrite.com/engineering/mobile-safari-why/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756876238,Fix footer not sticking to bottom in short pages,
https://github.com/simonw/datasette/issues/1125#issuecomment-738554392,https://api.github.com/repos/simonw/datasette/issues/1125,738554392,MDEyOklzc3VlQ29tbWVudDczODU1NDM5Mg==,9599,simonw,2020-12-04T04:16:57Z,2020-12-04T04:16:57Z,OWNER,"https://latest.datasette.io/-/versions now shows this:
```json
{
    ""python"": {
        ""version"": ""3.8.6"",
        ""full"": ""3.8.6 (default, Nov 18 2020, 13:49:49) \n[GCC 8.3.0]""
    },
    ""datasette"": {
        ""version"": ""0.52.3"",
        ""note"": ""49d8fc056844d5a537d6cfd96dab0dd5686fe718""
    },
    ""asgi"": ""3.0"",
    ""uvicorn"": ""0.12.3"",
    ""sqlite"": {
        ""version"": ""3.33.0"",
        ""fts_versions"": [
            ""FTS5"",
            ""FTS4"",
            ""FTS3""
        ],
        ""extensions"": {
            ""json1"": null
        },
        ""compile_options"": []
    },
    ""pysqlite3"": ""0.4.4""
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1125#issuecomment-738551280,https://api.github.com/repos/simonw/datasette/issues/1125,738551280,MDEyOklzc3VlQ29tbWVudDczODU1MTI4MA==,9599,simonw,2020-12-04T04:03:54Z,2020-12-04T04:03:54Z,OWNER,"I'm going to check `pkg_resources.get_distribution(""pysqlite3-binary"").version` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1125#issuecomment-738550588,https://api.github.com/repos/simonw/datasette/issues/1125,738550588,MDEyOklzc3VlQ29tbWVudDczODU1MDU4OA==,9599,simonw,2020-12-04T04:01:10Z,2020-12-04T04:01:10Z,OWNER,"Urgh, figuring out the version of `pysqlite3` is WAY harder than I expected. The `getversion` module looks like the smartest attempt at solving this problem generally, but I'd like to avoid adding another dependency just for this: https://github.com/smarie/python-getversion","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1125#issuecomment-738548693,https://api.github.com/repos/simonw/datasette/issues/1125,738548693,MDEyOklzc3VlQ29tbWVudDczODU0ODY5Mw==,9599,simonw,2020-12-04T03:52:51Z,2020-12-04T03:52:51Z,OWNER,"That didn't work - https://latest.datasette.io/-/versions isn't showing the package.

I bet that's because I'm actually installing `pysqlite3-binary` here: https://github.com/simonw/datasette/blob/e2fea36540e952d8d72c1bd0af7144b85b7a4671/.github/workflows/deploy-latest.yml#L57","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1126#issuecomment-738548393,https://api.github.com/repos/simonw/datasette/issues/1126,738548393,MDEyOklzc3VlQ29tbWVudDczODU0ODM5Mw==,9599,simonw,2020-12-04T03:51:38Z,2020-12-04T03:51:38Z,OWNER,That worked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756761963,Switch to google-github-actions/setup-gcloud for demo deploy,
https://github.com/simonw/datasette/issues/1124#issuecomment-738313399,https://api.github.com/repos/simonw/datasette/issues/1124,738313399,MDEyOklzc3VlQ29tbWVudDczODMxMzM5OQ==,9599,simonw,2020-12-03T21:10:54Z,2020-12-03T21:10:54Z,OWNER,Confirmed that installing a fresh copy of Datasette 0.52.3 on that server works correctly as expected.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1125#issuecomment-738347171,https://api.github.com/repos/simonw/datasette/issues/1125,738347171,MDEyOklzc3VlQ29tbWVudDczODM0NzE3MQ==,9599,simonw,2020-12-03T22:04:52Z,2020-12-03T22:04:52Z,OWNER,"```
pkg_resources.get_distribution(""pysqlite3"").version
Out[14]: '0.4.4'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1124#issuecomment-738224865,https://api.github.com/repos/simonw/datasette/issues/1124,738224865,MDEyOklzc3VlQ29tbWVudDczODIyNDg2NQ==,9599,simonw,2020-12-03T19:01:52Z,2020-12-03T19:01:52Z,OWNER,"https://github.com/simonw/datasette/runs/1494631261
```
/home/runner/work/datasette/datasette/tests/test_html.py:81: AssertionError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File ""/home/runner/work/datasette/datasette/datasette/app.py"", line 1039, in route_path
    response = await view(request, send)
  File ""/home/runner/work/datasette/datasette/datasette/utils/asgi.py"", line 297, in inner_static
    full_path.relative_to(root_path.resolve())
AttributeError: 'str' object has no attribute 'resolve'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738220067,https://api.github.com/repos/simonw/datasette/issues/1124,738220067,MDEyOklzc3VlQ29tbWVudDczODIyMDA2Nw==,9599,simonw,2020-12-03T18:58:17Z,2020-12-03T18:58:17Z,OWNER,"I tested this by running:

    pipx uninstall datasette
    pipx install 'https://github.com/simonw/datasette/archive/6b4c55efea3e9d34d92cbe5f0066553ad9b14071.zip'

To replace that version of Datasette (in the correct virtual environment) with this patch. It worked!

```
[ec2-user@ip-172-31-30-7 ~]$ datasette --get /-/static/app.css
/* Reset and Page Setup ==================================================== */
...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738215686,https://api.github.com/repos/simonw/datasette/issues/1124,738215686,MDEyOklzc3VlQ29tbWVudDczODIxNTY4Ng==,9599,simonw,2020-12-03T18:50:48Z,2020-12-03T21:42:02Z,OWNER,I'm going to punt on writing a unit test for this (not sure how I'd simulate those symlinks) - I'll manually test it and push out a dot release instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738215487,https://api.github.com/repos/simonw/datasette/issues/1124,738215487,MDEyOklzc3VlQ29tbWVudDczODIxNTQ4Nw==,9599,simonw,2020-12-03T18:50:26Z,2020-12-03T21:41:25Z,OWNER,"This fix works - calling `.resolve()` on the `root_path` before the comparison to ensure symlinks are resolved:
```python
        # Ensure full_path is within root_path to avoid weird ""../"" tricks
        try:
            print(""full_path={}, root_path={}"".format(full_path, root_path))
            full_path.relative_to(root_path.resolve())
        except ValueError as e:
            print(""  ValueError:"", e)
            await asgi_send_html(send, ""404"", 404)
            return
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738213342,https://api.github.com/repos/simonw/datasette/issues/1124,738213342,MDEyOklzc3VlQ29tbWVudDczODIxMzM0Mg==,9599,simonw,2020-12-03T18:46:22Z,2020-12-03T21:40:51Z,OWNER,"I replaced that function with this code:
```python
def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):
    async def inner_static(request, send):
        path = request.scope[""url_route""][""kwargs""][""path""]
        print(""path ="", path)
        try:
            full_path = (Path(root_path) / path).resolve().absolute()
        except FileNotFoundError as e:
            print(""FileNotFoundError:"", e)
            await asgi_send_html(send, ""404"", 404)
            return
        if full_path.is_dir():
            await asgi_send_html(send, ""403: Directory listing is not allowed"", 403)
            return
        # Ensure full_path is within root_path to avoid weird ""../"" tricks
        try:
            print(""full_path={}, root_path={}"".format(full_path, root_path))
            full_path.relative_to(root_path)
        except ValueError as e:
            print(""  ValueError:"", e)
            await asgi_send_html(send, ""404"", 404)
            return
        try:
            await asgi_send_file(send, full_path, chunk_size=chunk_size)
        except FileNotFoundError:
            await asgi_send_html(send, ""404"", 404)
            return

    return inner_static
```
Edited using `vi /home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/utils/asgi.py`

The output shows me what the bug is:
```
$ datasette --get /-/static/app.css --pdb
app_root =  /home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages
path = app.css
full_path=/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css, root_path=/home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages/datasette/static
  ValueError: '/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css' does not start with '/home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages/datasette/static'
404
```
`  ValueError: '/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css' does not start with '/home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages/datasette/static'`

One is `../lib/python3.7/..` and the other is `../lib64/python3.7/..` - there's clearly some kind of symlink in play here which I'm not taking into account.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738211776,https://api.github.com/repos/simonw/datasette/issues/1124,738211776,MDEyOklzc3VlQ29tbWVudDczODIxMTc3Ng==,9599,simonw,2020-12-03T18:43:21Z,2020-12-03T18:43:21Z,OWNER,I'm suspicious of this code here:https://github.com/simonw/datasette/blob/e048791a9a2686f47d81a2c8aa88aa1966d82521/datasette/utils/asgi.py#L284-L307,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738211152,https://api.github.com/repos/simonw/datasette/issues/1124,738211152,MDEyOklzc3VlQ29tbWVudDczODIxMTE1Mg==,9599,simonw,2020-12-03T18:42:12Z,2020-12-03T18:42:12Z,OWNER,"Added a line to print out `app_root` from https://github.com/simonw/datasette/blob/e048791a9a2686f47d81a2c8aa88aa1966d82521/datasette/app.py#L848-L853
```
app_root =  /home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738209642,https://api.github.com/repos/simonw/datasette/issues/1124,738209642,MDEyOklzc3VlQ29tbWVudDczODIwOTY0Mg==,9599,simonw,2020-12-03T18:39:19Z,2020-12-03T18:39:19Z,OWNER,"The CSS files are in the expected location:
```
[ec2-user@ip-172-31-30-7 ~]$ find /home/ec2-user/.local/pipx/venvs/datasette | grep css
/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css
/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/codemirror-5.57.0.min.css
```
Wow it's running an ANCIENT version of SQLite:
```
[ec2-user@ip-172-31-30-7 ~]$ datasette --get /-/versions.json
{""python"": {""version"": ""3.7.9"", ""full"": ""3.7.9 (default, Aug 27 2020, 21:58:41) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)]""}, ""datasette"": {""version"": ""0.52.2""}, ""asgi"": ""3.0"", ""uvicorn"": ""0.12.3"", ""sqlite"": {""version"": ""3.7.17"", ""fts_versions"": [""FTS4"", ""FTS3""], ""extensions"": {}, ""compile_options"": [""DISABLE_DIRSYNC"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_FTS3"", ""ENABLE_RTREE"", ""ENABLE_UNLOCK_NOTIFY"", ""SECURE_DELETE"", ""TEMP_STORE=1"", ""THREADSAFE=1""]}}
```
http://www.sqlite.org/releaselog/3_7_17.html - SQLite Release 3.7.17 On 2013-05-20","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1121#issuecomment-737591281,https://api.github.com/repos/simonw/datasette/issues/1121,737591281,MDEyOklzc3VlQ29tbWVudDczNzU5MTI4MQ==,9599,simonw,2020-12-03T01:03:18Z,2020-12-03T01:03:18Z,OWNER,"Demo: https://latest.datasette.io/fixtures?_bot=1

<img width=""729"" alt=""fixtures"" src=""https://user-images.githubusercontent.com/9599/100950004-43c0b500-34c0-11eb-918c-aa959376461f.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",754178780,Table actions cog is misaligned,
https://github.com/simonw/datasette/issues/1100#issuecomment-737589314,https://api.github.com/repos/simonw/datasette/issues/1100,737589314,MDEyOklzc3VlQ29tbWVudDczNzU4OTMxNA==,9599,simonw,2020-12-03T00:57:35Z,2020-12-03T00:57:35Z,OWNER,"Fixed in the demo:
```
% curl -XOPTIONS https://latest.datasette.io/fixtures.json
ok%
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",747702144,Error on OPTIONS request to database,
https://github.com/simonw/datasette/issues/1123#issuecomment-737586248,https://api.github.com/repos/simonw/datasette/issues/1123,737586248,MDEyOklzc3VlQ29tbWVudDczNzU4NjI0OA==,9599,simonw,2020-12-03T00:47:37Z,2020-12-03T00:47:37Z,OWNER,"Affected tests:
```
FAILED tests/test_plugins.py::test_hook_table_actions[facetable] - AssertionE...
FAILED tests/test_plugins.py::test_hook_table_actions[simple_view] - Assertio...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",755721275,"Table actions hook are order dependent, should not be",
https://github.com/simonw/datasette/issues/1100#issuecomment-737581719,https://api.github.com/repos/simonw/datasette/issues/1100,737581719,MDEyOklzc3VlQ29tbWVudDczNzU4MTcxOQ==,9599,simonw,2020-12-03T00:35:23Z,2020-12-03T00:35:23Z,OWNER,"Replicated this against the live demo as well:
```
/tmp % curl -XOPTIONS https://latest.datasette.io/fixtures.json
{""ok"": false, ""error"": ""object Response can't be used in 'await' expression"", ""status"": 500, ""title"": null}%                                                                                                                                  /tmp % 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",747702144,Error on OPTIONS request to database,
https://github.com/simonw/datasette/pull/1122#issuecomment-737580813,https://api.github.com/repos/simonw/datasette/issues/1122,737580813,MDEyOklzc3VlQ29tbWVudDczNzU4MDgxMw==,9599,simonw,2020-12-03T00:33:09Z,2020-12-03T00:33:09Z,OWNER,"This is a very neat fix, thank you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",754179035,Fix misaligned table actions cog,
https://github.com/simonw/datasette/issues/749#issuecomment-737580084,https://api.github.com/repos/simonw/datasette/issues/749,737580084,MDEyOklzc3VlQ29tbWVudDczNzU4MDA4NA==,9599,simonw,2020-12-03T00:31:14Z,2020-12-03T00:31:14Z,OWNER,"This works!

```
/tmp % wget 'https://covid-19.datasettes.com/covid.db'
--2020-12-02 16:28:02--  https://covid-19.datasettes.com/covid.db
Resolving covid-19.datasettes.com (covid-19.datasettes.com)... 172.217.5.83
Connecting to covid-19.datasettes.com (covid-19.datasettes.com)|172.217.5.83|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘covid.db’

covid.db                [              <=>   ] 306.42M  3.27MB/s    in 98s     

2020-12-02 16:29:40 (3.13 MB/s) - ‘covid.db’ saved [321306624]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610829227,Cloud Run fails to serve database files larger than 32MB,
https://github.com/simonw/datasette/issues/749#issuecomment-737563699,https://api.github.com/repos/simonw/datasette/issues/749,737563699,MDEyOklzc3VlQ29tbWVudDczNzU2MzY5OQ==,9599,simonw,2020-12-02T23:45:42Z,2020-12-02T23:45:42Z,OWNER,"I asked about this on Twitter - https://twitter.com/steren/status/1334281184965140483

> You simply need to send the `Transfer-Encoding: chunked` header.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610829227,Cloud Run fails to serve database files larger than 32MB,
https://github.com/simonw/datasette/issues/942#issuecomment-737402392,https://api.github.com/repos/simonw/datasette/issues/942,737402392,MDEyOklzc3VlQ29tbWVudDczNzQwMjM5Mg==,9599,simonw,2020-12-02T18:08:55Z,2020-12-02T18:08:55Z,OWNER,"SQLite does let you add comments in your CREATE TABLE statements:
```sql
CREATE TABLE something (
    id integer primary key, -- integer primary key
    created text -- created date as ISO datetime
);
```
But the only mechanism for reading those back is to retrieve that `CREATE TABLE` block of SQL from the `sqlite_master` table and run a parser against it.

I've so far resisted adding a SQL syntax parser to Datasette for complexity reasons - though I'm increasingly thinking I'll need to do it at some point.

I think I'll leave this to plugins. I'm definitely going to build a plugin that lets you store metadata for tables and columns in a SQLite database table, which will then support interactively editing metadata through a UI.

A plugin which extracts column comments from the SQLite CREATE TABLE comments would be feasible too, if I design the plugin hooks well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-737463116,https://api.github.com/repos/simonw/datasette/issues/942,737463116,MDEyOklzc3VlQ29tbWVudDczNzQ2MzExNg==,9599,simonw,2020-12-02T20:02:10Z,2020-12-02T20:03:01Z,OWNER,"My idea is that if you installed my proposed plugin you wouldn't need `metadata.json` at all - your metadata would instead live in a table in the connected SQLite database files - either one table per database (so the metadata can live in the same place as the data) or maybe also in a dedicated separate database file, for if you want to add metadata to an otherwise read-only database.

The plugin would then provide a UI for editing that metadata - maybe by configuring some writable canned queries or maybe something more custom than that. Or you could edit the metadata by manually editing the SQLite database file (or loading data into it using a tool like [yaml-to-sqlite](https://github.com/simonw/yaml-to-sqlite)).","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/1119#issuecomment-736142201,https://api.github.com/repos/simonw/datasette/issues/1119,736142201,MDEyOklzc3VlQ29tbWVudDczNjE0MjIwMQ==,9599,simonw,2020-12-01T00:41:14Z,2020-12-01T00:41:14Z,OWNER,"On my laptop:

<img width=""1123"" alt=""fixtures__generated_columns__1_row"" src=""https://user-images.githubusercontent.com/9599/100681839-8734eb00-3329-11eb-8aea-813bdc18efa1.png"">

https://latest.datasette.io/-/versions is running SQLite 3.27.2 at the moment so it won't show that table until it gets to 3.31.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753876808,"Include generated columns in fixtures.db, if SQLite version supports it",
https://github.com/simonw/datasette/issues/1116#issuecomment-736030599,https://api.github.com/repos/simonw/datasette/issues/1116,736030599,MDEyOklzc3VlQ29tbWVudDczNjAzMDU5OQ==,9599,simonw,2020-11-30T20:41:41Z,2020-11-30T20:41:41Z,OWNER,"Here's the problem: https://www.sqlite.org/changes.html#version_3_26_0

> ### 2018-12-01 (3.26.0)
> 
> -  Added [PRAGMA table_xinfo](https://www.sqlite.org/pragma.html#pragma_table_xinfo) that works just like [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that it also shows [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables.

CI is running 3.22.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/pull/1117#issuecomment-736029337,https://api.github.com/repos/simonw/datasette/issues/1117,736029337,MDEyOklzc3VlQ29tbWVudDczNjAyOTMzNw==,9599,simonw,2020-11-30T20:39:06Z,2020-11-30T20:39:06Z,OWNER,"Here's the problem: https://www.sqlite.org/changes.html#version_3_26_0

> ### 2018-12-01 (3.26.0)
> 
> -  Added [PRAGMA table_xinfo](https://www.sqlite.org/pragma.html#pragma_table_xinfo) that works just like [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that it also shows [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables.

CI is running 3.22.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/pull/1117#issuecomment-736028726,https://api.github.com/repos/simonw/datasette/issues/1117,736028726,MDEyOklzc3VlQ29tbWVudDczNjAyODcyNg==,9599,simonw,2020-11-30T20:37:50Z,2020-11-30T20:37:50Z,OWNER,"This kind of problem is why I have a `tmate` workflow:

<img width=""1086"" alt=""Actions_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/100660845-72dff680-3307-11eb-89dc-211d226f68dd.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/pull/1117#issuecomment-736023089,https://api.github.com/repos/simonw/datasette/issues/1117,736023089,MDEyOklzc3VlQ29tbWVudDczNjAyMzA4OQ==,9599,simonw,2020-11-30T20:26:27Z,2020-11-30T20:26:27Z,OWNER,"On my laptop:
```
platform darwin -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
SQLite: 3.33.0
```
In CI they are all SQLite: 3.22.0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/pull/1117#issuecomment-736018609,https://api.github.com/repos/simonw/datasette/issues/1117,736018609,MDEyOklzc3VlQ29tbWVudDczNjAxODYwOQ==,9599,simonw,2020-11-30T20:17:31Z,2020-11-30T20:17:31Z,OWNER,I need to replicate these failures on my laptop. My hunch is that this is down to the version of SQLite available to Python.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/issues/1116#issuecomment-736015487,https://api.github.com/repos/simonw/datasette/issues/1116,736015487,MDEyOklzc3VlQ29tbWVudDczNjAxNTQ4Nw==,9599,simonw,2020-11-30T20:11:07Z,2020-11-30T20:11:07Z,OWNER,Working on this in a pull request: https://github.com/simonw/datasette/pull/1117,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-736014372,https://api.github.com/repos/simonw/datasette/issues/1116,736014372,MDEyOklzc3VlQ29tbWVudDczNjAxNDM3Mg==,9599,simonw,2020-11-30T20:08:48Z,2020-11-30T20:08:48Z,OWNER,"Ouch, the tests pass on my laptop but failed in CI: https://github.com/simonw/datasette/actions/runs/392367997

Lots of failures look like this:
```
ERROR: conn=<sqlite3.Connection object at 0x7f44f0494030>, sql = 'select rowid,  from facetable order by rowid limit 51', params = {}: near ""from"": syntax error
```
Note the `select rowid, from...` - so it looks like invalid SQL queries are being constructed maybe due to mis-detecting columns somehow.

I wonder why it didn't fail on my laptop?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-736010720,https://api.github.com/repos/simonw/datasette/issues/1116,736010720,MDEyOklzc3VlQ29tbWVudDczNjAxMDcyMA==,9599,simonw,2020-11-30T20:01:53Z,2020-11-30T20:01:53Z,OWNER,"I'm OK exposing hidden columns, unless someone comes up with a pressing reason not to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-736004383,https://api.github.com/repos/simonw/datasette/issues/1116,736004383,MDEyOklzc3VlQ29tbWVudDczNjAwNDM4Mw==,9599,simonw,2020-11-30T19:51:51Z,2020-11-30T19:51:51Z,OWNER,"This change will also have an impact on how hidden virtual FTS tables are displayed, since apparently those have some hidden columns: https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_xinfo%28%27searchable_fts%27%29

| cid | name | type | notnull | dflt_value | pk | hidden |
| --- | --- | --- | --- | --- | --- | --- |
| 0 | text1 |   | 0 |   | 0 | 0 |
| 1 | text2 |   | 0 |   | 0 | 0 |
| 2 | name with . and spaces |   | 0 |   | 0 | 0 |
| 3 | searchable_fts |   | 0 |   | 0 | 1 |
| 4 | docid |   | 0 |   | 0 | 1 |
| 5 | __langid |   | 0 |   | 0 | 1 |","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-735995695,https://api.github.com/repos/simonw/datasette/issues/1116,735995695,MDEyOklzc3VlQ29tbWVudDczNTk5NTY5NQ==,9599,simonw,2020-11-30T19:34:15Z,2020-11-30T19:34:15Z,OWNER,"Generated column support was added in SQLite 3.31.0, so any unit tests I write for this should use skipIf to only run on that version or later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-735993935,https://api.github.com/repos/simonw/datasette/issues/1116,735993935,MDEyOklzc3VlQ29tbWVudDczNTk5MzkzNQ==,9599,simonw,2020-11-30T19:30:44Z,2020-11-30T19:32:15Z,OWNER,"It looks like `PRAGMA table_info` skips ""hidden"" columns: https://www.sqlite.org/pragma.html#pragma_table_info

But `PRAGMA table_xinfo` does not: https://www.sqlite.org/pragma.html#pragma_table_xinfo

Compare https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_info%28%27searchable%27%29 to https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_xinfo%28%27searchable%27%29 - the `xinfo` one has an additional `hidden` column.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-735992106,https://api.github.com/repos/simonw/datasette/issues/1116,735992106,MDEyOklzc3VlQ29tbWVudDczNTk5MjEwNg==,9599,simonw,2020-11-30T19:27:10Z,2020-11-30T19:27:10Z,OWNER,"I'm treating this as a bug - these columns should definitely be visible in Datasette.

I created my own test database using SQLite from Homebrew like this:
```
/usr/local/Cellar/sqlite/3.33.0/bin/sqlite3 deeds.db << EOF
CREATE TABLE deeds (
    body TEXT,
    id INT GENERATED ALWAYS AS (json_extract(body, '$.id')) STORED,
    consideration INT GENERATED ALWAYS AS (json_extract(body, '$.consideration')) STORED
);
INSERT INTO deeds (body) VALUES ('{
    ""id"": 1,
    ""consideration"": ""This is the consideration""
}');
EOF
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/263#issuecomment-735960132,https://api.github.com/repos/simonw/datasette/issues/263,735960132,MDEyOklzc3VlQ29tbWVudDczNTk2MDEzMg==,9599,simonw,2020-11-30T18:25:17Z,2020-11-30T18:25:17Z,OWNER,Fixing this would unblock this issue for switching `datasette-graphql` to using `datasette.client` internally: https://github.com/simonw/datasette-graphql/issues/61,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323671577,Facets should not execute for ?shape=array|object,
https://github.com/simonw/datasette/issues/1114#issuecomment-735429041,https://api.github.com/repos/simonw/datasette/issues/1114,735429041,MDEyOklzc3VlQ29tbWVudDczNTQyOTA0MQ==,9599,simonw,2020-11-29T17:36:18Z,2020-11-29T17:36:18Z,OWNER,"This is a great catch, thanks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752966476,--load-extension=spatialite not working with datasetteproject/datasette docker image,
https://github.com/simonw/datasette/issues/1114#issuecomment-735447635,https://api.github.com/repos/simonw/datasette/issues/1114,735447635,MDEyOklzc3VlQ29tbWVudDczNTQ0NzYzNQ==,9599,simonw,2020-11-29T20:16:32Z,2020-11-29T20:17:11Z,OWNER,"The new Docker container is pushed to Docker Hub too. Here's it in action:

```
% docker pull datasetteproject/datasette
% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette --version
datasette, version 0.52.1
% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/nps-spatialite.db --load-extension=spatialite
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752966476,--load-extension=spatialite not working with datasetteproject/datasette docker image,
https://github.com/simonw/datasette/issues/1115#issuecomment-735447198,https://api.github.com/repos/simonw/datasette/issues/1115,735447198,MDEyOklzc3VlQ29tbWVudDczNTQ0NzE5OA==,9599,simonw,2020-11-29T20:12:49Z,2020-11-29T20:12:49Z,OWNER,"```
$ datasette ../shapefile-to-sqlite/nps-spatialite.db
Usage: datasette serve [OPTIONS] [FILES]...

Error: It looks like you're trying to load a SpatiaLite database without first loading the SpatiaLite module.

Try adding the --load-extension=spatialite option.

Read more: https://docs.datasette.io/en/stable/spatialite.html
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752995227,SpatiaLite error could suggest --load-extension=spatialite,
https://github.com/simonw/datasette/issues/1099#issuecomment-735444858,https://api.github.com/repos/simonw/datasette/issues/1099,735444858,MDEyOklzc3VlQ29tbWVudDczNTQ0NDg1OA==,9599,simonw,2020-11-29T19:51:58Z,2020-11-29T19:51:58Z,OWNER,"My fix in deb0be4ae56f191f121239b29e83dd53b62d6305 for #1098 was to have Datasette deliberately pretend that compound foreign keys don't exist: https://github.com/simonw/datasette/blob/deb0be4ae56f191f121239b29e83dd53b62d6305/datasette/utils/__init__.py#L470-L495

This workaround will need to be rethought to implement real support for them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/1098#issuecomment-735444698,https://api.github.com/repos/simonw/datasette/issues/1098,735444698,MDEyOklzc3VlQ29tbWVudDczNTQ0NDY5OA==,9599,simonw,2020-11-29T19:50:14Z,2020-11-29T19:50:14Z,OWNER,Demo of the fix: https://latest.datasette.io/fixtures/foreign_key_references (the compound foreign key columns do not link to anything),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1114#issuecomment-735443626,https://api.github.com/repos/simonw/datasette/issues/1114,735443626,MDEyOklzc3VlQ29tbWVudDczNTQ0MzYyNg==,9599,simonw,2020-11-29T19:40:49Z,2020-11-29T19:40:49Z,OWNER,Fix is out in 0.52.1: https://docs.datasette.io/en/latest/changelog.html#v0-52-1,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752966476,--load-extension=spatialite not working with datasetteproject/datasette docker image,
https://github.com/simonw/datasette/issues/1098#issuecomment-735443654,https://api.github.com/repos/simonw/datasette/issues/1098,735443654,MDEyOklzc3VlQ29tbWVudDczNTQ0MzY1NA==,9599,simonw,2020-11-29T19:41:01Z,2020-11-29T19:41:01Z,OWNER,Fix is out in 0.52.1: https://docs.datasette.io/en/latest/changelog.html#v0-52-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1098#issuecomment-735442226,https://api.github.com/repos/simonw/datasette/issues/1098,735442226,MDEyOklzc3VlQ29tbWVudDczNTQ0MjIyNg==,9599,simonw,2020-11-29T19:28:04Z,2020-11-29T19:28:04Z,OWNER,"For the moment I'm going to solve this by teaching Datasette's internal introspection methods - in particular these ones - to ignore compound foreign keys entirely:

https://github.com/simonw/datasette/blob/e800ffcf7cc6a915eb554b369c654f87162575e5/datasette/utils/__init__.py#L470-L505","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1111#issuecomment-735320736,https://api.github.com/repos/simonw/datasette/issues/1111,735320736,MDEyOklzc3VlQ29tbWVudDczNTMyMDczNg==,9599,simonw,2020-11-29T02:46:23Z,2020-11-29T02:46:23Z,OWNER,"This is a really useful bug report, thanks! I agree: more aggressive timeouts on table counts sounds like the right solution here.

I've learned that avoiding `count(*)` is crucial for handling these larger databases. Datasette has been getting better about that over time but there are still some edge-cases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",751195017,Accessing a database's `.json` is slow for very large SQLite files,
https://github.com/simonw/datasette/issues/1108#issuecomment-735320499,https://api.github.com/repos/simonw/datasette/issues/1108,735320499,MDEyOklzc3VlQ29tbWVudDczNTMyMDQ5OQ==,9599,simonw,2020-11-29T02:42:42Z,2020-11-29T02:42:42Z,OWNER,https://docs.datasette.io/en/stable/config.html now redirects correctly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750087350,Configure /en/stable/config.html redirect when I ship 0.52,
https://github.com/simonw/datasette/issues/1113#issuecomment-735304071,https://api.github.com/repos/simonw/datasette/issues/1113,735304071,MDEyOklzc3VlQ29tbWVudDczNTMwNDA3MQ==,9599,simonw,2020-11-28T23:23:31Z,2020-11-28T23:23:31Z,OWNER,https://github.com/simonw/datasette/blob/bbde835a1fec01458e8d00929e7bab6d6a5ba948/datasette/views/table.py#L1013-L1026 - looks like I fixed this in #1088.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752789159,500 error on row page if query against foreign keys hits time limit,
https://github.com/simonw/datasette/issues/1102#issuecomment-735356882,https://api.github.com/repos/simonw/datasette/issues/1102,735356882,MDEyOklzc3VlQ29tbWVudDczNTM1Njg4Mg==,9599,simonw,2020-11-29T07:47:22Z,2020-11-29T07:47:22Z,OWNER,Live: https://docs.datasette.io/en/latest/testing_plugins.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749289611,Plugin testing docs should show datasette.client,
https://github.com/simonw/datasette/pull/1112#issuecomment-735283033,https://api.github.com/repos/simonw/datasette/issues/1112,735283033,MDEyOklzc3VlQ29tbWVudDczNTI4MzAzMw==,9599,simonw,2020-11-28T19:53:36Z,2020-11-28T19:53:36Z,OWNER,Thanks!,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752749485,Fix --metadata doc usage,
https://github.com/simonw/datasette/issues/1110#issuecomment-733432916,https://api.github.com/repos/simonw/datasette/issues/1110,733432916,MDEyOklzc3VlQ29tbWVudDczMzQzMjkxNg==,9599,simonw,2020-11-25T03:04:29Z,2020-11-25T03:04:29Z,OWNER,Already have a pattern for extra packages in the form of the `--spatialite` feature: https://github.com/simonw/datasette/blob/f2e2bfcdd9ad4891f3f66c9104c09943d943ffe4/datasette/utils/__init__.py#L50-L55,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750330029,datasette publish option for installing extra apt-get packages,
https://github.com/simonw/datasette/issues/1110#issuecomment-733432768,https://api.github.com/repos/simonw/datasette/issues/1110,733432768,MDEyOklzc3VlQ29tbWVudDczMzQzMjc2OA==,9599,simonw,2020-11-25T03:04:00Z,2020-11-25T03:04:00Z,OWNER,"Design:

    datasette publish cloudrun blah.db --apt-get-install ripgrep --install datasette-ripgrep","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750330029,datasette publish option for installing extra apt-get packages,
https://github.com/simonw/datasette/issues/1106#issuecomment-733221359,https://api.github.com/repos/simonw/datasette/issues/1106,733221359,MDEyOklzc3VlQ29tbWVudDczMzIyMTM1OQ==,9599,simonw,2020-11-24T20:40:21Z,2020-11-24T20:40:21Z,OWNER,https://readthedocs.org/dashboard/datasette/redirects/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749983857,Rebrand and redirect config.rst as settings.rst,
https://github.com/simonw/datasette/issues/1104#issuecomment-733212084,https://api.github.com/repos/simonw/datasette/issues/1104,733212084,MDEyOklzc3VlQ29tbWVudDczMzIxMjA4NA==,9599,simonw,2020-11-24T20:20:33Z,2020-11-24T20:20:33Z,OWNER,I'll throw an error if a `config.json` file is detected.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749981663,config.json in directory config mode should be settings.json,
https://github.com/simonw/datasette/issues/860#issuecomment-733288841,https://api.github.com/repos/simonw/datasette/issues/860,733288841,MDEyOklzc3VlQ29tbWVudDczMzI4ODg0MQ==,9599,simonw,2020-11-24T23:19:47Z,2020-11-24T23:20:24Z,OWNER,Here's what I have today - it's an undocumented `datasette.metadata()` method that returns a full JSON dictionary of values OR a single value if the optional `key=` argument is provided: https://github.com/simonw/datasette/blob/f2e2bfcdd9ad4891f3f66c9104c09943d943ffe4/datasette/app.py#L357-L388,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-733288522,https://api.github.com/repos/simonw/datasette/issues/860,733288522,MDEyOklzc3VlQ29tbWVudDczMzI4ODUyMg==,9599,simonw,2020-11-24T23:18:47Z,2020-11-24T23:18:47Z,OWNER,"In #942 I want to add support for per-column metadata - which means this new lookup mechanism will need to be able to answer the question ""what description is available for this column"".

So what should the `.metadata()` method look like? A couple of options:

- `datasette.metadata(""description"", table=x, database=y)` - can take optional `column=` too.
- `datasette.table_metadata(""description"", table=x, database=y)` and `datasette.database_metadata(""description"", database=y)` and so on - multiple methods for the different types of metadata.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-733287619,https://api.github.com/repos/simonw/datasette/issues/860,733287619,MDEyOklzc3VlQ29tbWVudDczMzI4NzYxOQ==,9599,simonw,2020-11-24T23:16:21Z,2020-11-24T23:16:21Z,OWNER,I'll also allow any key to be looked up - so if users want to invent their own metadata keys other than the default `license_url` etc they can do so.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-733287416,https://api.github.com/repos/simonw/datasette/issues/860,733287416,MDEyOklzc3VlQ29tbWVudDczMzI4NzQxNg==,9599,simonw,2020-11-24T23:15:44Z,2020-11-24T23:15:44Z,OWNER,"I'm going to go with a plugin hook (and Datasette method) that returns individual values - so you ask it for e.g. the `license_url` for a specific table and it returns a string or `None`.

The default plugin hook implementation that ships with Datasette will then implement cascading lookups against `metadata.json` - but other plugins will be able to provide their own implementations, which should make it easy to build a plugin that lets you keep metadata in a database file and edit it interactively.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/1107#issuecomment-733261501,https://api.github.com/repos/simonw/datasette/issues/1107,733261501,MDEyOklzc3VlQ29tbWVudDczMzI2MTUwMQ==,9599,simonw,2020-11-24T22:09:11Z,2020-11-24T22:09:11Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#setting-key,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750079085,Rename datasette.config() method to datasette.setting(),
https://github.com/simonw/datasette/issues/1107#issuecomment-733257071,https://api.github.com/repos/simonw/datasette/issues/1107,733257071,MDEyOklzc3VlQ29tbWVudDczMzI1NzA3MQ==,9599,simonw,2020-11-24T21:59:32Z,2020-11-24T21:59:32Z,OWNER,I'm going to make this a documented method in https://docs.datasette.io/en/latest/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750079085,Rename datasette.config() method to datasette.setting(),
https://github.com/simonw/datasette/issues/1105#issuecomment-733249176,https://api.github.com/repos/simonw/datasette/issues/1105,733249176,MDEyOklzc3VlQ29tbWVudDczMzI0OTE3Ng==,9599,simonw,2020-11-24T21:40:28Z,2020-11-24T21:40:28Z,OWNER,This rebranding is complete - #1107 is a follow-up internal refactor.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749982022,Rebrand config as settings,
https://github.com/simonw/datasette/issues/1106#issuecomment-733248437,https://api.github.com/repos/simonw/datasette/issues/1106,733248437,MDEyOklzc3VlQ29tbWVudDczMzI0ODQzNw==,9599,simonw,2020-11-24T21:38:50Z,2020-11-24T21:38:50Z,OWNER,"I used an ""exact redirect"" instead and it worked:

<img width=""654"" alt=""Edit_Redirects___Read_the_Docs"" src=""https://user-images.githubusercontent.com/9599/100154418-60bf0d80-2e5a-11eb-92bf-527a14edb5c5.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749983857,Rebrand and redirect config.rst as settings.rst,
https://github.com/simonw/datasette/issues/1106#issuecomment-733247101,https://api.github.com/repos/simonw/datasette/issues/1106,733247101,MDEyOklzc3VlQ29tbWVudDczMzI0NzEwMQ==,9599,simonw,2020-11-24T21:35:29Z,2020-11-24T21:36:04Z,OWNER,"<img width=""861"" alt=""Edit_Redirects___Read_the_Docs"" src=""https://user-images.githubusercontent.com/9599/100153884-a3341a80-2e59-11eb-8d8c-7027c714b9cf.png"">

https://docs.datasette.io/en/latest/config.html isn't redirecting though, even after I tried running a rebuild of the `latest` version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749983857,Rebrand and redirect config.rst as settings.rst,
https://github.com/simonw/datasette/issues/1106#issuecomment-733245596,https://api.github.com/repos/simonw/datasette/issues/1106,733245596,MDEyOklzc3VlQ29tbWVudDczMzI0NTU5Ng==,9599,simonw,2020-11-24T21:32:11Z,2020-11-24T21:32:11Z,OWNER,https://docs.datasette.io/en/latest/settings.html is now live - need to redirect https://docs.datasette.io/en/latest/config.html to it using the ReadTheDocs redirects interface.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749983857,Rebrand and redirect config.rst as settings.rst,
https://github.com/simonw/datasette/issues/1107#issuecomment-733245097,https://api.github.com/repos/simonw/datasette/issues/1107,733245097,MDEyOklzc3VlQ29tbWVudDczMzI0NTA5Nw==,9599,simonw,2020-11-24T21:31:10Z,2020-11-24T21:31:10Z,OWNER,"Most of these use `plugin_config` which is unaffected. It looks like the only code I need to worry about is this trick in `datasette-graphl`: https://github.com/simonw/datasette-graphql/blob/483c9a9e203bb90365def3df8b8f01dda1e75865/datasette_graphql/utils.py#L456-L460

```python
class DatasetteSpecialConfig(wrapt.ObjectProxy):
    def config(self, key):
        if key == ""suggest_facets"":
            return False
        return self.__wrapped__.config(key)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750079085,Rename datasette.config() method to datasette.setting(),
https://github.com/simonw/datasette/issues/1107#issuecomment-733244471,https://api.github.com/repos/simonw/datasette/issues/1107,733244471,MDEyOklzc3VlQ29tbWVudDczMzI0NDQ3MQ==,9599,simonw,2020-11-24T21:29:59Z,2020-11-24T21:29:59Z,OWNER,"I ran `rg '.config\(' datasette-*/` in my top-level directory:
```
datasette-sentry/test_datasette_sentry.py:    def plugin_config(self, name):
datasette-sentry/datasette_sentry.py:    config = datasette.plugin_config(""datasette-sentry"") or {}
datasette-render-markdown/datasette_render_markdown/__init__.py:        datasette.plugin_config(
datasette-render-images/datasette_render_images.py:        plugin_config = datasette.plugin_config(""datasette-render-images"") or {}
datasette-render-html/datasette_render_html.py:    config = datasette.plugin_config(
datasette-render-timestamps/datasette_render_timestamps/__init__.py:        datasette.plugin_config(
datasette-permissions-sql/datasette_permissions_sql/__init__.py:        for rule in datasette.plugin_config(""datasette-permissions-sql"") or []:
datasette-mask-columns/datasette_mask_columns/__init__.py:            datasette.plugin_config(""datasette-mask-columns"", database=database) or {}
datasette-mask-columns/datasette_mask_columns/__init__.py:    masks = datasette.plugin_config(""datasette-mask-columns"", database=database) or {}
datasette-media/datasette_media/__init__.py:    plugin_config = datasette.plugin_config(""datasette-media"") or {}
datasette-insert/datasette_insert/__init__.py:    plugin_config = datasette.plugin_config(""datasette-insert"") or {}
datasette-indieauth/datasette_indieauth/__init__.py:    plugin_config = datasette.plugin_config(""datasette-indieauth"") or {}
datasette-graphql/datasette_graphql/__init__.py:    config = datasette.plugin_config(""datasette-graphql"") or {}
datasette-graphql/datasette_graphql/__init__.py:    config = datasette.plugin_config(""datasette-graphql"") or {}
datasette-graphql/datasette_graphql/utils.py:            auto_camelcase=(datasette.plugin_config(""datasette-graphql"") or {}).get(
datasette-graphql/datasette_graphql/utils.py:    table_plugin_config = datasette.plugin_config(
datasette-graphql/datasette_graphql/utils.py:    def config(self, key):
datasette-graphql/datasette_graphql/utils.py:        return self.__wrapped__.config(key)
datasette-init/datasette_init/__init__.py:        config = datasette.plugin_config(""datasette-init"")
datasette-edit-templates/datasette_edit_templates/__init__.py:    plugin_config = datasette.plugin_config(""datasette-edit-templates"") or {}
datasette-cors/datasette_cors.py:    config = datasette.plugin_config(""datasette-cors"") or {}
datasette-cluster-map-old/build/lib/datasette_cluster_map/__init__.py:        datasette.plugin_config(""datasette-cluster-map"", database=database, table=table)
datasette-cluster-map/datasette_cluster_map/__init__.py:        datasette.plugin_config(""datasette-cluster-map"", database=database, table=table)
datasette-cluster-map/datasette_cluster_map/__init__.py:        datasette.plugin_config(""datasette-cluster-map"", database=database, table=table)
datasette-cluster-map/tests/test_cluster_map.py:async def test_plugin_config(db_path, config, table, expected_fragments):
datasette-configure-asgi/datasette_configure_asgi.py:        configs = datasette.plugin_config(""datasette-configure-asgi"") or []
datasette-configure-asgi/test_datasette_configure_asgi.py:    def plugin_config(self, name):
datasette-cluster-map-old/datasette_cluster_map/__init__.py:        datasette.plugin_config(""datasette-cluster-map"", database=database, table=table)
datasette-auth-passwords/datasette_auth_passwords/__init__.py:    config = datasette.plugin_config(""datasette-auth-passwords"") or {}
datasette-auth-github/datasette_auth_github/views.py:def verify_config(config):
datasette-auth-github/datasette_auth_github/views.py:    config = datasette.plugin_config(""datasette-auth-github"")
datasette-auth-github/datasette_auth_github/views.py:    verify_config(config)
datasette-auth-github/datasette_auth_github/views.py:    config = datasette.plugin_config(""datasette-auth-github"")
datasette-auth-github/datasette_auth_github/views.py:    verify_config(config)
datasette-atom/datasette_atom/__init__.py:        plugin_config = datasette.plugin_config(""datasette-atom"")
datasette-auth-google/datasette_auth_google/__init__.py:    config = datasette.plugin_config(""datasette-auth-github"") or {}
datasette-auth-existing-cookies/test_datasette_auth_existing_cookies.py:    def plugin_config(self, name):
datasette-auth-passwords/build/lib/datasette_auth_passwords/__init__.py:    config = datasette.plugin_config(""datasette-auth-passwords"") or {}
datasette-annotate/datasette_annotate/utils.py:    plugin_config = datasette.plugin_config(""datasette-annotate"") or {}
datasette-auth-existing-cookies/datasette_auth_existing_cookies/__init__.py:    config = datasette.plugin_config(""datasette-auth-existing-cookies"") or {}
datasette-auth-simple/datasette_auth_simple/__init__.py:datasette.plugin_config(""datasette-auth-simple"")
datasette-auth-tokens/datasette_auth_tokens/__init__.py:        config = datasette.plugin_config(""datasette-auth-tokens"") or {}
datasette-block-robots/datasette_block_robots/__init__.py:    config = datasette.plugin_config(""datasette-block-robots"") or {}
datasette-block-robots/datasette_block_robots/__init__.py:    config = datasette.plugin_config(""datasette-block-robots"") or {}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750079085,Rename datasette.config() method to datasette.setting(),
https://github.com/simonw/datasette/issues/1107#issuecomment-733241949,https://api.github.com/repos/simonw/datasette/issues/1107,733241949,MDEyOklzc3VlQ29tbWVudDczMzI0MTk0OQ==,9599,simonw,2020-11-24T21:24:26Z,2020-11-24T21:24:26Z,OWNER,Are there any plugins that use this API even though it isn't documented?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750079085,Rename datasette.config() method to datasette.setting(),
https://github.com/simonw/datasette/issues/226#issuecomment-733198051,https://api.github.com/repos/simonw/datasette/issues/226,733198051,MDEyOklzc3VlQ29tbWVudDczMzE5ODA1MQ==,9599,simonw,2020-11-24T19:52:46Z,2020-11-24T19:52:46Z,OWNER,This is well handled now: https://github.com/simonw/datasette/tree/0.51.1/tests/plugins,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",315738696,Unit tests for installable plugins,
https://github.com/simonw/datasette/issues/1105#issuecomment-733190827,https://api.github.com/repos/simonw/datasette/issues/1105,733190827,MDEyOklzc3VlQ29tbWVudDczMzE5MDgyNw==,9599,simonw,2020-11-24T19:38:02Z,2020-11-24T19:38:02Z,OWNER,I'd like to redirect https://docs.datasette.io/en/stable/config.html to a new https://docs.datasette.io/en/stable/settings.html page too. I can use https://docs.readthedocs.io/en/stable/user-defined-redirects.html for that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749982022,Rebrand config as settings,
https://github.com/simonw/datasette/issues/1103#issuecomment-733189737,https://api.github.com/repos/simonw/datasette/issues/1103,733189737,MDEyOklzc3VlQ29tbWVudDczMzE4OTczNw==,9599,simonw,2020-11-24T19:35:45Z,2020-11-24T19:35:45Z,OWNER,Part of #1105,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749979454,Rename /-/config to /-/settings,
https://github.com/simonw/datasette/issues/1104#issuecomment-733189620,https://api.github.com/repos/simonw/datasette/issues/1104,733189620,MDEyOklzc3VlQ29tbWVudDczMzE4OTYyMA==,9599,simonw,2020-11-24T19:35:30Z,2020-11-24T19:35:30Z,OWNER,Part of #1105,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749981663,config.json in directory config mode should be settings.json,
https://github.com/simonw/datasette/issues/992#issuecomment-733189693,https://api.github.com/repos/simonw/datasette/issues/992,733189693,MDEyOklzc3VlQ29tbWVudDczMzE4OTY5Mw==,9599,simonw,2020-11-24T19:35:38Z,2020-11-24T19:35:38Z,OWNER,Part of #1105,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714449879,"Change ""--config foo:bar"" to ""--setting foo bar""",
https://github.com/simonw/datasette/issues/1103#issuecomment-733187586,https://api.github.com/repos/simonw/datasette/issues/1103,733187586,MDEyOklzc3VlQ29tbWVudDczMzE4NzU4Ng==,9599,simonw,2020-11-24T19:31:23Z,2020-11-24T19:31:23Z,OWNER,I'll set up a redirect from `/-/config` to `/-/settings`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749979454,Rename /-/config to /-/settings,
https://github.com/simonw/datasette/issues/992#issuecomment-733180289,https://api.github.com/repos/simonw/datasette/issues/992,733180289,MDEyOklzc3VlQ29tbWVudDczMzE4MDI4OQ==,9599,simonw,2020-11-24T19:16:30Z,2020-11-24T19:16:30Z,OWNER,Need to figure out the `--setting foo bar` alternative for this `--config foo:bar` logic: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/cli.py#L31-L63,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714449879,"Change ""--config foo:bar"" to ""--setting foo bar""",
https://github.com/simonw/datasette/issues/992#issuecomment-733176252,https://api.github.com/repos/simonw/datasette/issues/992,733176252,MDEyOklzc3VlQ29tbWVudDczMzE3NjI1Mg==,9599,simonw,2020-11-24T19:07:49Z,2020-11-24T19:07:49Z,OWNER,I'm going to keep `--config` for the moment but show a deprecation warning that it will be gone in Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714449879,"Change ""--config foo:bar"" to ""--setting foo bar""",
https://github.com/simonw/datasette/issues/992#issuecomment-733175965,https://api.github.com/repos/simonw/datasette/issues/992,733175965,MDEyOklzc3VlQ29tbWVudDczMzE3NTk2NQ==,9599,simonw,2020-11-24T19:07:13Z,2020-11-24T19:07:13Z,OWNER,"This is blocking progress on other metadata tickets like #860 because I want to split the concept of concrete metadata (source, license, etc) from configuration that currently lives in metadata (default sort order, default facets).

I'm going to solve this next to unblock that stuff.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714449879,"Change ""--config foo:bar"" to ""--setting foo bar""",
https://github.com/simonw/datasette/issues/860#issuecomment-733175454,https://api.github.com/repos/simonw/datasette/issues/860,733175454,MDEyOklzc3VlQ29tbWVudDczMzE3NTQ1NA==,9599,simonw,2020-11-24T19:06:07Z,2020-11-24T19:06:07Z,OWNER,"I see two ways this plugin hook could work. It could be asked about a specific instance, database or table and return the full metadata for that object. OR it could ask for a specific metadata field - e.g. `source_url` for table X, and return that.

The more finely grained one would allow plugins to implement their own cascading rules pretty easily. Is there a reason it would be better for the hook to return an entire block of JSON for a specific table or database?

I also need to decide if this hook is just going to be about source/license/about displayed metadata, or if it will include the functionality that has been sneaking into `metadata.json` over time - stuff like page size, default sort order or default facets.

Perhaps I should split those out into a ""configuration"" concept first, after renaming `--config` to `--setting` in #992.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/1101#issuecomment-732544590,https://api.github.com/repos/simonw/datasette/issues/1101,732544590,MDEyOklzc3VlQ29tbWVudDczMjU0NDU5MA==,9599,simonw,2020-11-24T02:22:55Z,2020-11-24T02:22:55Z,OWNER,"The trick I'm using here is to follow the `next_url` in order to paginate through all of the matching results. The loop calls the `data()` method multiple times, once for each page of results: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/views/base.py#L304-L307","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1101#issuecomment-732543700,https://api.github.com/repos/simonw/datasette/issues/1101,732543700,MDEyOklzc3VlQ29tbWVudDczMjU0MzcwMA==,9599,simonw,2020-11-24T02:20:30Z,2020-11-24T02:20:30Z,OWNER,"Current design: https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette

```python
@hookimpl
def register_output_renderer(datasette):
    return {
        ""extension"": ""test"",
        ""render"": render_demo,
        ""can_render"": can_render_demo,  # Optional
    }
```
Where `render_demo` looks something like this:
```python
async def render_demo(datasette, columns, rows):
    db = datasette.get_database()
    result = await db.execute(""select sqlite_version()"")
    first_row = "" | "".join(columns)
    lines = [first_row]
    lines.append(""="" * len(first_row))
    for row in rows:
        lines.append("" | "".join(row))
    return Response(
        ""\n"".join(lines),
        content_type=""text/plain; charset=utf-8"",
        headers={""x-sqlite-version"": result.first()[0]}
    )
```
Meanwhile here's where the CSV streaming mode is implemented: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/views/base.py#L297-L380","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1096#issuecomment-732542285,https://api.github.com/repos/simonw/datasette/issues/1096,732542285,MDEyOklzc3VlQ29tbWVudDczMjU0MjI4NQ==,9599,simonw,2020-11-24T02:16:22Z,2020-11-24T02:16:22Z,OWNER,"I'd like to implement this by first extending the `register_output_renderer()` hook to support streaming huge responses, then switching CSV to use the plugin hook in addition to TSV using it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743359646,TSV should be a default export option,
https://github.com/simonw/datasette/issues/860#issuecomment-731658059,https://api.github.com/repos/simonw/datasette/issues/860,731658059,MDEyOklzc3VlQ29tbWVudDczMTY1ODA1OQ==,9599,simonw,2020-11-22T00:31:47Z,2020-11-22T00:33:48Z,OWNER,"Documented behaviour right now, for metadata set at the instance level, is: https://docs.datasette.io/en/stable/metadata.html

> The above metadata will be displayed on the index page of your Datasette-powered site. The source and license information will also be included in the footer of every page served by Datasette.
>
> ...
> 
> Metadata at the top level of the JSON will be shown on the index page and in the footer on every page of the site. The license and source is expected to apply to all of your data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-731657660,https://api.github.com/repos/simonw/datasette/issues/860,731657660,MDEyOklzc3VlQ29tbWVudDczMTY1NzY2MA==,9599,simonw,2020-11-22T00:27:32Z,2020-11-22T00:31:54Z,OWNER,"Open question: how should cascading work? If a table is missing a field but the database or instance has it, should that value cascade down to the table?

It feels like `license` should definitely cascade: if an instance lists a certain `license` that should absolutely filter through to all databases and tables.

But... should the other fields cascade? Cascading `description` doesn't feel right at all, and neither does `title`.

What about `about` and `about_url` and `source` and `source_url`? I'm a bit torn on whether they should cascade or not. I'm leaning towards cascading them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-731657486,https://api.github.com/repos/simonw/datasette/issues/860,731657486,MDEyOklzc3VlQ29tbWVudDczMTY1NzQ4Ng==,9599,simonw,2020-11-22T00:25:34Z,2020-11-22T00:25:34Z,OWNER,"There are three layers of metadata: table, database and instance.

Currently the metadata fields are (ignoring not-quite-metadata like `sort` and `sort_desc`):

- `title`
- `description` (or `description_html`)
- `about` / `about_url`
- `source` / `source_url`
- `license` / `license_url`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/1084#issuecomment-731654132,https://api.github.com/repos/simonw/datasette/issues/1084,731654132,MDEyOklzc3VlQ29tbWVudDczMTY1NDEzMg==,9599,simonw,2020-11-21T23:45:59Z,2020-11-21T23:45:59Z,OWNER,https://datasette-graphql-demo.datasette.io/github/users now demonstrates the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737394470,Table/database action menu cut off if too short ,
https://github.com/simonw/datasette/issues/1084#issuecomment-731653083,https://api.github.com/repos/simonw/datasette/issues/1084,731653083,MDEyOklzc3VlQ29tbWVudDczMTY1MzA4Mw==,9599,simonw,2020-11-21T23:35:07Z,2020-11-21T23:35:07Z,OWNER,I got to use CSS `calc()` for this: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/static/app.css#L367-L371,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737394470,Table/database action menu cut off if too short ,
https://github.com/simonw/datasette/issues/1084#issuecomment-731652991,https://api.github.com/repos/simonw/datasette/issues/1084,731652991,MDEyOklzc3VlQ29tbWVudDczMTY1Mjk5MQ==,9599,simonw,2020-11-21T23:34:22Z,2020-11-21T23:34:22Z,OWNER,"Fixed by positioning the menu relative to the `<div class=""page-header"">` element rather than the cog icon:

<img width=""966"" alt=""data"" src=""https://user-images.githubusercontent.com/9599/99889950-02eab580-2c0f-11eb-85dc-dd4bc840a92e.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737394470,Table/database action menu cut off if too short ,
https://github.com/simonw/datasette/issues/1084#issuecomment-731644064,https://api.github.com/repos/simonw/datasette/issues/1084,731644064,MDEyOklzc3VlQ29tbWVudDczMTY0NDA2NA==,9599,simonw,2020-11-21T22:10:15Z,2020-11-21T22:10:15Z,OWNER,"Another example of this bug: https://datasette-graphql-demo.datasette.io/github/users

<img width=""350"" alt=""github__users__4_823_rows_and_Comparing_1_1___main_·_simonw_datasette-graphql"" src=""https://user-images.githubusercontent.com/9599/99888639-3889a180-2c03-11eb-83b9-41f2eb12287e.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737394470,Table/database action menu cut off if too short ,
https://github.com/simonw/datasette/issues/1091#issuecomment-728262974,https://api.github.com/repos/simonw/datasette/issues/1091,728262974,MDEyOklzc3VlQ29tbWVudDcyODI2Mjk3NA==,9599,simonw,2020-11-16T19:05:08Z,2020-11-16T19:05:08Z,OWNER,I have a hunch that there may be some extra configuration in play here - could Apache itself be rewriting some of the links using [mod_proxy_html](https://httpd.apache.org/docs/2.4/mod/mod_proxy_html.html)?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/942#issuecomment-727626657,https://api.github.com/repos/simonw/datasette/issues/942,727626657,MDEyOklzc3VlQ29tbWVudDcyNzYyNjY1Nw==,9599,simonw,2020-11-15T19:54:44Z,2020-11-15T19:54:44Z,OWNER,This will also benefit from the metadata plugin hook: #860 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/1098#issuecomment-727656208,https://api.github.com/repos/simonw/datasette/issues/1098,727656208,MDEyOklzc3VlQ29tbWVudDcyNzY1NjIwOA==,9599,simonw,2020-11-15T23:26:14Z,2020-11-15T23:26:14Z,OWNER,"Schema for that broken example:
```sql
CREATE TABLE generators_eia860 (
	id INTEGER NOT NULL, 
	plant_id_eia INTEGER, 
	generator_id TEXT, 
	report_date DATE, 
	operational_status_code TEXT, 
	operational_status TEXT, 
	ownership_code TEXT, 
	utility_id_eia INTEGER, 
	capacity_mw FLOAT, 
	summer_capacity_mw FLOAT, 
	winter_capacity_mw FLOAT, 
	energy_source_code_1 TEXT, 
	energy_source_code_2 TEXT, 
	energy_source_code_3 TEXT, 
	energy_source_code_4 TEXT, 
	energy_source_code_5 TEXT, 
	energy_source_code_6 TEXT, 
	fuel_type_code_pudl TEXT, 
	multiple_fuels BOOLEAN, 
	deliver_power_transgrid BOOLEAN, 
	syncronized_transmission_grid BOOLEAN, 
	turbines_num INTEGER, 
	planned_modifications BOOLEAN, 
	planned_net_summer_capacity_uprate_mw FLOAT, 
	planned_net_winter_capacity_uprate_mw FLOAT, 
	planned_uprate_date DATE, 
	planned_net_summer_capacity_derate_mw FLOAT, 
	planned_net_winter_capacity_derate_mw FLOAT, 
	planned_derate_date DATE, 
	planned_new_prime_mover_code TEXT, 
	planned_energy_source_code_1 TEXT, 
	planned_repower_date DATE, 
	other_planned_modifications BOOLEAN, 
	other_modifications_date DATE, 
	planned_retirement_date DATE, 
	carbon_capture BOOLEAN, 
	startup_source_code_1 TEXT, 
	startup_source_code_2 TEXT, 
	startup_source_code_3 TEXT, 
	startup_source_code_4 TEXT, 
	technology_description TEXT, 
	turbines_inverters_hydrokinetics TEXT, 
	time_cold_shutdown_full_load_code TEXT, 
	planned_new_capacity_mw FLOAT, 
	cofire_fuels BOOLEAN, 
	switch_oil_gas BOOLEAN, 
	nameplate_power_factor FLOAT, 
	minimum_load_mw FLOAT, 
	uprate_derate_during_year BOOLEAN, 
	uprate_derate_completed_date DATE, 
	current_planned_operating_date DATE, 
	summer_estimated_capability_mw FLOAT, 
	winter_estimated_capability_mw FLOAT, 
	retirement_date DATE, 
	PRIMARY KEY (id), 
	FOREIGN KEY(plant_id_eia, generator_id) REFERENCES generators_entity_eia (plant_id_eia, generator_id), 
	FOREIGN KEY(utility_id_eia) REFERENCES utilities_entity_eia (utility_id_eia), 
	CHECK (multiple_fuels IN (0, 1)), 
	CHECK (deliver_power_transgrid IN (0, 1)), 
	CHECK (syncronized_transmission_grid IN (0, 1)), 
	CHECK (planned_modifications IN (0, 1)), 
	CHECK (other_planned_modifications IN (0, 1)), 
	CHECK (carbon_capture IN (0, 1)), 
	CHECK (cofire_fuels IN (0, 1)), 
	CHECK (switch_oil_gas IN (0, 1)), 
	CHECK (uprate_derate_during_year IN (0, 1))
);
```
https://pudl-datasette-xl7xwcpe2a-uc.a.run.app/pudl/generators_entity_eia is:
```sql
CREATE TABLE generators_entity_eia (
	plant_id_eia INTEGER NOT NULL, 
	generator_id TEXT NOT NULL, 
	prime_mover_code TEXT, 
	duct_burners BOOLEAN, 
	operating_date DATE, 
	topping_bottoming_code TEXT, 
	solid_fuel_gasification BOOLEAN, 
	pulverized_coal_tech BOOLEAN, 
	fluidized_bed_tech BOOLEAN, 
	subcritical_tech BOOLEAN, 
	supercritical_tech BOOLEAN, 
	ultrasupercritical_tech BOOLEAN, 
	stoker_tech BOOLEAN, 
	other_combustion_tech BOOLEAN, 
	bypass_heat_recovery BOOLEAN, 
	rto_iso_lmp_node_id TEXT, 
	rto_iso_location_wholesale_reporting_id TEXT, 
	associated_combined_heat_power BOOLEAN, 
	original_planned_operating_date DATE, 
	operating_switch TEXT, 
	previously_canceled BOOLEAN, 
	PRIMARY KEY (plant_id_eia, generator_id), 
	FOREIGN KEY(plant_id_eia) REFERENCES plants_entity_eia (plant_id_eia), 
	CHECK (duct_burners IN (0, 1)), 
	CHECK (solid_fuel_gasification IN (0, 1)), 
	CHECK (pulverized_coal_tech IN (0, 1)), 
	CHECK (fluidized_bed_tech IN (0, 1)), 
	CHECK (subcritical_tech IN (0, 1)), 
	CHECK (supercritical_tech IN (0, 1)), 
	CHECK (ultrasupercritical_tech IN (0, 1)), 
	CHECK (stoker_tech IN (0, 1)), 
	CHECK (other_combustion_tech IN (0, 1)), 
	CHECK (bypass_heat_recovery IN (0, 1)), 
	CHECK (associated_combined_heat_power IN (0, 1)), 
	CHECK (previously_canceled IN (0, 1))
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1098#issuecomment-727655636,https://api.github.com/repos/simonw/datasette/issues/1098,727655636,MDEyOklzc3VlQ29tbWVudDcyNzY1NTYzNg==,9599,simonw,2020-11-15T23:22:27Z,2020-11-15T23:22:27Z,OWNER,"Need to replicate this in the fixtures, then fix it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1091#issuecomment-727233553,https://api.github.com/repos/simonw/datasette/issues/1091,727233553,MDEyOklzc3VlQ29tbWVudDcyNzIzMzU1Mw==,9599,simonw,2020-11-14T16:46:52Z,2020-11-14T16:46:52Z,OWNER,@tballison could I see the section of your Apache config that configures the proxying to `/datasette/`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/268#issuecomment-726419027,https://api.github.com/repos/simonw/datasette/issues/268,726419027,MDEyOklzc3VlQ29tbWVudDcyNjQxOTAyNw==,9599,simonw,2020-11-13T00:09:04Z,2020-11-13T00:09:04Z,OWNER,Part of the challenge here is that this is the first time the `TableView` will have had a complete rewrite of the SQL it is going to execute. That SQL is currently constructed here: https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/views/table.py#L628-L636,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/749#issuecomment-726417847,https://api.github.com/repos/simonw/datasette/issues/749,726417847,MDEyOklzc3VlQ29tbWVudDcyNjQxNzg0Nw==,9599,simonw,2020-11-13T00:05:14Z,2020-11-13T00:05:14Z,OWNER,"https://cloud.google.com/blog/products/serverless/cloud-run-now-supports-http-grpc-server-streaming indicates this limit should no longer apply:

> With this addition, Cloud Run can now ... Send responses larger than the previous 32 MB limit

But I'm still getting errors from Cloud Run attempting to download `.db` files larger than 32 MB.

I filed a question in their issue tracker about that here: https://issuetracker.google.com/issues/173038375","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610829227,Cloud Run fails to serve database files larger than 32MB,
https://github.com/simonw/datasette/issues/1091#issuecomment-726416330,https://api.github.com/repos/simonw/datasette/issues/1091,726416330,MDEyOklzc3VlQ29tbWVudDcyNjQxNjMzMA==,9599,simonw,2020-11-13T00:00:43Z,2020-11-13T00:00:43Z,OWNER,Here's where `url_csv` comes from: https://github.com/simonw/datasette/blob/11eb1e026f3d84cb771f8d6e204939cbaee130cd/datasette/views/base.py#L542-L545,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-726415991,https://api.github.com/repos/simonw/datasette/issues/1091,726415991,MDEyOklzc3VlQ29tbWVudDcyNjQxNTk5MQ==,9599,simonw,2020-11-12T23:59:34Z,2020-11-12T23:59:34Z,OWNER,"The sort headers are generated by this template code: https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/templates/_table.html#L11-L15

The export links use this code:

https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/templates/table.html#L134

https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/templates/table.html#L180-L201","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-726415019,https://api.github.com/repos/simonw/datasette/issues/1091,726415019,MDEyOklzc3VlQ29tbWVudDcyNjQxNTAxOQ==,9599,simonw,2020-11-12T23:56:23Z,2020-11-12T23:56:23Z,OWNER,@tballison is there any chance you're running any custom templates in that installation? I'm really confused as to why I can't replicate the bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-726413829,https://api.github.com/repos/simonw/datasette/issues/1091,726413829,MDEyOklzc3VlQ29tbWVudDcyNjQxMzgyOQ==,9599,simonw,2020-11-12T23:52:50Z,2020-11-12T23:54:16Z,OWNER,"Hmm... it's not just the `.csv` and `.json` export links - it's the column headings (which can be clicked to change the sort order) as well. Here's an extract of the HTML from that page:
```html
<p class=""export-links"">This data as 
  <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json"">json</a>, 
  <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.csv?_size=max"">CSV</a> (
  <a href=""#export"">advanced</a>)
</p>
<div class=""table-wrapper"">
  <table class=""rows-and-columns"">
    <thead>
      <tr>
        <th class=""col-Link"" scope=""col"" data-column=""Link"" data-column-type="""" data-column-not-null=""0"" data-is-pk=""0"">
          Link
        </th>
        <th class=""col-rowid"" scope=""col"" data-column=""rowid"" data-column-type=""integer"" data-column-not-null=""0"" data-is-pk=""1"">
          <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort_desc=rowid"" rel=""nofollow"">rowid&nbsp;▼</a>
        </th>
        <th class=""col-PARSE_EXCEPTION_ID"" scope=""col"" data-column=""PARSE_EXCEPTION_ID"" data-column-type=""INTEGER"" data-column-not-null=""0"" data-is-pk=""0"">
          <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort=PARSE_EXCEPTION_ID"" rel=""nofollow"">PARSE_EXCEPTION_ID</a>
        </th>
        <th class=""col-PARSE_EXCEPTION_DESCRIPTION"" scope=""col"" data-column=""PARSE_EXCEPTION_DESCRIPTION"" data-column-type=""VARCHAR(128)"" data-column-not-null=""0"" data-is-pk=""0"">
          <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort=PARSE_EXCEPTION_DESCRIPTION"" rel=""nofollow"">PARSE_EXCEPTION_DESCRIPTION</a>
        </th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td class=""col-Link type-pk"">
          <a href=""/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/1"">1</a>
        </td>
        <td class=""col-rowid type-int"">1</td>
        <td class=""col-PARSE_EXCEPTION_ID type-int"">0</td>
        <td class=""col-PARSE_EXCEPTION_DESCRIPTION type-str"">RUNTIME</td>
      </tr>
      <tr>
        <td class=""col-Link type-pk"">
          <a href=""/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/2"">2</a>
        </td>
        <td class=""col-rowid type-int"">2</td>
        <td class=""col-PARSE_EXCEPTION_ID type-int"">1</td>
        <td class=""col-PARSE_EXCEPTION_DESCRIPTION type-str"">ENCRYPTION</td>
      </tr>
      <tr>
        <td class=""col-Link type-pk"">
          <a href=""/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/3"">3</a>
        </td>
        <td class=""col-rowid type-int"">3</td>
        <td class=""col-PARSE_EXCEPTION_ID type-int"">2</td>
        <td class=""col-PARSE_EXCEPTION_DESCRIPTION type-str"">ACCESS_PERMISSION</td>
      </tr>
      <tr>
        <td class=""col-Link type-pk"">
          <a href=""/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/4"">4</a>
        </td>
        <td class=""col-rowid type-int"">4</td>
        <td class=""col-PARSE_EXCEPTION_ID type-int"">3</td>
        <td class=""col-PARSE_EXCEPTION_DESCRIPTION type-str"">UNSUPPORTED_VERSION</td>
      </tr>
    </tbody>
  </table>
</div>
<div id=""export"" class=""advanced-export"">
  <h3>Advanced export</h3>
  <p>JSON shape:
    <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json"">default</a>,
    <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json?_shape=array"">array</a>,
    <a href=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json?_shape=array&amp;_nl=on"">newline-delimited</a>
  </p>
  <form action=""/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.csv"" method=""get"">
    <p>
      CSV options:            
      <label>
        <input type=""checkbox"" name=""_dl""> download file
        </label>
        <input type=""submit"" value=""Export CSV"">
          <input type=""hidden"" name=""_size"" value=""max"">
          </p>
        </form>
      </div>
```
But here's something _really_ weird - the links to the individual rows DO include the `/datasette/` prefix:
```html
<td class=""col-Link type-pk"">
    <a href=""/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/2"">2</a>
</td>
```
The navigation bar on that page is correct too:
```html
<p class=""crumbs"">
        <a href=""/datasette/"">home</a> /
        <a href=""/datasette/corpora-metadata"">corpora-metadata</a>
</p>
```
I've also been unable to replicate this in my own local environment, running `datasette fixtures.db --config base_url:/datasette/`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/865#issuecomment-726412057,https://api.github.com/repos/simonw/datasette/issues/865,726412057,MDEyOklzc3VlQ29tbWVudDcyNjQxMjA1Nw==,9599,simonw,2020-11-12T23:49:23Z,2020-11-12T23:49:23Z,OWNER,"@tballison thanks, I've split that out into a new issue #1091","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/1089#issuecomment-726127465,https://api.github.com/repos/simonw/datasette/issues/1089,726127465,MDEyOklzc3VlQ29tbWVudDcyNjEyNzQ2NQ==,9599,simonw,2020-11-12T14:54:11Z,2020-11-12T14:54:11Z,OWNER,"Suggested list to look out for from that PR:

- simply/simple
- easy/easier/easiest
- obvious/obviously
- just
- merely
- straightforward
- ridiculous","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741665726,Sweep documentation for words that minimize involved difficulty,
https://github.com/simonw/datasette/issues/1088#issuecomment-725830716,https://api.github.com/repos/simonw/datasette/issues/1088,725830716,MDEyOklzc3VlQ29tbWVudDcyNTgzMDcxNg==,9599,simonw,2020-11-12T04:35:38Z,2020-11-12T04:35:38Z,OWNER,"I'm going to fix this without a test, because writing a test for this is a bit fiddly and it's a very minor bug. If it comes back again I'll do the work to test for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741268956,OperationalError('interrupted') can 500 on row page,
https://github.com/simonw/datasette/issues/1088#issuecomment-725830533,https://api.github.com/repos/simonw/datasette/issues/1088,725830533,MDEyOklzc3VlQ29tbWVudDcyNTgzMDUzMw==,9599,simonw,2020-11-12T04:35:08Z,2020-11-12T04:35:08Z,OWNER,"Yup, swapping `QueryInterrupted` fixed this against my giant database.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741268956,OperationalError('interrupted') can 500 on row page,
https://github.com/simonw/datasette/issues/1088#issuecomment-725829903,https://api.github.com/repos/simonw/datasette/issues/1088,725829903,MDEyOklzc3VlQ29tbWVudDcyNTgyOTkwMw==,9599,simonw,2020-11-12T04:33:14Z,2020-11-12T04:33:14Z,OWNER,"I'm suspicious of this code: https://github.com/simonw/datasette/blob/e8e0a6f284ca953b2980186c4356594c07bd1929/datasette/views/table.py#L1032-L1045

This code uses a different exception: https://github.com/simonw/datasette/blob/e8e0a6f284ca953b2980186c4356594c07bd1929/datasette/views/table.py#L658-L663

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741268956,OperationalError('interrupted') can 500 on row page,
https://github.com/simonw/datasette/issues/1086#issuecomment-725729857,https://api.github.com/repos/simonw/datasette/issues/1086,725729857,MDEyOklzc3VlQ29tbWVudDcyNTcyOTg1Nw==,9599,simonw,2020-11-11T23:55:39Z,2020-11-11T23:55:39Z,OWNER,"Demo: https://latest.datasette.io/fixtures/foreign_key_references?_facet=foreign_key_with_blank_label

<img width=""772"" alt=""fixtures__foreign_key_references__2_rows"" src=""https://user-images.githubusercontent.com/9599/98877868-53575b80-2436-11eb-8669-f24eeb8d0f3c.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741021342,Foreign keys with blank titles result in non-clickable links,
https://github.com/simonw/datasette/issues/1086#issuecomment-725714908,https://api.github.com/repos/simonw/datasette/issues/1086,725714908,MDEyOklzc3VlQ29tbWVudDcyNTcxNDkwOA==,9599,simonw,2020-11-11T23:17:26Z,2020-11-11T23:17:26Z,OWNER,I'm just going to use a regular coloured hyphen.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741021342,Foreign keys with blank titles result in non-clickable links,
https://github.com/simonw/datasette/issues/1086#issuecomment-725622784,https://api.github.com/repos/simonw/datasette/issues/1086,725622784,MDEyOklzc3VlQ29tbWVudDcyNTYyMjc4NA==,9599,simonw,2020-11-11T19:41:45Z,2020-11-11T19:41:45Z,OWNER,Maybe use a grey hyphen here?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741021342,Foreign keys with blank titles result in non-clickable links,
https://github.com/simonw/sqlite-utils/issues/168#issuecomment-723829147,https://api.github.com/repos/simonw/sqlite-utils/issues/168,723829147,MDEyOklzc3VlQ29tbWVudDcyMzgyOTE0Nw==,9599,simonw,2020-11-09T07:43:30Z,2020-11-09T07:43:30Z,OWNER,Yeah whatever process the have in place is working great without any extra intervention: they upgraded to 3.0 four hours ago.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706167456,Automate (as much as possible) updates published to Homebrew,
https://github.com/simonw/datasette/issues/268#issuecomment-723740546,https://api.github.com/repos/simonw/datasette/issues/268,723740546,MDEyOklzc3VlQ29tbWVudDcyMzc0MDU0Ng==,9599,simonw,2020-11-09T04:01:50Z,2020-11-09T04:01:50Z,OWNER,I should depend on `sqlite-fts4` - I'm doing that in `sqlite-utils` now and it works great: https://github.com/simonw/sqlite-utils/issues/198,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/sqlite-utils/issues/192#issuecomment-723637930,https://api.github.com/repos/simonw/sqlite-utils/issues/192,723637930,MDEyOklzc3VlQ29tbWVudDcyMzYzNzkzMA==,9599,simonw,2020-11-08T17:06:56Z,2020-11-08T17:06:56Z,OWNER,"This looks pretty good now!
```
% sqlite-utils search 24ways.db articles simon -c title -c author -t 
title                                                                          author
-----------------------------------------------------------------------------  ------------------
Don't be eval()                                                                Simon Willison
DOM Scripting Your Way to Better Blockquotes                                   Jeremy Keith
Swooshy Curly Quotes Without Images                                            Simon Collison
The Articulate Web Designer of Tomorrow                                        Simon Collison
Writing Responsible JavaScript                                                 Drew McLellan
Going Nuts with CSS Transitions                                                Natalie Downe
Managing a Mind                                                                Christopher Murphy
Design Systems                                                                 Laura Kalbag
Bringing Your Code to the Streets                                              Ruth John
Taming Complexity                                                              Simon Collison
Unobtrusively Mapping Microformats with jQuery                                 Simon Willison
Crafting the Front-end                                                         Ben Bodien
Develop Your Naturalist Superpowers with Observable Notebooks and iNaturalist  Natalie Downe
Fast Autocomplete Search for Your Website                                      Simon Willison
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command,
https://github.com/simonw/sqlite-utils/issues/201#issuecomment-723369033,https://api.github.com/repos/simonw/sqlite-utils/issues/201,723369033,MDEyOklzc3VlQ29tbWVudDcyMzM2OTAzMw==,9599,simonw,2020-11-07T01:28:11Z,2020-11-07T01:28:11Z,OWNER,Need to fix this to close #192 and #197.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",738128913,.search(columns=) and sqlite-utils search -c ... bug,
https://github.com/simonw/sqlite-utils/issues/194#issuecomment-723368528,https://api.github.com/repos/simonw/sqlite-utils/issues/194,723368528,MDEyOklzc3VlQ29tbWVudDcyMzM2ODUyOA==,9599,simonw,2020-11-07T01:24:55Z,2020-11-07T01:24:55Z,OWNER,Here's the alpha release: https://github.com/simonw/sqlite-utils/releases/tag/3.0a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735650864,3.0 release with some minor breaking changes,
https://github.com/simonw/sqlite-utils/issues/197#issuecomment-723365651,https://api.github.com/repos/simonw/sqlite-utils/issues/197,723365651,MDEyOklzc3VlQ29tbWVudDcyMzM2NTY1MQ==,9599,simonw,2020-11-07T01:06:32Z,2020-11-07T01:06:32Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#searching-with-table-search,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737153927,Rethink how table.search() method works,
https://github.com/simonw/sqlite-utils/issues/197#issuecomment-723360842,https://api.github.com/repos/simonw/sqlite-utils/issues/197,723360842,MDEyOklzc3VlQ29tbWVudDcyMzM2MDg0Mg==,9599,simonw,2020-11-07T00:40:55Z,2020-11-07T00:40:55Z,OWNER,"The `order=` parameter should be called `order_by` for consistency with this:
```python
for row in db[""dogs""].rows_where(""age > 1"", order_by=""age""):
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737153927,Rethink how table.search() method works,
https://github.com/simonw/sqlite-utils/issues/200#issuecomment-723357855,https://api.github.com/repos/simonw/sqlite-utils/issues/200,723357855,MDEyOklzc3VlQ29tbWVudDcyMzM1Nzg1NQ==,9599,simonw,2020-11-07T00:24:37Z,2020-11-07T00:24:37Z,OWNER,"```
(sqlite-utils) sqlite-utils % sqlite-utils rows 24ways-fts4.db articles -c title -c author | head -n 3
[{""title"": ""Why Bother with Accessibility?"", ""author"": ""Laura Kalbag""},
 {""title"": ""Levelling Up"", ""author"": ""Ashley Baxter""},
 {""title"": ""Project Hubs: A Home Base for Design Projects"", ""author"": ""Brad Frost""},
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",738115165,sqlite-utils rows -c option,
https://github.com/simonw/sqlite-utils/issues/192#issuecomment-723357117,https://api.github.com/repos/simonw/sqlite-utils/issues/192,723357117,MDEyOklzc3VlQ29tbWVudDcyMzM1NzExNw==,9599,simonw,2020-11-07T00:21:05Z,2020-11-07T00:21:05Z,OWNER,"I'm going to have it only output the exact `-c` columns you requested (if you requested any). Add `--rank` to specify the rank column, since you may not know what its name is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command,
https://github.com/simonw/sqlite-utils/issues/194#issuecomment-723356020,https://api.github.com/repos/simonw/sqlite-utils/issues/194,723356020,MDEyOklzc3VlQ29tbWVudDcyMzM1NjAyMA==,9599,simonw,2020-11-07T00:16:06Z,2020-11-07T00:16:06Z,OWNER,"I'm going to release this as an alpha first and sit on it for a few days, since I don't want to ship any mistakes that would result in having to bump straight to 4.0!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735650864,3.0 release with some minor breaking changes,
https://github.com/simonw/sqlite-utils/issues/91#issuecomment-723350956,https://api.github.com/repos/simonw/sqlite-utils/issues/91,723350956,MDEyOklzc3VlQ29tbWVudDcyMzM1MDk1Ng==,9599,simonw,2020-11-06T23:53:25Z,2020-11-06T23:53:25Z,OWNER,"This is now possible, for both FTS4 and FTS5 - see #197.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",577302229,Enable ordering FTS results by rank,
https://github.com/simonw/sqlite-utils/issues/192#issuecomment-723348722,https://api.github.com/repos/simonw/sqlite-utils/issues/192,723348722,MDEyOklzc3VlQ29tbWVudDcyMzM0ODcyMg==,9599,simonw,2020-11-06T23:43:09Z,2020-11-06T23:43:09Z,OWNER,"Also that order looks incorrect. It looks like most relevant came back last, not first.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command,
https://github.com/simonw/sqlite-utils/issues/192#issuecomment-723348614,https://api.github.com/repos/simonw/sqlite-utils/issues/192,723348614,MDEyOklzc3VlQ29tbWVudDcyMzM0ODYxNA==,9599,simonw,2020-11-06T23:42:38Z,2020-11-06T23:42:38Z,OWNER,"This is a bit surprising:
```
(sqlite-utils) sqlite-utils % sqlite-utils search 24ways-fts4.db articles maps -c title      
[{""rowid"": 41, ""title"": ""What Is Vagrant and Why Should I Care?"", ""rank"": -1.9252039178908076},
 {""rowid"": 298, ""title"": ""First Steps in VR"", ""rank"": -1.9945466378736434},
 {""rowid"": 43, ""title"": ""Content Production Planning"", ""rank"": -2.1928058363046143},
 {""rowid"": 100, ""title"": ""Moo'y Christmas"", ""rank"": -2.2698482999851675},
 {""rowid"": 91, ""title"": ""Infinite Canvas: Moving Beyond the Page"", ""rank"": -2.290928999035195},
 {""rowid"": 175, ""title"": ""Front-End Code Reusability with CSS and JavaScript"", ""rank"": -2.498731782924352},
 {""rowid"": 209, ""title"": ""Feeding the Audio Graph"", ""rank"": -2.619968930100356},
 {""rowid"": 296, ""title"": ""Animation in Design Systems"", ""rank"": -2.62060151817201},
 {""rowid"": 118, ""title"": ""Ghosts On The Internet"", ""rank"": -2.7224894534521087},
 {""rowid"": 77, ""title"": ""Colour Accessibility"", ""rank"": -2.7389782859427343},
 {""rowid"": 245, ""title"": ""Web Content Accessibility Guidelines 2.1\u2014for People Who Haven\u2019t Read the Update"", ""rank"": -2.9750992611162888},
 {""rowid"": 56, ""title"": ""Helping VIPs Care About Performance"", ""rank"": -3.0819662908932535},
 {""rowid"": 109, ""title"": ""Geotag Everywhere with Fire Eagle"", ""rank"": -3.1371975973877277},
 {""rowid"": 203, ""title"": ""Jobs-to-Be-Done in Your UX Toolbox"", ""rank"": -3.2416719461682733},
 {""rowid"": 276, ""title"": ""Your jQuery: Now With 67% Less Suck"", ""rank"": -3.4947916564653028},
 {""rowid"": 58, ""title"": ""Beyond the Style Guide"", ""rank"": -3.7508321464447905},
 {""rowid"": 225, ""title"": ""Good Ideas Grow on Paper"", ""rank"": -4.120077674716844},
 {""rowid"": 168, ""title"": ""Unobtrusively Mapping Microformats with jQuery"", ""rank"": -4.662224207228984},
 {""rowid"": 27, ""title"": ""Putting Design on the Map"", ""rank"": -5.667327088267961},
 {""rowid"": 220, ""title"": ""Finding Your Way with Static Maps"", ""rank"": -9.952534352591737}]
```
I requested just `-c title` but also got back `rowid` and `rank`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command,
https://github.com/simonw/sqlite-utils/issues/194#issuecomment-723234493,https://api.github.com/repos/simonw/sqlite-utils/issues/194,723234493,MDEyOklzc3VlQ29tbWVudDcyMzIzNDQ5Mw==,9599,simonw,2020-11-06T18:32:34Z,2020-11-06T18:32:34Z,OWNER,"The breaking changes will be:

- `table.search()` now returns a generator that produces dictionaries, similar to `table.rows_where()`
- The `-c` shortcut no longer works, use `--csv` instead.
- The `-f` shortcut no longer works, use `--fmt` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735650864,3.0 release with some minor breaking changes,
https://github.com/simonw/sqlite-utils/issues/197#issuecomment-723230732,https://api.github.com/repos/simonw/sqlite-utils/issues/197,723230732,MDEyOklzc3VlQ29tbWVudDcyMzIzMDczMg==,9599,simonw,2020-11-06T18:24:29Z,2020-11-06T18:24:29Z,OWNER,Still need to update docs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737153927,Rethink how table.search() method works,
https://github.com/simonw/sqlite-utils/pull/195#issuecomment-723148906,https://api.github.com/repos/simonw/sqlite-utils/issues/195,723148906,MDEyOklzc3VlQ29tbWVudDcyMzE0ODkwNg==,9599,simonw,2020-11-06T15:43:51Z,2020-11-06T15:43:51Z,OWNER,Thanks to #198 (introducing a `rank_bm25()` custom function for FTS4) this feature will be able to offer relevance search for both FTS5 AND FTS4 tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735663855,table.search() improvements plus sqlite-utils search command,
https://github.com/simonw/sqlite-utils/issues/197#issuecomment-723148310,https://api.github.com/repos/simonw/sqlite-utils/issues/197,723148310,MDEyOklzc3VlQ29tbWVudDcyMzE0ODMxMA==,9599,simonw,2020-11-06T15:42:43Z,2020-11-06T15:42:43Z,OWNER,Having `.search()` return tuples when `.rows_where()` returns dictionaries just feels like bad API design to me - it's inconsistent. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737153927,Rethink how table.search() method works,
https://github.com/simonw/sqlite-utils/issues/199#issuecomment-723147463,https://api.github.com/repos/simonw/sqlite-utils/issues/199,723147463,MDEyOklzc3VlQ29tbWVudDcyMzE0NzQ2Mw==,9599,simonw,2020-11-06T15:41:00Z,2020-11-06T15:41:00Z,OWNER,"Something like this:
```
@db.register_function(replace=True)
def my_function(a):
    return a.upper()
```
If `replace=True` then this function will be registered even if a `my_function` of arity 1 has already been registered previously.

It defaults to `False` though which means the Database object tracks what functions and arities have been registered in the past and silently ignores any new attempts to register the same name/arity.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737855731,"@db.register_function(..., replace=False) to avoid double-registering custom functions",
https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723145383,https://api.github.com/repos/simonw/sqlite-utils/issues/198,723145383,MDEyOklzc3VlQ29tbWVudDcyMzE0NTM4Mw==,9599,simonw,2020-11-06T15:36:47Z,2020-11-06T15:36:47Z,OWNER,"Should I register the custom `rank_bm25` SQLite function for every connection, or should I register it against the connection just the first time the user attempts an FTS4 search? I think I'd rather register it only if it is needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737476423,Support order by relevance against FTS4,
https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723144893,https://api.github.com/repos/simonw/sqlite-utils/issues/198,723144893,MDEyOklzc3VlQ29tbWVudDcyMzE0NDg5Mw==,9599,simonw,2020-11-06T15:35:45Z,2020-11-06T15:35:45Z,OWNER,"Here's a demo of that rank query. I had to sort by `rank` instead of `rank desc` - need to double-check that:

https://datasette-sqlite-fts4.datasette.io/24ways-fts4?sql=with+original+as+(%0D%0A++++select%0D%0A++++++++rowid%2C%0D%0A++++++++*%0D%0A++++from+[articles]%0D%0A)%0D%0Aselect%0D%0A++++original.*%2C%0D%0A++++rank_bm25(matchinfo([articles_fts]%2C+%27pcnalx%27))+as+rank%0D%0Afrom%0D%0A++++[original]%0D%0A++++join+[articles_fts]+on+[original].rowid+%3D+[articles_fts].rowid%0D%0Awhere%0D%0A++++[articles_fts]+match+%3Aquery%0D%0Aorder+by%0D%0A++++rank%0D%0Alimit+20&query=jquery+maps","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737476423,Support order by relevance against FTS4,
https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723143633,https://api.github.com/repos/simonw/sqlite-utils/issues/198,723143633,MDEyOklzc3VlQ29tbWVudDcyMzE0MzYzMw==,9599,simonw,2020-11-06T15:33:12Z,2020-11-06T15:33:12Z,OWNER,"Here's the FTS5 query:
```sql
with original as (
    select
        rowid,
        *
    from [global-power-plants]
)
select
    original.*,
    [global-power-plants_fts].rank as rank
from
    [original]
    join [global-power-plants_fts] on [original].rowid = [global-power-plants_fts].rowid
where
    [global-power-plants_fts] match :query
order by
    rank desc
limit 20
```
The equivalent using `rank_bm25()` for FTS4 would be:
```sql
with original as (
    select
        rowid,
        *
    from [global-power-plants]
)
select
    original.*,
    rank_bm25(matchinfo([global-power-plants_fts], 'pcnalx')) as rank
from
    [original]
    join [global-power-plants_fts] on [original].rowid = [global-power-plants_fts].rowid
where
    [global-power-plants_fts] match :query
order by
    rank desc
limit 20
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737476423,Support order by relevance against FTS4,
https://github.com/simonw/sqlite-utils/issues/198#issuecomment-722895825,https://api.github.com/repos/simonw/sqlite-utils/issues/198,722895825,MDEyOklzc3VlQ29tbWVudDcyMjg5NTgyNQ==,9599,simonw,2020-11-06T06:29:17Z,2020-11-06T06:29:17Z,OWNER,I released a 1.0 (and 1.0.1) version of https://github.com/simonw/sqlite-fts4,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737476423,Support order by relevance against FTS4,
https://github.com/simonw/sqlite-utils/issues/198#issuecomment-722852262,https://api.github.com/repos/simonw/sqlite-utils/issues/198,722852262,MDEyOklzc3VlQ29tbWVudDcyMjg1MjI2Mg==,9599,simonw,2020-11-06T05:41:58Z,2020-11-06T05:41:58Z,OWNER,"Example query (from the tests):
```sql
select c0, c1, rank_bm25(matchinfo(search, 'pcnalx')) as bm25
        from search where search match ?
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737476423,Support order by relevance against FTS4,
https://github.com/simonw/sqlite-utils/issues/198#issuecomment-722849539,https://api.github.com/repos/simonw/sqlite-utils/issues/198,722849539,MDEyOklzc3VlQ29tbWVudDcyMjg0OTUzOQ==,9599,simonw,2020-11-06T05:39:17Z,2020-11-06T05:39:17Z,OWNER,I'd have to copy almost all of the code in https://github.com/simonw/sqlite-fts4/blob/master/sqlite_fts4/__init__.py so I think I will add it as a dependency instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737476423,Support order by relevance against FTS4,
https://github.com/simonw/sqlite-utils/issues/197#issuecomment-722545442,https://api.github.com/repos/simonw/sqlite-utils/issues/197,722545442,MDEyOklzc3VlQ29tbWVudDcyMjU0NTQ0Mg==,9599,simonw,2020-11-05T18:05:33Z,2020-11-05T18:05:33Z,OWNER,This is likely to result in a 3.0 release due to a backwards-incompatible change to the current `.search()` method - #194,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737153927,Rethink how table.search() method works,
https://github.com/simonw/sqlite-utils/pull/195#issuecomment-722542895,https://api.github.com/repos/simonw/sqlite-utils/issues/195,722542895,MDEyOklzc3VlQ29tbWVudDcyMjU0Mjg5NQ==,9599,simonw,2020-11-05T18:01:33Z,2020-11-05T18:01:33Z,OWNER,"Latest test failure:
```
114  ->	        assert [(""racoons are biting trash pandas"", ""USA"", ""bar"")] == table.search(
115  	            ""bite"", order=""rowid""
116  	        )
117  	
118  	
119  	def test_optimize_fts(fresh_db):
(Pdb) table.search(""bite"")
[(2, 'racoons are biting trash pandas', 'USA', 'bar', -9.641434262948206e-07)]
```
The problem here is that the `table.search()` method now behaves differently for FTS4 v.s. FTS5 tables.

With FTS4 you get back just the table columns.

With FTS5 you also get back the `rowid` as the first column and the `rank` score as the last column.

This is weird. It also makes me question whether having `.search()` return a list of tuples is the right API design.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735663855,table.search() improvements plus sqlite-utils search command,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722086105,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722086105,MDEyOklzc3VlQ29tbWVudDcyMjA4NjEwNQ==,9599,simonw,2020-11-05T02:29:50Z,2020-11-05T03:39:58Z,OWNER,"The finished monster:

```python
_virtual_table_using_re = re.compile(r""""""
^ # Start of string
\s*CREATE\s+VIRTUAL\s+TABLE\s+ # CREATE VIRTUAL TABLE
(
    '(?P<squoted_table>[^']*(?:''[^']*)*)' | # single quoted name
    ""(?P<dquoted_table>[^""]*(?:""""[^""]*)*)"" | # double quoted name
    `(?P<backtick_table>[^`]+)`            | # `backtick` quoted name
    \[(?P<squarequoted_table>[^\]]+)\]     | # [...] quoted name
    (?P<identifier>                          # SQLite non-quoted identifier
        [A-Za-z_\u0080-\uffff]  # \u0080-\uffff = ""any character larger than u007f""
        [A-Za-z_\u0080-\uffff0-9\$]* # zero-or-more alphanemuric or $
    )
)
\s+(IF\s+NOT\s+EXISTS\s+)?      # IF NOT EXISTS (optional)
USING\s+(?P<using>\w+)          # e.g. USING FTS5
"""""", re.VERBOSE | re.IGNORECASE)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722084593,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722084593,MDEyOklzc3VlQ29tbWVudDcyMjA4NDU5Mw==,9599,simonw,2020-11-05T02:24:47Z,2020-11-05T02:24:47Z,OWNER,"And an identifier is ""ALPHABETIC character and continue with zero or more ALPHANUMERIC characters and/or ""$"" (u0024) characters""

So...

    [\u0041-\u005a\u0061-\u0071\u007f-\uffff\u005f][\u0041-\u005a\u0061-\u0071\u007f-\uffff\u005f\u0030-\u0039\u0024]+","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722084213,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722084213,MDEyOklzc3VlQ29tbWVudDcyMjA4NDIxMw==,9599,simonw,2020-11-05T02:23:37Z,2020-11-05T02:23:37Z,OWNER,"So...

ALPHABETIC: `[\u0041-\u005a\u0061-\u0071\u007f-\uffff\u005f]`

NUMERIC: `[\u0030-\u0039]`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722083527,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722083527,MDEyOklzc3VlQ29tbWVudDcyMjA4MzUyNw==,9599,simonw,2020-11-05T02:21:26Z,2020-11-05T02:21:26Z,OWNER,I think that's `\u007F-\uFFFF` in regex range speak.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722082874,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722082874,MDEyOklzc3VlQ29tbWVudDcyMjA4Mjg3NA==,9599,simonw,2020-11-05T02:19:18Z,2020-11-05T02:19:18Z,OWNER,"""any other character larger than u007f."" Need to figure that out!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722082759,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722082759,MDEyOklzc3VlQ29tbWVudDcyMjA4Mjc1OQ==,9599,simonw,2020-11-05T02:18:58Z,2020-11-05T02:18:58Z,OWNER,"More from that document, describing `ALPHANUMERIC`:

> **ALPHABETIC**
> 
> Any of the characters in the range u0041 through u005a (letters ""A"" through ""Z"") or in the range u0061 through u007a (letters ""a"" through ""z"") or the character u005f (""_"") or any other character larger than u007f.
> 
> **NUMERIC**
> 
> Any of the characters in the range u0030 through u0039 (digits ""0"" through ""9"")
> 
> **ALPHANUMERIC**
> 
> Any character which is either ALPHABETIC or NUMERIC","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722082497,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722082497,MDEyOklzc3VlQ29tbWVudDcyMjA4MjQ5Nw==,9599,simonw,2020-11-05T02:18:08Z,2020-11-05T02:18:08Z,OWNER,"I'm missing the case where a table has no quotes around it at all - `create virtual table foo using fts5`

So I need to know how to create a regex for a SQLite identifier. https://www.sqlite.org/draft/tokenreq.html seems to be the only available documentation for that.

> 
> ### Identifier tokens
> 
> Identifiers follow the usual rules with the exception that SQLite allows the dollar-sign symbol in the interior of an identifier. The dollar-sign is for compatibility with Microsoft SQL-Server and is not part of the SQL standard.
> 
> > **H41130:** SQLite shall recognize as an ID token any sequence of characters that begins with an ALPHABETIC character and continue with zero or more ALPHANUMERIC characters and/or ""$"" (u0024) characters and which is not a keyword token.
> 
> Identifiers can be arbitrary character strings within square brackets. This feature is also for compatibility with Microsoft SQL-Server and not a part of the SQL standard.
> 
> > **H41140:** SQLite shall recognize as an ID token any sequence of non-zero characters that begins with ""["" (u005b) and continuing through the first ""]"" (u005d) character.
> 
> The standard way of quoting SQL identifiers is to use double-quotes.
> 
> > **H41150:** SQLite shall recognize as an ID token any sequence of characters that begins with a double-quote (u0022), is followed by zero or more non-zero characters and/or pairs of double-quotes (u0022) and terminates with a double-quote (u0022) that is not part of a pair.
> 
> MySQL allows identifiers to be quoted using the grave accent character. SQLite supports this for interoperability.
> 
> > **H41160:** SQLite shall recognize as an ID token any sequence of characters that begins with a grave accent (u0060), is followed by zero or more non-zero characters and/or pairs ofgrave accents (u0060) and terminates with a grave accent (u0022) that is not part of a pair.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722078361,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722078361,MDEyOklzc3VlQ29tbWVudDcyMjA3ODM2MQ==,9599,simonw,2020-11-05T02:04:33Z,2020-11-05T02:04:33Z,OWNER,Next step: lots of unit tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722078286,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722078286,MDEyOklzc3VlQ29tbWVudDcyMjA3ODI4Ng==,9599,simonw,2020-11-05T02:04:18Z,2020-11-05T02:04:18Z,OWNER,"I think this might be it:
```python
create_virtual_table_re = re.compile(r""""""
\s*CREATE\s+VIRTUAL\s+TABLE\s+ # CREATE VIRTUAL TABLE
(
   '(?P<squoted_table>[^']*(?:''[^']*)*)' | # single quoted name
   ""(?P<dquoted_table>[^""]*(?:""""[^""]*)*)"" | # double quoted name
   `(?P<backtick_table>[^`]+)` |            # `backtick` quoted name
   \[(?P<squarequoted_table>[^\]]+)\]       # [...] quoted name
)
\s+(IF\s+NOT\s+EXISTS\s+)?     # IF NOT EXISTS (optional)
USING\s+(?P<using>\w+)
"""""", re.VERBOSE | re.IGNORECASE)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722070569,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722070569,MDEyOklzc3VlQ29tbWVudDcyMjA3MDU2OQ==,9599,simonw,2020-11-05T01:38:40Z,2020-11-05T01:38:40Z,OWNER,"I'm going to try `re.VERBOSE` to see if I can make this readable with comments. https://docs.python.org/3/howto/regex.html
```python
charref = re.compile(r""""""
 &[#]                # Start of a numeric entity reference
 (
     0[0-7]+         # Octal form
   | [0-9]+          # Decimal form
   | x[0-9a-fA-F]+   # Hexadecimal form
 )
 ;                   # Trailing semicolon
"""""", re.VERBOSE)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722064258,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722064258,MDEyOklzc3VlQ29tbWVudDcyMjA2NDI1OA==,9599,simonw,2020-11-05T01:18:07Z,2020-11-05T01:21:31Z,OWNER,"```
In [8]: r = re.compile(r""""""'[^']*(?:''[^']*)*'"""""")
In [9]: r.match(""'fo'o'"")
Out[9]: <re.Match object; span=(0, 4), match=""'fo'"">
In [10]: r.match(""'fo''o'"")
Out[10]: <re.Match object; span=(0, 7), match=""'fo''o'"">
```

`'[^']*(?:''[^']*)*'`

This matches a single quote, then 0+ not-single-quotes, then 0+ (either 0+ not-single quotes or a double single quote), then a single quote.

Unrolling the loop technique described here: http://www.softec.lu/site/RegularExpressions/UnrollingTheLoop","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722062449,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722062449,MDEyOklzc3VlQ29tbWVudDcyMjA2MjQ0OQ==,9599,simonw,2020-11-05T01:12:14Z,2020-11-05T01:12:14Z,OWNER,"Good news: I don't think I have to deal with `foo.tablename`, because that doesn't get reflected in the `sqlite_master` table:
```
sqlite> attach 'foo.db' as foo;
sqlite> create table foo.`bant` (id int);
sqlite> select * from foo.sqlite_master;
table|bant|bant|2|CREATE TABLE `bant` (id int)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722062082,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722062082,MDEyOklzc3VlQ29tbWVudDcyMjA2MjA4Mg==,9599,simonw,2020-11-05T01:10:51Z,2020-11-05T01:10:51Z,OWNER,"I confirmed all three of these are valid syntax for creating tables:
```
~ % sqlite3 tmp.db
SQLite version 3.28.0 2019-04-15 14:49:49
Enter "".help"" for usage hints.
sqlite> create table 'foo''and' (id int);
sqlite> create table ""bar""""and"" (id int);
sqlite> create table [baz] (id int);
sqlite> create table `bant` (id int);
sqlite> .schema
CREATE TABLE IF NOT EXISTS 'foo''and' (id int);
CREATE TABLE IF NOT EXISTS ""bar""""and"" (id int);
CREATE TABLE [baz] (id int);
CREATE TABLE `bant` (id int);
sqlite> select * from sqlite_master;
table|foo'and|foo'and|2|CREATE TABLE 'foo''and' (id int)
table|bar""and|bar""and|3|CREATE TABLE ""bar""""and"" (id int)
table|baz|baz|4|CREATE TABLE [baz] (id int)
table|bant|bant|5|CREATE TABLE `bant` (id int)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722058598,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722058598,MDEyOklzc3VlQ29tbWVudDcyMjA1ODU5OA==,9599,simonw,2020-11-05T00:59:58Z,2020-11-05T00:59:58Z,OWNER,"That two-in-a-row thing works for `""` too:

https://latest.datasette.io/fixtures?sql=select+%22foo%22%2C+%27bar%27%2C+%22foo%22%22and%22%2C+%27bar%27%27and%27

<img width=""501"" alt=""fixtures__select__foo____bar____foo__and____bar__and_"" src=""https://user-images.githubusercontent.com/9599/98184381-1da6f580-1ebf-11eb-9396-beb4586d69ca.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722057923,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722057923,MDEyOklzc3VlQ29tbWVudDcyMjA1NzkyMw==,9599,simonw,2020-11-05T00:57:22Z,2020-11-05T00:57:22Z,OWNER,"Then https://sqlite.org/lang_expr.html#literal_values_constants_ says:

>  A string constant is formed by enclosing the string in single quotes ('). A single quote within the string can be encoded by putting two single quotes in a row - as in Pascal. C-style escapes using the backslash character are not supported because they are not standard SQL. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722057392,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722057392,MDEyOklzc3VlQ29tbWVudDcyMjA1NzM5Mg==,9599,simonw,2020-11-05T00:55:31Z,2020-11-05T00:55:51Z,OWNER,"https://sqlite.org/lang_keywords.html says:

> There are four ways of quoting keywords in SQLite:
>
> **'keyword'** A keyword in single quotes is a string literal.
> **""keyword""** A keyword in double-quotes is an identifier.
> **[keyword]** A keyword enclosed in square brackets is an identifier. This is not standard SQL. This quoting mechanism is used by MS Access and SQL Server and is included in SQLite for compatibility.
> **\`keyword\`** A keyword enclosed in grave accents (ASCII code 96) is an identifier. This is not standard SQL. This quoting mechanism is used by MySQL and is included in SQLite for compatibility.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722056576,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722056576,MDEyOklzc3VlQ29tbWVudDcyMjA1NjU3Ng==,9599,simonw,2020-11-05T00:52:42Z,2020-11-05T00:52:42Z,OWNER,"I could use a parsing library like https://parsy.readthedocs.io/en/latest/tutorial.html for this - or `pyparsing` which has a SQLite example here: https://github.com/pyparsing/pyparsing/blob/master/examples/select_parser.py

I'd rather not add a new dependency for this though so I'm going to see if I can get something that's good-enough just using a regular expression.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722055291,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722055291,MDEyOklzc3VlQ29tbWVudDcyMjA1NTI5MQ==,9599,simonw,2020-11-05T00:48:10Z,2020-11-05T00:48:10Z,OWNER,This is blocking landing `.search()` in #195,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722055104,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722055104,MDEyOklzc3VlQ29tbWVudDcyMjA1NTEwNA==,9599,simonw,2020-11-05T00:47:34Z,2020-11-05T00:47:34Z,OWNER,"This is surprisingly difficult. I need to parse the `CREATE VIRTUAL TABLE` statement, which will look something like this:

```sql
CREATE VIRTUAL TABLE ""global-power-plants_fts"" USING FTS5 (""name"", content=""global-power-plants"")
```

The problem is I need to be able to handle various different quoting formats for the table name (`mytable` v.s. `""mytable""` v.s. `[mytable]`) plus I need to look out for `CREATE TABLE IF NOT EXISTS`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5,
https://github.com/simonw/sqlite-utils/issues/192#issuecomment-722054264,https://api.github.com/repos/simonw/sqlite-utils/issues/192,722054264,MDEyOklzc3VlQ29tbWVudDcyMjA1NDI2NA==,9599,simonw,2020-11-05T00:44:39Z,2020-11-05T00:44:39Z,OWNER,"I want `.search()` to work against both FTS5 and FTS4 tables - but sort by rank should only work for FTS5.

This means I need to be able to introspect and tell if a table is FTS4 or FTS5.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command,
https://github.com/simonw/datasette/issues/1082#issuecomment-721931504,https://api.github.com/repos/simonw/datasette/issues/1082,721931504,MDEyOklzc3VlQ29tbWVudDcyMTkzMTUwNA==,9599,simonw,2020-11-04T19:32:47Z,2020-11-04T19:35:44Z,OWNER,"I wonder if setting a soft memory limit within Datasette would help here: https://www.sqlite.org/malloc.html#_setting_memory_usage_limits 

>  If attempts are made to allocate more memory than specified by the soft heap limit, then SQLite will first attempt to free cache memory before continuing with the allocation request.

https://www.sqlite.org/pragma.html#pragma_soft_heap_limit

> **PRAGMA soft_heap_limit**
> **PRAGMA soft_heap_limit=N**
> 
> This pragma invokes the [sqlite3_soft_heap_limit64()](https://www.sqlite.org/c3ref/hard_heap_limit64.html) interface with the argument N, if N is specified and is a non-negative integer. The soft_heap_limit pragma always returns the same integer that would be returned by the [sqlite3_soft_heap_limit64](https://www.sqlite.org/c3ref/hard_heap_limit64.html)(-1) C-language function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735852274,DigitalOcean buildpack memory errors for large sqlite db?,
https://github.com/simonw/datasette/issues/1083#issuecomment-721927254,https://api.github.com/repos/simonw/datasette/issues/1083,721927254,MDEyOklzc3VlQ29tbWVudDcyMTkyNzI1NA==,9599,simonw,2020-11-04T19:24:34Z,2020-11-04T19:24:34Z,OWNER,"Related: #856 - if it's possible to paginate correctly configured canned query then the CSV option to ""stream all rows"" could work for queries as well as tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736365306,Advanced CSV export for arbitrary queries,
https://github.com/simonw/datasette/issues/1083#issuecomment-721926827,https://api.github.com/repos/simonw/datasette/issues/1083,721926827,MDEyOklzc3VlQ29tbWVudDcyMTkyNjgyNw==,9599,simonw,2020-11-04T19:23:42Z,2020-11-04T19:23:42Z,OWNER,"https://latest.datasette.io/fixtures/sortable#export has advanced export options, but https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+order+by+pk1%2C+pk2+limit+101 does not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736365306,Advanced CSV export for arbitrary queries,
https://github.com/simonw/datasette/issues/268#issuecomment-721896822,https://api.github.com/repos/simonw/datasette/issues/268,721896822,MDEyOklzc3VlQ29tbWVudDcyMTg5NjgyMg==,9599,simonw,2020-11-04T18:23:29Z,2020-11-04T18:23:29Z,OWNER,"Worth noting that joining to get the rank works for FTS5 but not for FTS4 - see comment here: https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721420539

Easiest solution would be to only support sort-by-rank for FTS5 tables. Alternative would be to depend on https://github.com/simonw/sqlite-fts4","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/1082#issuecomment-721545090,https://api.github.com/repos/simonw/datasette/issues/1082,721545090,MDEyOklzc3VlQ29tbWVudDcyMTU0NTA5MA==,9599,simonw,2020-11-04T06:47:15Z,2020-11-04T06:47:15Z,OWNER,"I've run into a similar problem with Google Cloud Run: beyond a certain size of database file I find myself needing to run instances there with more RAM assigned to them.

I haven't yet figured out a method to estimate the amount of RAM that will be needed to successfully serve a database file of a specific size- I've been using trial and error.

5GB is quite a big database file, so it doesn't surprise me that it may need a bigger instance. I recommend trying it on a 1GB or 2GB of RAM Digital Ocean instance (their default is 512MB) and see if that works.

Let me know what you find out!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735852274,DigitalOcean buildpack memory errors for large sqlite db?,
https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721420907,https://api.github.com/repos/simonw/sqlite-utils/issues/192,721420907,MDEyOklzc3VlQ29tbWVudDcyMTQyMDkwNw==,9599,simonw,2020-11-03T23:07:01Z,2020-11-03T23:07:01Z,OWNER,"I could depend on my `sqlite-fts4` library to solve this: https://github.com/simonw/sqlite-fts4

Or I could say that only `FTS5` is supported for ranked searches - but still support `.search()` against FTS4 just without the option to sort by relevance.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command,
https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721420539,https://api.github.com/repos/simonw/sqlite-utils/issues/192,721420539,MDEyOklzc3VlQ29tbWVudDcyMTQyMDUzOQ==,9599,simonw,2020-11-03T23:05:53Z,2020-11-03T23:05:53Z,OWNER,"Just realized there's a problem with the SQL I am using here: joining to get `rank` in this way only works against FTS5 tables, it doesn't work against FTS4.

 https://github.com/simonw/sqlite-utils/blob/c8a900df59efd34f394c863c0adff9912f1bf1d7/sqlite_utils/db.py#L1303-L1319","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command,
https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721453779,https://api.github.com/repos/simonw/sqlite-utils/issues/192,721453779,MDEyOklzc3VlQ29tbWVudDcyMTQ1Mzc3OQ==,9599,simonw,2020-11-04T00:59:24Z,2020-11-04T00:59:36Z,OWNER,"FTS5 was added in SQLite 3.9.0 in 2015-10-14 - so about a year after CTEs, which means CTEs will always be safe to use with FTS5 queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command,
https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721319602,https://api.github.com/repos/simonw/sqlite-utils/issues/192,721319602,MDEyOklzc3VlQ29tbWVudDcyMTMxOTYwMg==,9599,simonw,2020-11-03T19:05:05Z,2020-11-03T19:05:05Z,OWNER,"Relevant example using a SQLite CTE: https://github.com/simonw/datasette/issues/268#issuecomment-675725464

CTEs were added in SQLite 3.8.3 in 2014-02-03 so they should be safe to use. If someone tries to run `sqlite-utils search` no an older version of SQLite they'll get an error, which I think is OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command,
https://github.com/simonw/sqlite-utils/pull/195#issuecomment-721397665,https://api.github.com/repos/simonw/sqlite-utils/issues/195,721397665,MDEyOklzc3VlQ29tbWVudDcyMTM5NzY2NQ==,9599,simonw,2020-11-03T22:02:57Z,2020-11-03T22:02:57Z,OWNER,Documentation so far: https://github.com/simonw/sqlite-utils/blob/6cadc6103ff1ba58c6409ce7fba74259e72965d9/docs/cli.rst#executing-searches,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735663855,table.search() improvements plus sqlite-utils search command,
https://github.com/simonw/datasette/issues/830#issuecomment-720700065,https://api.github.com/repos/simonw/datasette/issues/830,720700065,MDEyOklzc3VlQ29tbWVudDcyMDcwMDA2NQ==,9599,simonw,2020-11-02T20:15:36Z,2020-11-02T20:15:36Z,OWNER,"#427 had a bunch of ambitious plans for faceting that I haven't realized yet:

> Think of all of the potential kinds of facets:
> 
> * `?_facet_array=tags` where tags is a JSON array of values
> * `_facet_date=datetimecol` - faceted by date part of a datetime
> * `_facet_bins=numeric_column` - can I do some kind of fancy binning here? Might need to take an argument
> * `?_facet_bins=numeric_column:5` - could be a way to take an argument. We’ll ignore columns with a : in their name.
> * `?_facet_json=jsoncol:jsonpath` - could use a JSON path to extract out something to facet on?
> * `?_facet_percentile=numericcolumn` - could this work?
> * `?_facet_function=column:sqlfunctionname` - maybe this could be interesting? Would allow for e.g. facet by soundex
> * `?_facet_prefix=column:prefix` - facet by terms but only if they start with a specific prefix
> * `?_facet_substring=column:3,6` - facet by a substr(column, 3, 6)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636511683,Redesign register_facet_classes plugin hook,
https://github.com/simonw/datasette/issues/1077#issuecomment-720637322,https://api.github.com/repos/simonw/datasette/issues/1077,720637322,MDEyOklzc3VlQ29tbWVudDcyMDYzNzMyMg==,9599,simonw,2020-11-02T18:09:17Z,2020-11-02T18:09:17Z,OWNER,Here's the `table_actions` implementation: 2f7731e9e5ff9b324beb5039fbe2be55d704a184,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733829385,database_actions plugin hook,
https://github.com/simonw/datasette/issues/1080#issuecomment-720699160,https://api.github.com/repos/simonw/datasette/issues/1080,720699160,MDEyOklzc3VlQ29tbWVudDcyMDY5OTE2MA==,9599,simonw,2020-11-02T20:13:42Z,2020-11-02T20:13:42Z,OWNER,Also relevant to this issue: #830 - redesigning the facet plugin hook in preparation for Datasette 1.0. And #972 supporting faceting against arbitrary queries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",
https://github.com/simonw/datasette/issues/1080#issuecomment-720698577,https://api.github.com/repos/simonw/datasette/issues/1080,720698577,MDEyOklzc3VlQ29tbWVudDcyMDY5ODU3Nw==,9599,simonw,2020-11-02T20:12:26Z,2020-11-02T20:12:26Z,OWNER,"For regular column faceting, here's the query that is used:

https://github.com/simonw/datasette/blob/13d1228d80c91d382a05b1a9549ed02c300ef851/datasette/facets.py#L196-L204

Since it uses `order by count desc, value` maybe those values could be used to implement cursor-based pagination.

That wouldn't be robust in the face of changing data, but I'm not sure it's possible to implement paginated faceting in a way that survives ongoing changes to the underlying data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",
https://github.com/simonw/datasette/issues/1080#issuecomment-720697226,https://api.github.com/repos/simonw/datasette/issues/1080,720697226,MDEyOklzc3VlQ29tbWVudDcyMDY5NzIyNg==,9599,simonw,2020-11-02T20:09:38Z,2020-11-02T20:09:38Z,OWNER,"Maybe this ends up being code that defers to a simulated canned query, rendered using the existing `query.html` template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",
https://github.com/simonw/datasette/issues/1080#issuecomment-720696827,https://api.github.com/repos/simonw/datasette/issues/1080,720696827,MDEyOklzc3VlQ29tbWVudDcyMDY5NjgyNw==,9599,simonw,2020-11-02T20:08:49Z,2020-11-02T20:13:56Z,OWNER,"Implementing pagination for facets will be interesting. Would be easier if I had a nicer reusable internal pagination mechanism, which is also needed for #856 (pagination of canned queries).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",
https://github.com/simonw/datasette/issues/1080#issuecomment-720695174,https://api.github.com/repos/simonw/datasette/issues/1080,720695174,MDEyOklzc3VlQ29tbWVudDcyMDY5NTE3NA==,9599,simonw,2020-11-02T20:05:26Z,2020-11-02T20:05:26Z,OWNER,"URL design:

`/database/table/-/facet/colname`

And for other types of facet (to be supported later):

`/database/table/-/facet/colname?_type=m2m`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them",
https://github.com/simonw/datasette/issues/596#issuecomment-720689653,https://api.github.com/repos/simonw/datasette/issues/596,720689653,MDEyOklzc3VlQ29tbWVudDcyMDY4OTY1Mw==,9599,simonw,2020-11-02T19:53:36Z,2020-11-02T19:53:47Z,OWNER,"In #998 I implemented a horizontal scrollbar for these tables, which is a big improvement - demo here: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",507454958,Handle really wide tables better,
https://github.com/simonw/datasette/issues/1077#issuecomment-720654925,https://api.github.com/repos/simonw/datasette/issues/1077,720654925,MDEyOklzc3VlQ29tbWVudDcyMDY1NDkyNQ==,9599,simonw,2020-11-02T18:43:25Z,2020-11-02T18:43:25Z,OWNER,Demo: https://latest.datasette.io/fixtures?_bot=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733829385,database_actions plugin hook,
https://github.com/simonw/datasette/issues/1079#issuecomment-720110298,https://api.github.com/repos/simonw/datasette/issues/1079,720110298,MDEyOklzc3VlQ29tbWVudDcyMDExMDI5OA==,9599,simonw,2020-11-01T15:58:22Z,2020-11-01T15:58:22Z,OWNER,Might try a drop shadow on that menu too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733999615,Handle long breadcrumbs better with new menu,
https://github.com/simonw/datasette/issues/782#issuecomment-720028476,https://api.github.com/repos/simonw/datasette/issues/782,720028476,MDEyOklzc3VlQ29tbWVudDcyMDAyODQ3Ng==,9599,simonw,2020-11-01T05:00:05Z,2020-11-01T05:00:05Z,OWNER,This should be the key focus for Datasette 0.52.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/949#issuecomment-720021029,https://api.github.com/repos/simonw/datasette/issues/949,720021029,MDEyOklzc3VlQ29tbWVudDcyMDAyMTAyOQ==,9599,simonw,2020-11-01T03:29:48Z,2020-11-01T03:29:48Z,OWNER,I'm not going to do any more work on this - SQL isn't an auto-complete friendly enough language.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684961449,Try out CodeMirror SQL hints,
https://github.com/simonw/datasette/issues/1077#issuecomment-720003026,https://api.github.com/repos/simonw/datasette/issues/1077,720003026,MDEyOklzc3VlQ29tbWVudDcyMDAwMzAyNg==,9599,simonw,2020-10-31T23:48:21Z,2020-10-31T23:50:07Z,OWNER,Needed by https://github.com/simonw/datasette-backup/issues/6,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733829385,database_actions plugin hook,
https://github.com/simonw/datasette/issues/1048#issuecomment-719996693,https://api.github.com/repos/simonw/datasette/issues/1048,719996693,MDEyOklzc3VlQ29tbWVudDcxOTk5NjY5Mw==,9599,simonw,2020-10-31T22:32:09Z,2022-07-10T16:22:48Z,OWNER,"The `row_path` part of these really isn't very user friendly, since you need to properly URL-encode the values. The safest way to do so is by calling this obscure, undocumented utility function: https://github.com/simonw/datasette/blob/f0bd2d05f5f7832df4879822afb99d2096c00d48/datasette/utils/__init__.py#L84-L98

This feels like it should be improved before I turn it into a documented API.

(Note that this API deals with a `row` that is a potentially-nested dictionary - not with a `sqlite3.Row` object.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728905098,Documentation and unit tests for urls.row() urls.row_blob() methods,
https://github.com/simonw/datasette/issues/1047#issuecomment-719994676,https://api.github.com/repos/simonw/datasette/issues/1047,719994676,MDEyOklzc3VlQ29tbWVudDcxOTk5NDY3Ng==,9599,simonw,2020-10-31T22:11:25Z,2020-10-31T22:11:25Z,OWNER,https://docs.datasette.io/en/latest/binary_data.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728895233,A new section in the docs about how Datasette handles BLOB columns,
https://github.com/simonw/datasette/issues/1027#issuecomment-719988113,https://api.github.com/repos/simonw/datasette/issues/1027,719988113,MDEyOklzc3VlQ29tbWVudDcxOTk4ODExMw==,9599,simonw,2020-10-31T21:03:37Z,2020-10-31T21:03:37Z,OWNER,"On my Mac, I run:

    datasette . -p 8009 --config base_url:/datasette-prefix/

Then I edited `/usr/local/etc/httpd/httpd.conf` and add this section:
```
LoadModule proxy_module lib/httpd/modules/mod_proxy.so
LoadModule proxy_http_module lib/httpd/modules/mod_proxy_http.so
ProxyPass          /datasette-prefix/    http://localhost:8009/datasette-prefix/
```
I ran Apache in the foreground like so:

    apachectl -X

Now hitting http://localhost:8081/datasette-prefix/fixtures/compound_three_primary_keys worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/1023#issuecomment-719986922,https://api.github.com/repos/simonw/datasette/issues/1023,719986922,MDEyOklzc3VlQ29tbWVudDcxOTk4NjkyMg==,9599,simonw,2020-10-31T20:51:01Z,2020-10-31T20:51:01Z,OWNER,This should all be working correctly now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url,
https://github.com/simonw/datasette/issues/838#issuecomment-719986904,https://api.github.com/repos/simonw/datasette/issues/838,719986904,MDEyOklzc3VlQ29tbWVudDcxOTk4NjkwNA==,9599,simonw,2020-10-31T20:50:41Z,2020-10-31T20:50:41Z,OWNER,"OK, this should be working now. You can use the `datasette.urls.static_plugins()` method to generate the correct URLs in the `extra_css_urls` plugin hook: https://docs.datasette.io/en/latest/internals.html#datasette-urls","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/1041#issuecomment-719986800,https://api.github.com/repos/simonw/datasette/issues/1041,719986800,MDEyOklzc3VlQ29tbWVudDcxOTk4NjgwMA==,9599,simonw,2020-10-31T20:49:28Z,2020-10-31T20:49:28Z,OWNER,Implemented in a4ca26a2659d21779adf625183061d8879954c15,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting,
https://github.com/simonw/datasette/issues/1072#issuecomment-719986698,https://api.github.com/repos/simonw/datasette/issues/1072,719986698,MDEyOklzc3VlQ29tbWVudDcxOTk4NjY5OA==,9599,simonw,2020-10-31T20:48:17Z,2020-10-31T20:48:17Z,OWNER,Here's the `datasette-edit-templates` plugin WIP I had before removing the hook: https://github.com/simonw/datasette-edit-templates/tree/82855c2612b84bc09c48fca885f831633a0d1552,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1075#issuecomment-719983750,https://api.github.com/repos/simonw/datasette/issues/1075,719983750,MDEyOklzc3VlQ29tbWVudDcxOTk4Mzc1MA==,9599,simonw,2020-10-31T20:22:29Z,2020-10-31T20:22:29Z,OWNER,I bet this is because I'm mucking around with one of those `__` methods. I'll try just doing the non-underscore methods instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719983565,https://api.github.com/repos/simonw/datasette/issues/1075,719983565,MDEyOklzc3VlQ29tbWVudDcxOTk4MzU2NQ==,9599,simonw,2020-10-31T20:21:03Z,2020-10-31T20:21:03Z,OWNER,"Here's the output of `dir(str)`:

`['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'capitalize', 'casefold', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index', 'isalnum', 'isalpha', 'isascii', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719983484,https://api.github.com/repos/simonw/datasette/issues/1075,719983484,MDEyOklzc3VlQ29tbWVudDcxOTk4MzQ4NA==,9599,simonw,2020-10-31T20:20:28Z,2020-10-31T20:20:28Z,OWNER,"It looks like this is specific to the way `PrefixedUrlString` is built.
```
(Pdb) class Weird(str): pass
(Pdb) isinstance(Weird('bob'), collections.abc.Awaitable)
False
```
So subclassing strings doesn't trigger this bug, but something about `PrefixedUrlString` causes the problem.

Here's the current `PrefixedUrlString` implementation:

https://github.com/simonw/datasette/blob/bf18b9ba175a7b25fb8b765847397dd6efb8bb7b/datasette/utils/__init__.py#L1015-L1035","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719983240,https://api.github.com/repos/simonw/datasette/issues/1075,719983240,MDEyOklzc3VlQ29tbWVudDcxOTk4MzI0MA==,9599,simonw,2020-10-31T20:18:49Z,2020-10-31T20:18:49Z,OWNER,"Here's the core problem:
```
(Pdb) isinstance('bob', collections.abc.Awaitable)
False
(Pdb) isinstance(PrefixedUrlString('bob'), collections.abc.Awaitable)
*** TypeError: issubclass() arg 1 must be a class
```
For some reason `isinstance()` does not like being handed an instance of PrefixedUrlString.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719981173,https://api.github.com/repos/simonw/datasette/issues/1075,719981173,MDEyOklzc3VlQ29tbWVudDcxOTk4MTE3Mw==,9599,simonw,2020-10-31T20:02:30Z,2020-10-31T20:03:45Z,OWNER,"I wonder how Jinja's `Markup()` class works? It uses https://pypi.org/project/MarkupSafe/

It's a subclass of `str`, defined here: https://github.com/pallets/markupsafe/blob/master/src/markupsafe/__init__.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719980742,https://api.github.com/repos/simonw/datasette/issues/1075,719980742,MDEyOklzc3VlQ29tbWVudDcxOTk4MDc0Mg==,9599,simonw,2020-10-31T19:58:57Z,2020-10-31T19:58:57Z,OWNER,"Sample traceback:
```
    <link rel=""stylesheet"" href=""{{ urls.static('app.css') }}?{{ app_css_hash }}"">
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/jinja2/asyncsupport.py:173: in auto_await
    if inspect.isawaitable(value):
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/inspect.py:226: in isawaitable
    isinstance(object, collections.abc.Awaitable))
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/abc.py:139: in __instancecheck__
    return _abc_instancecheck(cls, instance)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'collections.abc.Awaitable'>
subclass = <bound method PrefixedUrlString.__getattribute__.<locals>.method of '/-/static/app.css'>

    def __subclasscheck__(cls, subclass):
        """"""Override for issubclass(subclass, cls).""""""
>       return _abc_subclasscheck(cls, subclass)
E       TypeError: issubclass() arg 1 must be a class
```
This is within Jinja. It looks like Jinja really doesn't like methods that return non-string objects like `PrefixedUrlString`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1074#issuecomment-719977864,https://api.github.com/repos/simonw/datasette/issues/1074,719977864,MDEyOklzc3VlQ29tbWVudDcxOTk3Nzg2NA==,9599,simonw,2020-10-31T19:35:01Z,2020-10-31T19:35:01Z,OWNER,"These plugins were not designed to be actually hosted online, so they do some nasty things like linking to the made-up `plugin-example.com` domain. I should fix that.

<img width=""853"" alt=""Datasette_Fixtures__fixtures"" src=""https://user-images.githubusercontent.com/9599/97788348-7b60d800-1b75-11eb-8519-a58e52108841.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733768037,latest.datasette.io should include plugins from fixtures,
https://github.com/simonw/datasette/issues/1067#issuecomment-719966176,https://api.github.com/repos/simonw/datasette/issues/1067,719966176,MDEyOklzc3VlQ29tbWVudDcxOTk2NjE3Ng==,9599,simonw,2020-10-31T17:51:31Z,2020-10-31T17:51:31Z,OWNER,"Demo:

- https://latest.datasette.io/fixtures/facetable?_bot=1
- https://latest.datasette.io/fixtures/simple_view?_bot=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1074#issuecomment-719965426,https://api.github.com/repos/simonw/datasette/issues/1074,719965426,MDEyOklzc3VlQ29tbWVudDcxOTk2NTQyNg==,9599,simonw,2020-10-31T17:45:00Z,2020-10-31T17:45:00Z,OWNER,"This is working. Go to https://latest.datasette.io/login-as-root and click the button, then visit this page to see extra content added by plugins: https://latest.datasette.io/fixtures/compound_three_primary_keys

![latest-plugins](https://user-images.githubusercontent.com/9599/97786052-19e53d00-1b66-11eb-8268-b452e08965e3.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733768037,latest.datasette.io should include plugins from fixtures,
https://github.com/simonw/datasette/issues/1074#issuecomment-719963074,https://api.github.com/repos/simonw/datasette/issues/1074,719963074,MDEyOklzc3VlQ29tbWVudDcxOTk2MzA3NA==,9599,simonw,2020-10-31T17:23:48Z,2020-10-31T17:23:48Z,OWNER,"Needs a way to login as root, seeing as several plugins only show extra content if the user is logged in as root.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733768037,latest.datasette.io should include plugins from fixtures,
https://github.com/simonw/datasette/issues/1067#issuecomment-719961701,https://api.github.com/repos/simonw/datasette/issues/1067,719961701,MDEyOklzc3VlQ29tbWVudDcxOTk2MTcwMQ==,9599,simonw,2020-10-31T17:11:59Z,2020-10-31T17:11:59Z,OWNER,It bothers me that these aren't visible in any public demos. Maybe `latest.datasette.io` should include the `my_plugins.py` and `my_plugins2.py` plugins?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1026#issuecomment-719959754,https://api.github.com/repos/simonw/datasette/issues/1026,719959754,MDEyOklzc3VlQ29tbWVudDcxOTk1OTc1NA==,9599,simonw,2020-10-31T16:56:35Z,2020-10-31T16:56:35Z,OWNER,#1041 can also benefit from the string subclass that shows that `base_url` has been added.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1067#issuecomment-719959419,https://api.github.com/repos/simonw/datasette/issues/1067,719959419,MDEyOklzc3VlQ29tbWVudDcxOTk1OTQxOQ==,9599,simonw,2020-10-31T16:53:42Z,2020-10-31T16:53:42Z,OWNER,For the 0.51 release I'm going to add tests that show this works on view pages. I won't implement it for query pages.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1067#issuecomment-719956184,https://api.github.com/repos/simonw/datasette/issues/1067,719956184,MDEyOklzc3VlQ29tbWVudDcxOTk1NjE4NA==,9599,simonw,2020-10-31T16:26:09Z,2020-10-31T16:26:09Z,OWNER,"Should the hook provide an indication that it's running on a different type of page? I think yes for queries. Not sure about views  - they behave very much like tables, and the plugin can always introspect to see if something is a view if it needs to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1070#issuecomment-719955724,https://api.github.com/repos/simonw/datasette/issues/1070,719955724,MDEyOklzc3VlQ29tbWVudDcxOTk1NTcyNA==,9599,simonw,2020-10-31T16:22:45Z,2020-10-31T16:22:45Z,OWNER,I've removed this plugin hook in #1073.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733390884,load_template() example in documentation showing loading from a database,
https://github.com/simonw/datasette/issues/1072#issuecomment-719955491,https://api.github.com/repos/simonw/datasette/issues/1072,719955491,MDEyOklzc3VlQ29tbWVudDcxOTk1NTQ5MQ==,9599,simonw,2020-10-31T16:20:58Z,2020-10-31T16:20:58Z,OWNER,"Here's the proof of concept `FunctionLoader` that showed me that this wasn't going to work:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 4b28e71..b076be7 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -21,7 +21,7 @@ from pathlib import Path
 from markupsafe import Markup
 from itsdangerous import URLSafeSerializer
 import jinja2
-from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
+from jinja2 import ChoiceLoader, Environment, FileSystemLoader, FunctionLoader, PrefixLoader
 from jinja2.environment import Template
 from jinja2.exceptions import TemplateNotFound
 import uvicorn
@@ -300,6 +300,7 @@ class Datasette:
         template_paths.append(default_templates)
         template_loader = ChoiceLoader(
             [
+                FunctionLoader(self._load_template_from_plugins),
                 FileSystemLoader(template_paths),
                 # Support {% extends ""default:table.html"" %}:
                 PrefixLoader(
@@ -322,6 +323,17 @@ class Datasette:
         self._root_token = secrets.token_hex(32)
         self.client = DatasetteClient(self)
 
+    def _load_template_from_plugins(self, template):
+        # ""If auto reloading is enabled it’s called to check if the template changed""
+        uptodatefunc = lambda: True
+        source = pm.hook.load_template(
+            template=template,
+            datasette=self,
+        )
+        if source is None:
+            return None
+        return source, template, uptodatefunc
+
     @property
     def urls(self):
         return Urls(self)
@@ -719,35 +731,7 @@ class Datasette:
         else:
             if isinstance(templates, str):
                 templates = [templates]
-
-            # Give plugins first chance at loading the template
-            break_outer = False
-            plugin_template_source = None
-            plugin_template_name = None
-            template_name = None
-            for template_name in templates:
-                if break_outer:
-                    break
-                plugin_template_source = pm.hook.load_template(
-                    template=template_name,
-                    request=request,
-                    datasette=self,
-                )
-                plugin_template_source = await await_me_maybe(plugin_template_source)
-                if plugin_template_source:
-                    break_outer = True
-                    plugin_template_name = template_name
-                    break
-            if plugin_template_source is not None:
-                template = self.jinja_env.from_string(plugin_template_source)
-            else:
-                template = self.jinja_env.select_template(templates)
-            for template_name in templates:
-                from_plugin = template_name == plugin_template_name
-                used = from_plugin or template_name == template.name
-                templates_considered.append(
-                    {""name"": template_name, ""used"": used, ""from_plugin"": from_plugin}
-                )
+            template = self.jinja_env.select_template(templates)
         body_scripts = []
         # pylint: disable=no-member
         for extra_script in pm.hook.extra_body_script(
diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py
index ca84b35..7804def 100644
--- a/datasette/hookspecs.py
+++ b/datasette/hookspecs.py
@@ -50,7 +50,7 @@ def extra_template_vars(
 
 
 @hookspec(firstresult=True)
-def load_template(template, request, datasette):
+def load_template(template, datasette):
     ""Load the specified template, returning the template code as a string""
 
 
diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst
index 3c57b6a..8f2704e 100644
--- a/docs/plugin_hooks.rst
+++ b/docs/plugin_hooks.rst
@@ -273,15 +273,12 @@ Example: `datasette-cluster-map <https://github.com/simonw/datasette-cluster-map
 
 .. _plugin_hook_load_template:
 
-load_template(template, request, datasette)
--------------------------------------------
+load_template(template, datasette)
+----------------------------------
 
 ``template`` - string
     The template that is being rendered, e.g. ``database.html``
 
-``request`` - object or None
-    The current HTTP :ref:`internals_request`. This can be ``None`` if the request object is not available.
-
 ``datasette`` - :ref:`internals_datasette`
     You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``
 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1073#issuecomment-719834200,https://api.github.com/repos/simonw/datasette/issues/1073,719834200,MDEyOklzc3VlQ29tbWVudDcxOTgzNDIwMA==,9599,simonw,2020-10-30T22:52:48Z,2020-10-30T22:52:48Z,OWNER,Should mostly be a case of backing out the changes from this commit: 81dea4b07ab2b6f4eaaf248307d2b588472054a1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733560417,Remove load_template plugin hook,
https://github.com/simonw/datasette/issues/1072#issuecomment-719833744,https://api.github.com/repos/simonw/datasette/issues/1072,719833744,MDEyOklzc3VlQ29tbWVudDcxOTgzMzc0NA==,9599,simonw,2020-10-30T22:50:57Z,2020-10-30T22:50:57Z,OWNER,Yeah I'm going to remove the `load_template` plugin hook and see if it's possible to build the edit templates extension against `prepare_jinja2_environment` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719833070,https://api.github.com/repos/simonw/datasette/issues/1072,719833070,MDEyOklzc3VlQ29tbWVudDcxOTgzMzA3MA==,9599,simonw,2020-10-30T22:48:04Z,2020-10-30T22:48:04Z,OWNER,"https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L310-L318

So yeah that plugin hook can probably modify the list of loaders available to the `Environment`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719832853,https://api.github.com/repos/simonw/datasette/issues/1072,719832853,MDEyOklzc3VlQ29tbWVudDcxOTgzMjg1Mw==,9599,simonw,2020-10-30T22:47:12Z,2020-10-30T22:47:12Z,OWNER,Maybe I should ditch this hook entirely in favour of the existing `prepare_jinja2_environment` hook. Could that add new template loaders?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719832651,https://api.github.com/repos/simonw/datasette/issues/1072,719832651,MDEyOklzc3VlQ29tbWVudDcxOTgzMjY1MQ==,9599,simonw,2020-10-30T22:46:25Z,2020-10-30T22:46:25Z,OWNER,"I tried using a `FunctionLoader` and got this error on startup:
```python
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 989, in __init__
    for filepath in self.ds.jinja_env.list_templates()
  File ""/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/environment.py"", line 810, in list_templates
    names = self.loader.list_templates()
  File ""/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/loaders.py"", line 434, in list_templates
    found.update(loader.list_templates())
  File ""/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/loaders.py"", line 99, in list_templates
    raise TypeError(""this loader cannot iterate over all templates"")
TypeError: this loader cannot iterate over all templates
```
So if I'm going to define a custom Jinja loader I'll need to teach plugins to answer the ""list templates"" query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719819331,https://api.github.com/repos/simonw/datasette/issues/1072,719819331,MDEyOklzc3VlQ29tbWVudDcxOTgxOTMzMQ==,9599,simonw,2020-10-30T22:00:43Z,2020-10-30T22:00:43Z,OWNER,I'll try getting that to work. If I can't get it to work I'll drop the plugin hook for the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719819234,https://api.github.com/repos/simonw/datasette/issues/1072,719819234,MDEyOklzc3VlQ29tbWVudDcxOTgxOTIzNA==,9599,simonw,2020-10-30T22:00:21Z,2020-10-30T22:00:21Z,OWNER,"There might be a way to save this. Async template loading can't be supported, but what if you could define a `load_template()` hook which returned a sync function that returned templates...

Then the `datasette-edit-templates` plugin could reply to `load_template` by loading all DB templates into memory and returning a `load_template` sync function that looked up the values in those already-loaded templates.

It could even maintain an in-memory cache that gets updated when a template is edited.

If I do this, I could remove the ability to return an `async` function from `load_template()` but add that in the future should Jinja implement a mechanism for async template loading.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719814279,https://api.github.com/repos/simonw/datasette/issues/1072,719814279,MDEyOklzc3VlQ29tbWVudDcxOTgxNDI3OQ==,9599,simonw,2020-10-30T21:45:33Z,2020-10-30T21:45:33Z,OWNER,Sadly I'm going to bump `load_template` from Datasette 0.51 - I don't think I should block the release on resolving this issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719813970,https://api.github.com/repos/simonw/datasette/issues/1072,719813970,MDEyOklzc3VlQ29tbWVudDcxOTgxMzk3MA==,9599,simonw,2020-10-30T21:44:40Z,2020-10-30T21:44:40Z,OWNER,"I'm pretty sure that `run_in_executor()` workaround won't work. https://github.com/django/asgiref/blob/7becc9daca2628c46af1cb7e46b4c47c1ea27adf/asgiref/sync.py#L83 for example says ""You cannot use AsyncToSync in the same thread as an async event loop"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719813212,https://api.github.com/repos/simonw/datasette/issues/1072,719813212,MDEyOklzc3VlQ29tbWVudDcxOTgxMzIxMg==,9599,simonw,2020-10-30T21:42:35Z,2020-10-30T21:42:35Z,OWNER,Filed a feature request here: https://github.com/pallets/jinja/issues/1304,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719811312,https://api.github.com/repos/simonw/datasette/issues/1072,719811312,MDEyOklzc3VlQ29tbWVudDcxOTgxMTMxMg==,9599,simonw,2020-10-30T21:36:49Z,2020-10-30T21:36:49Z,OWNER,"There's one other option: in `datasette-edit-templates` I could maybe use `asyncio.get_event_loop().run_in_executor(...)` to load the templates asynchronously within the Jinja template loader mechanism.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719810533,https://api.github.com/repos/simonw/datasette/issues/1072,719810533,MDEyOklzc3VlQ29tbWVudDcxOTgxMDUzMw==,9599,simonw,2020-10-30T21:34:38Z,2020-10-30T21:34:38Z,OWNER,"... no wait, my comments above assume that I'm just building the `datasette-edit-templates` plugin. Does this work as a general solution for all of Datasette? I don't think it does.

This may mean I need to delay the whole feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719810023,https://api.github.com/repos/simonw/datasette/issues/1072,719810023,MDEyOklzc3VlQ29tbWVudDcxOTgxMDAyMw==,9599,simonw,2020-10-30T21:33:06Z,2020-10-30T21:33:06Z,OWNER,"The ideal solution is for Jinja to offer `async` template loading. I'll file a feature request, then I'll implement the second option above (async load all templates from the DB before each render).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719809780,https://api.github.com/repos/simonw/datasette/issues/1072,719809780,MDEyOklzc3VlQ29tbWVudDcxOTgwOTc4MA==,9599,simonw,2020-10-30T21:32:28Z,2020-10-30T21:32:28Z,OWNER,"Here's an alternative that would definitely work and would be a lot simpler, at the cost of a fair amount of RAM:

1. Before rendering the template, load ALL of the most-recent-versions of the templates that are stored in the DB. Use those to populate a `DictLoader`.
2. Render the template.

This does mean loading template bodies that we won't use. Provided an instance has less than 100 templates I imagine this will work just fine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719809259,https://api.github.com/repos/simonw/datasette/issues/1072,719809259,MDEyOklzc3VlQ29tbWVudDcxOTgwOTI1OQ==,9599,simonw,2020-10-30T21:31:10Z,2020-10-30T21:31:10Z,OWNER,"How can we tell what template Jinja will need to render?

One approach that could work:

1. Set up a dummy template loader which records the name of the template that was requested
2. Load the template
3. Now we know the list of templates that were requested. Async load those
4. The dummy template loader can now return the ones we have loaded. Load the template again.
5. Did it request any more templates? If so, load those, and repeat.
6. Keep on with this loop until a template load (which might even have to be a render) fails to request any templates that we have not yet loaded.
7. Render the template.

This is GROSS. It feels like a huge waste of CPU, and it could lead to very weird behaviour if any template variables have side effects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719807502,https://api.github.com/repos/simonw/datasette/issues/1072,719807502,MDEyOklzc3VlQ29tbWVudDcxOTgwNzUwMg==,9599,simonw,2020-10-30T21:26:49Z,2020-10-30T21:26:49Z,OWNER,"It looks like Jinja does not have a mechanism for asynchronous template loading - the loader API is synchronous.

One option may be to figure out which templates are needed (including inherited templates and includes) before rendering the template. Then async load those templates from the database into a `DictLoader`, then pass that `DictLoader` to Jinja.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719803880,https://api.github.com/repos/simonw/datasette/issues/1072,719803880,MDEyOklzc3VlQ29tbWVudDcxOTgwMzg4MA==,9599,simonw,2020-10-30T21:17:11Z,2020-10-30T21:17:11Z,OWNER,"Example from the Jinja docs: https://jinja.palletsprojects.com/en/2.11.x/api/#jinja2.BaseLoader

```python
from jinja2 import BaseLoader, TemplateNotFound
from os.path import join, exists, getmtime

class MyLoader(BaseLoader):

    def __init__(self, path):
        self.path = path

    def get_source(self, environment, template):
        path = join(self.path, template)
        if not exists(path):
            raise TemplateNotFound(template)
        mtime = getmtime(path)
        with file(path) as f:
            source = f.read().decode('utf-8')
        return source, path, lambda: mtime == getmtime(path)
```
Also available: `jinja2.FunctionLoader(load_func)` which lets me pass it a function like this one:
```
>>> def load_template(name):
...     if name == 'index.html':
...         return '...'
...
>>> loader = FunctionLoader(load_template)
```

Just one catch: I need to be able to load templates asynchronously, because they live in the database. Let's hope Jinja has a mechanism for that!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719785005,https://api.github.com/repos/simonw/datasette/issues/1072,719785005,MDEyOklzc3VlQ29tbWVudDcxOTc4NTAwNQ==,9599,simonw,2020-10-30T20:36:22Z,2020-10-30T20:36:22Z,OWNER,"It should be easy enough to show a comment that says which original template names were considered, but I may not be able to show which one was actually used (or which ones came from plugins).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719784606,https://api.github.com/repos/simonw/datasette/issues/1072,719784606,MDEyOklzc3VlQ29tbWVudDcxOTc4NDYwNg==,9599,simonw,2020-10-30T20:35:33Z,2020-10-30T20:35:33Z,OWNER,"To fix this I think I need to move the `load_template` implementation into a Jinja template loader.

I'm not sure I'll be able to keep the `Templates considered` comment working though:

https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L745-L750","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1071#issuecomment-719777499,https://api.github.com/repos/simonw/datasette/issues/1071,719777499,MDEyOklzc3VlQ29tbWVudDcxOTc3NzQ5OQ==,9599,simonw,2020-10-30T20:20:01Z,2020-10-30T20:20:01Z,OWNER,"Fixed: https://latest.datasette.io/-/messages

![demo-message](https://user-images.githubusercontent.com/9599/97753199-9c142980-1ab2-11eb-9be9-c0be41acc68e.gif)


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733485423,Messages should be displayed full width,
https://github.com/simonw/datasette/issues/1068#issuecomment-719630745,https://api.github.com/repos/simonw/datasette/issues/1068,719630745,MDEyOklzc3VlQ29tbWVudDcxOTYzMDc0NQ==,9599,simonw,2020-10-30T15:44:13Z,2020-10-30T15:44:13Z,OWNER,Documentation: https://docs.datasette.io/en/latest/authentication.html#debug-menu,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/pull/1069#issuecomment-719672967,https://api.github.com/repos/simonw/datasette/issues/1069,719672967,MDEyOklzc3VlQ29tbWVudDcxOTY3Mjk2Nw==,9599,simonw,2020-10-30T16:58:01Z,2020-10-30T16:58:01Z,OWNER,"OK, new hook specification is:
```python
@hookspec(firstresult=True)
def load_template(template, request, datasette):
    ""Load the specified template, returning the template code as a string""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook,
https://github.com/simonw/datasette/pull/1069#issuecomment-719670714,https://api.github.com/repos/simonw/datasette/issues/1069,719670714,MDEyOklzc3VlQ29tbWVudDcxOTY3MDcxNA==,9599,simonw,2020-10-30T16:53:56Z,2020-10-30T16:53:56Z,OWNER,"I'm having second thoughts about the design of the plugin hook.

Consider the following:
```python
            plugin_template_source = pm.hook.load_template(
                template=template_name,
                database=context.get(""database""),
                table=context.get(""table""),
                columns=context.get(""columns""),
                view_name=self.name,
                request=request,
                datasette=self.ds,
            )
```
It's a bit gross that `database`, `table` and `columns` are pulled out of the context like that. This doesn't make sense for pages that are rendered by plugins, for example.

So maybe for the first release of this plugin hook I should cut it down to just seeing `template`, `request` and `datasette`. I can add the table/view/etc stuff back in later if it turns out to be necessary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook,
https://github.com/simonw/datasette/pull/1069#issuecomment-719666912,https://api.github.com/repos/simonw/datasette/issues/1069,719666912,MDEyOklzc3VlQ29tbWVudDcxOTY2NjkxMg==,9599,simonw,2020-10-30T16:47:44Z,2020-10-30T16:47:44Z,OWNER,"Bringing over a comment from #1042:

> I'd like to do this all in the `datasette.render_template()` method to ensure it's available to plugins as well, not just core code that uses the `BaseView` class.
> 
> This code is the problem:
> 
> https://github.com/simonw/datasette/blob/d3e9b0aecb6f8e9b2befd9c654ccb7ce852db3e7/datasette/views/base.py#L114-L133
> 
> I think I'll fix this by moving the `select_templates` mechanism into `datasette.render_templates()`.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook,
https://github.com/simonw/datasette/pull/1069#issuecomment-719664530,https://api.github.com/repos/simonw/datasette/issues/1069,719664530,MDEyOklzc3VlQ29tbWVudDcxOTY2NDUzMA==,9599,simonw,2020-10-30T16:43:40Z,2020-10-30T16:43:40Z,OWNER,I should include an example in the documentation that shows loading templates from a database table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook,
https://github.com/simonw/datasette/pull/1069#issuecomment-719640430,https://api.github.com/repos/simonw/datasette/issues/1069,719640430,MDEyOklzc3VlQ29tbWVudDcxOTY0MDQzMA==,9599,simonw,2020-10-30T16:01:13Z,2020-10-30T16:01:13Z,OWNER,Next steps: build a demonstration plugin against this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook,
https://github.com/simonw/datasette/issues/1068#issuecomment-719332460,https://api.github.com/repos/simonw/datasette/issues/1068,719332460,MDEyOklzc3VlQ29tbWVudDcxOTMzMjQ2MA==,9599,simonw,2020-10-30T07:13:10Z,2020-10-30T07:13:10Z,OWNER,I mainly want this so I can add that debug menu to my Dogsheep.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1068#issuecomment-719331236,https://api.github.com/repos/simonw/datasette/issues/1068,719331236,MDEyOklzc3VlQ29tbWVudDcxOTMzMTIzNg==,9599,simonw,2020-10-30T07:11:58Z,2020-10-30T07:11:58Z,OWNER,Document the new permission here: https://docs.datasette.io/en/stable/authentication.html#built-in-permissions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1068#issuecomment-719329219,https://api.github.com/repos/simonw/datasette/issues/1068,719329219,MDEyOklzc3VlQ29tbWVudDcxOTMyOTIxOQ==,9599,simonw,2020-10-30T07:09:59Z,2020-10-30T07:09:59Z,OWNER,Permission idea: `debug-menu`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1068#issuecomment-719328661,https://api.github.com/repos/simonw/datasette/issues/1068,719328661,MDEyOklzc3VlQ29tbWVudDcxOTMyODY2MQ==,9599,simonw,2020-10-30T07:09:30Z,2020-10-30T07:09:30Z,OWNER,Then this can make it available to root: https://github.com/simonw/datasette/blob/18a64fbb29271ce607937110bbdb55488c43f4e0/datasette/default_permissions.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1067#issuecomment-719322666,https://api.github.com/repos/simonw/datasette/issues/1067,719322666,MDEyOklzc3VlQ29tbWVudDcxOTMyMjY2Ng==,9599,simonw,2020-10-30T07:04:02Z,2020-10-30T07:04:02Z,OWNER,"Maybe rename it to `actions_menu` and have it work for database, view, table and query pages using different arguments on each.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1067#issuecomment-719320948,https://api.github.com/repos/simonw/datasette/issues/1067,719320948,MDEyOklzc3VlQ29tbWVudDcxOTMyMDk0OA==,9599,simonw,2020-10-30T07:02:37Z,2020-10-30T07:02:37Z,OWNER,"Yes, this should be possible - no point restricting what plugin authors can do with the feature. Will need to add some extra arguments to the plugin hook for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1064#issuecomment-719117185,https://api.github.com/repos/simonw/datasette/issues/1064,719117185,MDEyOklzc3VlQ29tbWVudDcxOTExNzE4NQ==,9599,simonw,2020-10-30T01:35:17Z,2020-10-30T01:35:17Z,OWNER,"I'm going to go with a list of `{""label"": ..., ""href"": ...}` as the first iteration of this. The logout link will not be returned as part of the plugin output. A default plugin will provide the debug tools if the user is logged in as root.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719111597,https://api.github.com/repos/simonw/datasette/issues/1064,719111597,MDEyOklzc3VlQ29tbWVudDcxOTExMTU5Nw==,9599,simonw,2020-10-30T01:15:05Z,2020-10-30T01:15:05Z,OWNER,I'm torn on this one. I think I have a very slight preference for plugins returning structured objects as opposed to HTML. Less likely to regret that choice in the future?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719111373,https://api.github.com/repos/simonw/datasette/issues/1064,719111373,MDEyOklzc3VlQ29tbWVudDcxOTExMTM3Mw==,9599,simonw,2020-10-30T01:14:13Z,2020-10-30T01:14:13Z,OWNER,"Plugins returning HTML makes more sense for some of the other areas that plugins will be able to inject content - e.g. injecting content on the table or row page above the table.

If I'm going to have that as a pattern though it may make sense to use HTML here, since that will be consistent with other places that plugins can inject additional content.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719110808,https://api.github.com/repos/simonw/datasette/issues/1064,719110808,MDEyOklzc3VlQ29tbWVudDcxOTExMDgwOA==,9599,simonw,2020-10-30T01:12:09Z,2020-10-30T01:12:19Z,OWNER,Or... plugins could return HTML - maybe optionally using helper functions to generate common HTML such that plugins which use the helpers can have their HTML modified in the future.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719110582,https://api.github.com/repos/simonw/datasette/issues/1064,719110582,MDEyOklzc3VlQ29tbWVudDcxOTExMDU4Mg==,9599,simonw,2020-10-30T01:11:13Z,2020-10-30T01:11:13Z,OWNER,"Should plugins be able to add forms like the logout form here, or should they be restricted to adding navigation links?

I can't think of a reason a plugin would need to add a form. The logout form is a special case to protect against logout-csrf attacks.

So I think plugins get to return a list of dictionaries, each with a `label` and an `href`:

```python
return [{
  ""label"": ""Upload CSVs"",
  ""href"": datasette.urls.path(""/-/upload-csvs"")
}]
```
But... is there an argument for returning headings, to divide up the menu?

I think so. I also like the idea that a default plugin checks for the `root` user and outputs links to the different debugging tools - maybe those should be wrapped in a section heading.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719109770,https://api.github.com/repos/simonw/datasette/issues/1064,719109770,MDEyOklzc3VlQ29tbWVudDcxOTEwOTc3MA==,9599,simonw,2020-10-30T01:08:14Z,2020-10-30T01:08:14Z,OWNER,"How should the plugin hook work?

Here's the first version of the HTML:
```html
        <div class=""nav-menu-inner"">
            <ul>
                <li><a href=""{{ urls.instance() }}"">Home</a></li>
                <li><a href=""{{ urls.path('/-/plugins') }}"">Installed plugins</a></li>
                <li><a href=""{{ urls.path('/-/versions') }}"">Software versions</a></li>
                <li><a href=""{{ urls.path('/-/metadata') }}"">Metadata</a></li>
                {% if show_logout %}
                <form action=""{{ urls.logout() }}"" method=""post"">
                    <input type=""hidden"" name=""csrftoken"" value=""{{ csrftoken() }}"">
                    <button class=""button-as-link"">Log out</button>
                </form>{% endif %}
            </ul>
        </div>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719106174,https://api.github.com/repos/simonw/datasette/issues/1064,719106174,MDEyOklzc3VlQ29tbWVudDcxOTEwNjE3NA==,9599,simonw,2020-10-30T00:55:12Z,2020-10-30T00:55:12Z,OWNER,"So what should go in this menu?

If the user is logged in as root, I'll link to the various debug pages.

If they're not logged in at all I don't think the menu should appear.

If they are logged in as anyone, it should display to give them access to the ""log out"" button.

Plugins can add links to it. If those plugins add links, the menu will display.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719105641,https://api.github.com/repos/simonw/datasette/issues/1064,719105641,MDEyOklzc3VlQ29tbWVudDcxOTEwNTY0MQ==,9599,simonw,2020-10-30T00:53:00Z,2020-10-30T00:53:00Z,OWNER,Tips for making this accessible: https://css-tricks.com/accessible-svgs/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719105197,https://api.github.com/repos/simonw/datasette/issues/1064,719105197,MDEyOklzc3VlQ29tbWVudDcxOTEwNTE5Nw==,9599,simonw,2020-10-30T00:51:16Z,2020-10-30T00:51:16Z,OWNER,"I used a `<details><summary>` for this: https://github.com/simonw/datasette/blob/0d7ac764861d84be24d661cf4104ce61ea11a82a/datasette/templates/base.html#L16-L36

I added a bit of JavaScript so that clicking outside the menu would close it: https://github.com/simonw/datasette/blob/0d7ac764861d84be24d661cf4104ce61ea11a82a/datasette/templates/base.html#L59-L74","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719104883,https://api.github.com/repos/simonw/datasette/issues/1064,719104883,MDEyOklzc3VlQ29tbWVudDcxOTEwNDg4Mw==,9599,simonw,2020-10-30T00:50:01Z,2020-10-30T00:52:29Z,OWNER,"Here's what the prototype looks like so far:

![menu](https://user-images.githubusercontent.com/9599/97647443-7eda4f00-1a0f-11eb-8d78-b703b7a13616.gif)

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/690#issuecomment-719195346,https://api.github.com/repos/simonw/datasette/issues/690,719195346,MDEyOklzc3VlQ29tbWVudDcxOTE5NTM0Ng==,9599,simonw,2020-10-30T05:20:42Z,2020-10-30T05:20:42Z,OWNER,"I've now added two new plugin hooks: [menu_links()](https://docs.datasette.io/en/latest/plugin_hooks.html#menu-links-datasette-actor) and [table_actions()](https://docs.datasette.io/en/latest/plugin_hooks.html#table-actions-datasette-actor-database-table).

I'm going to close this issue. Further work (on column actions and and database actions) can happen in separate tickets, but I won't include them in Datasette 0.51 since they're much less interesting than table and instance actions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/issues/1066#issuecomment-719194756,https://api.github.com/repos/simonw/datasette/issues/1066,719194756,MDEyOklzc3VlQ29tbWVudDcxOTE5NDc1Ng==,9599,simonw,2020-10-30T05:18:35Z,2020-10-30T05:18:35Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#table-actions-datasette-actor-database-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732859030,Table actions menu plus plugin hook,
https://github.com/simonw/datasette/issues/1066#issuecomment-719194619,https://api.github.com/repos/simonw/datasette/issues/1066,719194619,MDEyOklzc3VlQ29tbWVudDcxOTE5NDYxOQ==,9599,simonw,2020-10-30T05:18:04Z,2020-10-30T05:18:04Z,OWNER,"The cog only appears if at least one table action has been registered by a plugin. It looks like this:

![table-actions](https://user-images.githubusercontent.com/9599/97662535-9bd54900-1a34-11eb-8e0f-56d159c8834e.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732859030,Table actions menu plus plugin hook,
https://github.com/simonw/datasette/issues/1066#issuecomment-719154646,https://api.github.com/repos/simonw/datasette/issues/1066,719154646,MDEyOklzc3VlQ29tbWVudDcxOTE1NDY0Ng==,9599,simonw,2020-10-30T03:48:15Z,2020-10-30T03:48:15Z,OWNER,This will use a very similar implementation to the navigation menu in #1064 - similar plugin hook and I'll use a `<details><summary>` to implement it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732859030,Table actions menu plus plugin hook,
https://github.com/simonw/datasette/pull/1061#issuecomment-719035336,https://api.github.com/repos/simonw/datasette/issues/1061,719035336,MDEyOklzc3VlQ29tbWVudDcxOTAzNTMzNg==,9599,simonw,2020-10-29T21:29:29Z,2020-10-29T21:29:29Z,OWNER,Those display_rows have already been processed by the `render_cell` plugin hook: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/views/database.py#L320-L346,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732634375,.blob output renderer,
https://github.com/simonw/datasette/pull/1061#issuecomment-719033013,https://api.github.com/repos/simonw/datasette/issues/1061,719033013,MDEyOklzc3VlQ29tbWVudDcxOTAzMzAxMw==,9599,simonw,2020-10-29T21:27:14Z,2020-10-29T21:27:14Z,OWNER,"Next challenge: link to `.blob` downloads from https://latest.datasette.io/fixtures?sql=select+rowid%2C+data+from+binary_data

This will be a bit tricky. Here's how that template works at the moment: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/templates/query.html#L69-L77","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732634375,.blob output renderer,
https://github.com/simonw/datasette/issues/1062#issuecomment-719031901,https://api.github.com/repos/simonw/datasette/issues/1062,719031901,MDEyOklzc3VlQ29tbWVudDcxOTAzMTkwMQ==,9599,simonw,2020-10-29T21:25:54Z,2020-10-29T21:25:54Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/views/base.py#L258-L345,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,
https://github.com/simonw/datasette/issues/1050#issuecomment-719021514,https://api.github.com/repos/simonw/datasette/issues/1050,719021514,MDEyOklzc3VlQ29tbWVudDcxOTAyMTUxNA==,9599,simonw,2020-10-29T21:05:08Z,2020-10-29T21:05:08Z,OWNER,"Idea: what if Datasette had a custom SQLite function that could be used to generate URLs to the row-level BLOB download for a value? Then custom SQL query authors could use that function to link to the relevant content.

This could be expanded to exposing other `datasette.urls` functionality as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-719001701,https://api.github.com/repos/simonw/datasette/issues/1050,719001701,MDEyOklzc3VlQ29tbWVudDcxOTAwMTcwMQ==,9599,simonw,2020-10-29T20:26:44Z,2020-10-29T20:26:44Z,OWNER,I'll do the rest of the work on this in the pull request #1061.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1034#issuecomment-719094027,https://api.github.com/repos/simonw/datasette/issues/1034,719094027,MDEyOklzc3VlQ29tbWVudDcxOTA5NDAyNw==,9599,simonw,2020-10-30T00:11:17Z,2020-10-30T00:11:17Z,OWNER,"Demos:

https://latest.datasette.io/fixtures/binary_data.csv?_size=max

```csv
rowid,data
1,http://latest.datasette.io/fixtures/binary_data/1.blob?_blob_column=data
2,http://latest.datasette.io/fixtures/binary_data/2.blob?_blob_column=data
3,
```

https://latest.datasette.io/fixtures.csv?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max

```csv
rowid,data
1,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=f3088978da8f9aea479ffc7f631370b968d2e855eeb172bea7f6c7a04262bb6d
2,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=b835b0483cedb86130b9a2c280880bf5fadc5318ddf8c18d0df5204d40df1724
3,
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1063#issuecomment-719066706,https://api.github.com/repos/simonw/datasette/issues/1063,719066706,MDEyOklzc3VlQ29tbWVudDcxOTA2NjcwNg==,9599,simonw,2020-10-29T22:46:28Z,2020-10-29T22:46:28Z,OWNER,I'm not going to do the base64 thing unless someone asks for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732685643,.csv should link to .blob downloads,
https://github.com/simonw/datasette/issues/1051#issuecomment-719053669,https://api.github.com/repos/simonw/datasette/issues/1051,719053669,MDEyOklzc3VlQ29tbWVudDcxOTA1MzY2OQ==,9599,simonw,2020-10-29T22:12:16Z,2020-10-29T22:12:16Z,OWNER,"https://latest.datasette.io/fixtures?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+101 now looks like this:

<img width=""816"" alt=""fixtures__select_rowid__data_from_binary_data_order_by_rowid_limit_101_and_understanding_Google_auths_-_for_your_information"" src=""https://user-images.githubusercontent.com/9599/97638086-15e7dc80-19f9-11eb-90a9-686e012c4db9.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1034#issuecomment-719050754,https://api.github.com/repos/simonw/datasette/issues/1034,719050754,MDEyOklzc3VlQ29tbWVudDcxOTA1MDc1NA==,9599,simonw,2020-10-29T22:04:52Z,2020-10-29T22:04:52Z,OWNER,I'm going to link to. the new `.blob` representation using the new `?_blob_hash=xxx` argument to ensure that the content served is the expected binary blob.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1063#issuecomment-719050390,https://api.github.com/repos/simonw/datasette/issues/1063,719050390,MDEyOklzc3VlQ29tbWVudDcxOTA1MDM5MA==,9599,simonw,2020-10-29T22:04:00Z,2020-10-29T22:04:00Z,OWNER,This will close #1034.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732685643,.csv should link to .blob downloads,
https://github.com/simonw/datasette/issues/1063#issuecomment-719043108,https://api.github.com/repos/simonw/datasette/issues/1063,719043108,MDEyOklzc3VlQ29tbWVudDcxOTA0MzEwOA==,9599,simonw,2020-10-29T21:46:48Z,2020-10-29T21:46:48Z,OWNER,Remove this `xfail` and `import pytest`: https://github.com/simonw/datasette/blob/503a5b7b4080a26ef9ceb1ecd1a4a6f4ef4ffc59/tests/test_csv.py#L83-L96,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732685643,.csv should link to .blob downloads,
https://github.com/simonw/datasette/pull/1061#issuecomment-719042601,https://api.github.com/repos/simonw/datasette/issues/1061,719042601,MDEyOklzc3VlQ29tbWVudDcxOTA0MjYwMQ==,9599,simonw,2020-10-29T21:45:35Z,2020-10-29T21:50:42Z,OWNER,Moving the CSV work to a separate issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732634375,.blob output renderer,
https://github.com/simonw/datasette/issues/1050#issuecomment-718989895,https://api.github.com/repos/simonw/datasette/issues/1050,718989895,MDEyOklzc3VlQ29tbWVudDcxODk4OTg5NQ==,9599,simonw,2020-10-29T20:04:15Z,2020-10-29T20:04:15Z,OWNER,I'll use `hashlib.sha256` for these hashes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718987852,https://api.github.com/repos/simonw/datasette/issues/1050,718987852,MDEyOklzc3VlQ29tbWVudDcxODk4Nzg1Mg==,9599,simonw,2020-10-29T20:00:32Z,2020-10-29T20:00:32Z,OWNER,"The reason I like the `?_blob_hash=` solution is that it feels really misleading to provide a link to ""download this binary"" which could conceivably download some other data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718980944,https://api.github.com/repos/simonw/datasette/issues/1050,718980944,MDEyOklzc3VlQ29tbWVudDcxODk4MDk0NA==,9599,simonw,2020-10-29T19:46:19Z,2020-10-29T19:46:19Z,OWNER,"Had an idea in https://github.com/simonw/datasette/issues/1051#issuecomment-718980659

> OK, alternative idea. The `.blob` output renderer from #1050 gets to see multiple rows at once.
> 
> For an arbitrary SQL query, how about if I link to this?
> 
> `/db.blob?sql=...&_blob_column=data&_blob_hash=bc4c24181ed3ce666`
> 
> Then the output renderer loops through all of the `data` results that are available to it and, if one of them hashes to that value, serves up that data?
> 
> If no matches are found it can show an error message telling you that the link has expired (presumably because the underlying database has changed since the link was generated).
> 
> I think this might be the best solution to the problem.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1051#issuecomment-718980659,https://api.github.com/repos/simonw/datasette/issues/1051,718980659,MDEyOklzc3VlQ29tbWVudDcxODk4MDY1OQ==,9599,simonw,2020-10-29T19:45:42Z,2020-10-29T19:45:42Z,OWNER,"OK, alternative idea. The `.blob` output renderer from #1050 gets to see multiple rows at once. 

For an arbitrary SQL query, how about if I link to this?

`/db.blob?sql=...&_blob_column=data&_blob_hash=bc4c24181ed3ce666`

Then the output renderer loops through all of the `data` results that are available to it and, if one of them hashes to that value, serves up that data?

If no matches are found it can show an error message telling you that the link has expired (presumably because the underlying database has changed since the link was generated).

I think this might be the best solution to the problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1053#issuecomment-718976679,https://api.github.com/repos/simonw/datasette/issues/1053,718976679,MDEyOklzc3VlQ29tbWVudDcxODk3NjY3OQ==,9599,simonw,2020-10-29T19:37:57Z,2020-10-29T19:37:57Z,OWNER,https://docs.datasette.io/en/latest/writing_plugins.html#designing-urls-for-your-plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729604838,Document recommendations for plugin authors to design URLs,
https://github.com/simonw/datasette/issues/1050#issuecomment-718346019,https://api.github.com/repos/simonw/datasette/issues/1050,718346019,MDEyOklzc3VlQ29tbWVudDcxODM0NjAxOQ==,9599,simonw,2020-10-29T04:05:07Z,2020-10-29T04:05:07Z,OWNER,"Yes, confirmed - this is a bug where if the `BLOB` column contains a `null` you get a nasty exception if you try to download it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718342036,https://api.github.com/repos/simonw/datasette/issues/1050,718342036,MDEyOklzc3VlQ29tbWVudDcxODM0MjAzNg==,9599,simonw,2020-10-29T03:49:57Z,2020-10-29T03:49:57Z,OWNER,"@thadk from that error it looks like the problem may have been that you had a BLOB column containing a `null` value? If so that's definitely a bug, I'll fix that.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/pull/1049#issuecomment-718341542,https://api.github.com/repos/simonw/datasette/issues/1049,718341542,MDEyOklzc3VlQ29tbWVudDcxODM0MTU0Mg==,9599,simonw,2020-10-29T03:48:12Z,2020-10-29T03:48:12Z,OWNER,"You could use Datasette's new `{{ urls.static_plugins(...) }}` template option - see https://docs.datasette.io/en/latest/internals.html#internals-datasette-urls - to generate a link to code that was bundled with the plugin:

```html+jinja
{% block extra_head %}
<script type=""text/x-thebe-config"">
  {
    requestKernel: true,
    binderOptions: {
      repo: ""binder-examples/requirements"",
    },
  }
</script>
<script src=""{{ urls.static_plugins(""datasette-thebelab-plugin"", ""thebelab-index.js"")""></script>
{% endblock %}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729017519,Add template block prior to extra URL loaders,
https://github.com/simonw/datasette/pull/1049#issuecomment-718340847,https://api.github.com/repos/simonw/datasette/issues/1049,718340847,MDEyOklzc3VlQ29tbWVudDcxODM0MDg0Nw==,9599,simonw,2020-10-29T03:45:47Z,2020-10-29T03:48:26Z,OWNER,"[thebe](https://thebelab.readthedocs.io/en/latest/examples/minimal_example.html) is the first time I've seen a library that requires you to set up some global JavaScript configuration before loading the script itself.

I'm hesitant to add an extra template block just to cover that one case since it's such a rare pattern. But it's important that `thebelab` can be used with Datasette.

Would this pattern work for you instead?

```html+jinja
{% block extra_head %}
<script type=""text/x-thebe-config"">
  {
    requestKernel: true,
    binderOptions: {
      repo: ""binder-examples/requirements"",
    },
  }
</script>
<script src=""https://unpkg.com/thebelab@latest/lib/index.js""></script>
{% endblock %}
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729017519,Add template block prior to extra URL loaders,
https://github.com/simonw/sqlite-utils/issues/191#issuecomment-718170295,https://api.github.com/repos/simonw/sqlite-utils/issues/191,718170295,MDEyOklzc3VlQ29tbWVudDcxODE3MDI5NQ==,9599,simonw,2020-10-28T19:50:16Z,2020-10-28T19:50:16Z,OWNER,"I think I made a mistake when I designed the initial decorator. I should have had it work like this:

```python
@db.register_function()
def reverse_string(s):
    return """".join(reversed(list(s)))
```
As this leaves open the option to add new parameters in the future.

To avoid breaking backwards compatibility I'll use the hack that detects the argument this time, but in the future I'll try to remember to always design decorators to be called like `@decorator()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731740458,Idea: @db.register_function(deterministic=True),
https://github.com/simonw/sqlite-utils/issues/191#issuecomment-718168730,https://api.github.com/repos/simonw/sqlite-utils/issues/191,718168730,MDEyOklzc3VlQ29tbWVudDcxODE2ODczMA==,9599,simonw,2020-10-28T19:47:20Z,2020-10-28T19:47:20Z,OWNER,"https://stackoverflow.com/a/3931903 looks useful:
```python
def trace(*args):
    def _trace(func):
        def wrapper(*args, **kwargs):
            print enter_string
            func(*args, **kwargs)
            print exit_string
        return wrapper
    if len(args) == 1 and callable(args[0]):
        # No arguments, this is the decorator
        # Set default values for the arguments
        enter_string = 'entering'
        exit_string = 'exiting'
        return _trace(args[0])
    else:
        # This is just returning the decorator
        enter_string, exit_string = args
        return _trace
```
Can improve that code with `functools.wraps`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731740458,Idea: @db.register_function(deterministic=True),
https://github.com/simonw/datasette/pull/1059#issuecomment-718078447,https://api.github.com/repos/simonw/datasette/issues/1059,718078447,MDEyOklzc3VlQ29tbWVudDcxODA3ODQ0Nw==,9599,simonw,2020-10-28T17:07:59Z,2020-10-28T17:08:14Z,OWNER,"> #### 0.6.0 (2020-10-27)
> 
> -   aiofiles is now tested on ppc64le.
> -   Added name and mode properties to async file objects. [#82](https://github.com/Tinche/aiofiles/pull/82)
> -   Fixed a DeprecationWarning internally. [#75](https://github.com/Tinche/aiofiles/pull/75)
> -   Python 3.9 support and tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731445447,"Update aiofiles requirement from <0.6,>=0.4 to >=0.4,<0.7",
https://github.com/simonw/datasette/issues/1057#issuecomment-717531272,https://api.github.com/repos/simonw/datasette/issues/1057,717531272,MDEyOklzc3VlQ29tbWVudDcxNzUzMTI3Mg==,9599,simonw,2020-10-27T20:51:09Z,2020-10-27T20:51:09Z,OWNER,"That works!

<img width=""1370"" alt=""Banners_and_Alerts_and_SQLite___Mike_Bostock___Observable"" src=""https://user-images.githubusercontent.com/9599/97360439-717c6380-185b-11eb-9996-e0064e4b22d4.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730797787,--cors should enable /fixtures.db CORS access,
https://github.com/simonw/datasette/issues/1058#issuecomment-717527606,https://api.github.com/repos/simonw/datasette/issues/1058,717527606,MDEyOklzc3VlQ29tbWVudDcxNzUyNzYwNg==,9599,simonw,2020-10-27T20:44:06Z,2020-10-27T20:44:06Z,OWNER,Example: https://github.com/simonw/datasette/blob/5a1519796037105bc20bcf2f91a76e022926c204/datasette/views/database.py#L26-L32,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730802994,Database download should implement cascading permissions,
https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717361487,https://api.github.com/repos/simonw/sqlite-utils/issues/189,717361487,MDEyOklzc3VlQ29tbWVudDcxNzM2MTQ4Nw==,9599,simonw,2020-10-27T16:24:04Z,2020-10-27T16:24:04Z,OWNER,"This is great, thank you very much.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records,
https://github.com/simonw/datasette/issues/1054#issuecomment-717051707,https://api.github.com/repos/simonw/datasette/issues/1054,717051707,MDEyOklzc3VlQ29tbWVudDcxNzA1MTcwNw==,9599,simonw,2020-10-27T07:41:21Z,2020-10-27T07:41:21Z,OWNER,Essentially it's this problem: https://github.com/python-versioneer/python-versioneer/issues/140,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730199464,Switch from versioneer to concrete version in setup.py,
https://github.com/simonw/datasette/issues/1054#issuecomment-717050585,https://api.github.com/repos/simonw/datasette/issues/1054,717050585,MDEyOklzc3VlQ29tbWVudDcxNzA1MDU4NQ==,9599,simonw,2020-10-27T07:38:50Z,2020-10-27T07:38:50Z,OWNER,"Maybe imitate how Django does this, e.g. https://github.com/django/django/commit/6b9b2af7352908d40ca4d31bdb1b80c013cab29a","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730199464,Switch from versioneer to concrete version in setup.py,
https://github.com/simonw/sqlite-utils/pull/189#issuecomment-716756103,https://api.github.com/repos/simonw/sqlite-utils/issues/189,716756103,MDEyOklzc3VlQ29tbWVudDcxNjc1NjEwMw==,9599,simonw,2020-10-26T18:56:19Z,2020-10-26T18:56:19Z,OWNER,"This is a great fix, thanks! If you add a unit test somewhere in here I'll merge the PR:  https://github.com/simonw/sqlite-utils/blob/main/tests/test_m2m.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records,
https://github.com/simonw/datasette/issues/1051#issuecomment-716681602,https://api.github.com/repos/simonw/datasette/issues/1051,716681602,MDEyOklzc3VlQ29tbWVudDcxNjY4MTYwMg==,9599,simonw,2020-10-26T16:51:58Z,2020-10-26T16:51:58Z,OWNER,"I still need to improve the current binary display on the query page though, where it outputs a Python `b'...'` literal.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1051#issuecomment-716681167,https://api.github.com/repos/simonw/datasette/issues/1051,716681167,MDEyOklzc3VlQ29tbWVudDcxNjY4MTE2Nw==,9599,simonw,2020-10-26T16:51:15Z,2020-10-26T16:51:15Z,OWNER,"Crazy idea: generate a signed URL containing a base64 of the gzip of the binary content (to try and reduce size).

No: this will blow through URL limits in various hosting providers and possibly even browsers. It could be made to work a little bit more reliably with some extra JavaScript that turns it into a download on the browser-side, but that would be hideously complicated.

Also the signed bit doesn't prevent people from generating SQL queries that generate nasty binary blobs for download.

I'm beginning to think that restricting this feature to just table view, not query view, is a better idea. Query view can still get at the binary using JSON and base64.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/976#issuecomment-716305890,https://api.github.com/repos/simonw/datasette/issues/976,716305890,MDEyOklzc3VlQ29tbWVudDcxNjMwNTg5MA==,9599,simonw,2020-10-26T05:07:10Z,2020-10-26T05:07:10Z,OWNER,"I used the new `datasette.urls` methods to handle escaping table names.

https://github.com/simonw/datasette/blob/f5dbe61a4568c0915ec6be820095c2960cf0857c/datasette/utils/__init__.py#L996-L1008","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708289783,Idea: -o could open to a more convenient location,
https://github.com/simonw/datasette/issues/1051#issuecomment-716204271,https://api.github.com/repos/simonw/datasette/issues/1051,716204271,MDEyOklzc3VlQ29tbWVudDcxNjIwNDI3MQ==,9599,simonw,2020-10-25T20:08:04Z,2020-10-25T20:08:04Z,OWNER,"This is bad though, because if I want to provide binary data in CSV as requested in #1034 I need some way of providing that data.

Which suggests to me that the base64 option is the only one that can make sense for arbitrary SQL queries represented as CSV. Download links won't work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1051#issuecomment-716204090,https://api.github.com/repos/simonw/datasette/issues/1051,716204090,MDEyOklzc3VlQ29tbWVudDcxNjIwNDA5MA==,9599,simonw,2020-10-25T20:06:42Z,2020-10-25T20:06:42Z,OWNER,"Providing a binary download link here is actually extremely difficult.

The problem is that the SQL query itself represents data that can change from one moment to the next. It's no good showing a ""Binary: 55 bytes"" message that links to that same SQL query but with a `.blob` extension and arguments to select the particular result, because the data may change in a way that causes that query to return a different row - at which point the download link will give you the wrong data, not the 55 bytes you asked for.

So providing a download link risks being misleading.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1052#issuecomment-716265360,https://api.github.com/repos/simonw/datasette/issues/1052,716265360,MDEyOklzc3VlQ29tbWVudDcxNjI2NTM2MA==,9599,simonw,2020-10-26T02:17:58Z,2020-10-26T02:17:58Z,OWNER,"The default z-index values for Leaflet are defined here: https://github.com/Leaflet/Leaflet/blob/b346bb8bf7bb80899baa1f4fc1536bae58e7e3e6/dist/leaflet.css#L81-L91

```css
.leaflet-pane         { z-index: 400; }

.leaflet-tile-pane    { z-index: 200; }
.leaflet-overlay-pane { z-index: 400; }
.leaflet-shadow-pane  { z-index: 500; }
.leaflet-marker-pane  { z-index: 600; }
.leaflet-tooltip-pane   { z-index: 650; }
.leaflet-popup-pane   { z-index: 700; }

.leaflet-map-pane canvas { z-index: 100; }
.leaflet-map-pane svg    { z-index: 200; }
```
So a `z-index` of 1000 on the menu should fix this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729183332,Column action menu overlapped by Leaflet maps,
https://github.com/simonw/datasette/issues/1050#issuecomment-716175236,https://api.github.com/repos/simonw/datasette/issues/1050,716175236,MDEyOklzc3VlQ29tbWVudDcxNjE3NTIzNg==,9599,simonw,2020-10-25T16:35:20Z,2020-10-25T16:35:20Z,OWNER,"This is clearly a better solution than the one I  implemented in #1040 - I don't have to add a new route, I don't have to implement permission checks, it reuses mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-716174203,https://api.github.com/repos/simonw/datasette/issues/1050,716174203,MDEyOklzc3VlQ29tbWVudDcxNjE3NDIwMw==,9599,simonw,2020-10-25T16:27:39Z,2020-10-25T16:53:27Z,OWNER,"Idea: `.blob` output rendererer, where you tell it which column you want using `?_blob_column=x`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1034#issuecomment-716078777,https://api.github.com/repos/simonw/datasette/issues/1034,716078777,MDEyOklzc3VlQ29tbWVudDcxNjA3ODc3Nw==,9599,simonw,2020-10-25T01:25:11Z,2020-10-25T01:25:11Z,OWNER,"SQLite actually has APIs that could help here: https://www.sqlite.org/c3ref/column_database_name.html - for any given SQL query they identify the origin/table/column that is the source of each resulting column.

Those aren't exposed in the Python `sqlite3` module though, so using them could be extremely tricky.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716078605,https://api.github.com/repos/simonw/datasette/issues/1034,716078605,MDEyOklzc3VlQ29tbWVudDcxNjA3ODYwNQ==,9599,simonw,2020-10-25T01:22:22Z,2020-10-25T01:22:22Z,OWNER,For arbitrary CSV the only solution I can think of is to embed the base64 value.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716078512,https://api.github.com/repos/simonw/datasette/issues/1034,716078512,MDEyOklzc3VlQ29tbWVudDcxNjA3ODUxMg==,9599,simonw,2020-10-25T01:21:11Z,2020-10-25T01:21:11Z,OWNER,"What should happen for CSV export of arbitrary SQL queries, where there's no obvious BLOB to link to?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716078420,https://api.github.com/repos/simonw/datasette/issues/1034,716078420,MDEyOklzc3VlQ29tbWVudDcxNjA3ODQyMA==,9599,simonw,2020-10-25T01:20:00Z,2020-10-25T01:20:00Z,OWNER,That documentation: https://docs.datasette.io/en/latest/internals.html#absolute-url-request-path,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716077508,https://api.github.com/repos/simonw/datasette/issues/1034,716077508,MDEyOklzc3VlQ29tbWVudDcxNjA3NzUwOA==,9599,simonw,2020-10-25T01:09:17Z,2020-10-25T01:09:17Z,OWNER,Here's how those absolute `next_url` values are generated: https://github.com/simonw/datasette/blob/5db7ae3ce165ded57c7fb1cfbdb3258b1cf06c10/datasette/views/table.py#L774-L776,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716077541,https://api.github.com/repos/simonw/datasette/issues/1034,716077541,MDEyOklzc3VlQ29tbWVudDcxNjA3NzU0MQ==,9599,simonw,2020-10-25T01:09:38Z,2020-10-25T01:10:04Z,OWNER,I should turn `datasette.absolute_url(...)` into a documented internal API on https://docs.datasette.io/en/stable/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716077436,https://api.github.com/repos/simonw/datasette/issues/1034,716077436,MDEyOklzc3VlQ29tbWVudDcxNjA3NzQzNg==,9599,simonw,2020-10-25T01:08:35Z,2020-10-25T01:08:42Z,OWNER,"This is actually a bit tricky to implement, for a few reasons:

- Need to generate a full URL, including the `https://host/` bit. I've done this for `next_url` in the JSON output before, thankfully.
- This only makes sense for CSV output for tables. If it's the CSV output of an arbitrary query there's no `/db/table/-/blob/pk/column.blob` page for me to link to.
- Need to generate those `/.../-/blob/...` URLs for the data that is being output as CSV.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1046#issuecomment-716071507,https://api.github.com/repos/simonw/datasette/issues/1046,716071507,MDEyOklzc3VlQ29tbWVudDcxNjA3MTUwNw==,9599,simonw,2020-10-25T00:06:47Z,2020-10-25T00:06:47Z,OWNER,"I used https://primer.style/octicons/download-16 instead.

<img width=""734"" alt=""favicons__favicons__71_rows"" src=""https://user-images.githubusercontent.com/9599/97096021-44417280-161b-11eb-9bc4-88def6afaca4.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728895193,Link to blob downloads in the right places,
https://github.com/simonw/datasette/issues/1046#issuecomment-716066342,https://api.github.com/repos/simonw/datasette/issues/1046,716066342,MDEyOklzc3VlQ29tbWVudDcxNjA2NjM0Mg==,9599,simonw,2020-10-24T23:02:07Z,2020-10-24T23:02:25Z,OWNER,"A download icon would be nice for the links in the table display. I like this one https://primer.style/octicons/download-24

```svg
<svg xmlns=""http://www.w3.org/2000/svg"" viewBox=""0 0 24 24"" width=""24"" height=""24"">
  <path d=""M4.97 11.03a.75.75 0 111.06-1.06L11 14.94V2.75a.75.75 0 011.5 0v12.19l4.97-4.97a.75.75 0 111.06 1.06l-6.25 6.25a.75.75 0 01-1.06 0l-6.25-6.25zm-.22 9.47a.75.75 0 000 1.5h14.5a.75.75 0 000-1.5H4.75z""></path>
</svg>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728895193,Link to blob downloads in the right places,
https://github.com/simonw/datasette/issues/1033#issuecomment-716048564,https://api.github.com/repos/simonw/datasette/issues/1033,716048564,MDEyOklzc3VlQ29tbWVudDcxNjA0ODU2NA==,9599,simonw,2020-10-24T20:08:31Z,2020-10-24T20:08:31Z,OWNER,Documentation here: https://docs.datasette.io/en/latest/internals.html#datasette-urls,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method,
https://github.com/simonw/datasette/issues/575#issuecomment-716048199,https://api.github.com/repos/simonw/datasette/issues/575,716048199,MDEyOklzc3VlQ29tbWVudDcxNjA0ODE5OQ==,9599,simonw,2020-10-24T20:05:44Z,2020-10-24T20:05:44Z,OWNER,https://docs.datasette.io/en/latest/writing_plugins.html#static-assets,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497162288,Plugin documentation should cover how to bundle static/templates in setup.py,
https://github.com/simonw/datasette/issues/1042#issuecomment-715618333,https://api.github.com/repos/simonw/datasette/issues/1042,715618333,MDEyOklzc3VlQ29tbWVudDcxNTYxODMzMw==,9599,simonw,2020-10-23T22:33:24Z,2020-10-23T22:33:24Z,OWNER,"It wouldn't be a disaster if template-loading plugins were unable to hook into the `/{slug1}/{slug2}.html` custom page mechanism, since plugins can define their own pages already using `register_routes()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715618077,https://api.github.com/repos/simonw/datasette/issues/1042,715618077,MDEyOklzc3VlQ29tbWVudDcxNTYxODA3Nw==,9599,simonw,2020-10-23T22:32:24Z,2020-10-23T22:32:24Z,OWNER,"Another option: the first version of the plugin hook could accept only the template filename. Subsequent releases could add more arguments, since Pluggy allows new arguments to be added without breaking backwards compatibility.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715617830,https://api.github.com/repos/simonw/datasette/issues/1042,715617830,MDEyOklzc3VlQ29tbWVudDcxNTYxNzgzMA==,9599,simonw,2020-10-23T22:31:26Z,2020-10-23T22:31:26Z,OWNER,"So maybe this should be a `register_template_loader` mechanism that returns a Jinja loader after all? That would mean that only the template filename could be used as the input to the plugin, which doesn't seem as useful as emulating the `extra_template_vars()` interface.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715617405,https://api.github.com/repos/simonw/datasette/issues/1042,715617405,MDEyOklzc3VlQ29tbWVudDcxNTYxNzQwNQ==,9599,simonw,2020-10-23T22:29:53Z,2020-10-23T22:29:53Z,OWNER,"Also consider that `DatasetteRouter` uses `.list_templates()` to gather together `{slug}.html` style templates for the custom page templates mechanism: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L949-L967

For that to work with the new plugin hook, custom template providing plugins will need a way to provide a list of templates that they know about.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715616757,https://api.github.com/repos/simonw/datasette/issues/1042,715616757,MDEyOklzc3VlQ29tbWVudDcxNTYxNjc1Nw==,9599,simonw,2020-10-23T22:27:28Z,2020-10-23T22:27:28Z,OWNER,"Almost all of the core template loading happens in the `BaseView.render` method: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/views/base.py#L114-L133

The one exception is the 404 handling code here: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L1034-L1042","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715614971,https://api.github.com/repos/simonw/datasette/issues/1042,715614971,MDEyOklzc3VlQ29tbWVudDcxNTYxNDk3MQ==,9599,simonw,2020-10-23T22:20:14Z,2020-10-23T22:23:51Z,OWNER,"Alternative plugin hook idea:
```python
@hookspec
def load_template(template, database, table, columns, view_name, request, datasette):
    ""Load the specified template, returning the template code as a string""
```
Imitating the existing `extra_template_vars` family of hooks: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-template-vars-template-database-table-columns-view-name-request-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715643763,https://api.github.com/repos/simonw/datasette/issues/1042,715643763,MDEyOklzc3VlQ29tbWVudDcxNTY0Mzc2Mw==,9599,simonw,2020-10-24T00:34:31Z,2020-10-24T00:34:52Z,OWNER,I'm going to rename that to template variable from `select_templates` to `templates_considered` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715643646,https://api.github.com/repos/simonw/datasette/issues/1042,715643646,MDEyOklzc3VlQ29tbWVudDcxNTY0MzY0Ng==,9599,simonw,2020-10-24T00:33:46Z,2020-10-24T00:33:46Z,OWNER,"I'd like to do this all in the `datasette.render_template()` method to ensure it's available to plugins as well, not just core code that uses the `BaseView` class.

This code is the problem:
https://github.com/simonw/datasette/blob/d3e9b0aecb6f8e9b2befd9c654ccb7ce852db3e7/datasette/views/base.py#L114-L133

I think I'll fix this by moving the `select_templates` mechanism into `datasette.render_templates()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1045#issuecomment-715641183,https://api.github.com/repos/simonw/datasette/issues/1045,715641183,MDEyOklzc3VlQ29tbWVudDcxNTY0MTE4Mw==,9599,simonw,2020-10-24T00:19:29Z,2020-10-24T00:19:29Z,OWNER,"It turns out it already does that:

https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L710-L720

But the documentation doesn't reflect that:

> `template` - string
> 
> > The template file to be rendered, e.g. `my_plugin.html`. Datasette will search for this file first in the `--template-dir=` location, if it was specified - then in the plugin's bundled templates and finally in Datasette's set of default templates.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728600048,"Document that datasette.render_template(template, ...) also accepts a list of templates",
https://github.com/simonw/datasette/pull/1040#issuecomment-715587715,https://api.github.com/repos/simonw/datasette/issues/1040,715587715,MDEyOklzc3VlQ29tbWVudDcxNTU4NzcxNQ==,9599,simonw,2020-10-23T21:01:07Z,2020-10-23T21:03:10Z,OWNER,"A download icon would be nice for the links in the table display.

I like this one https://primer.style/octicons/download-24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726910999,/db/table/-/blob/pk/column.blob download URL,
https://github.com/simonw/datasette/pull/1043#issuecomment-715586711,https://api.github.com/repos/simonw/datasette/issues/1043,715586711,MDEyOklzc3VlQ29tbWVudDcxNTU4NjcxMQ==,9599,simonw,2020-10-23T20:58:26Z,2020-10-23T20:58:26Z,OWNER,I misunderstood - `asgi-csrf` already has an sdist.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727915394,Include LICENSE in sdist,
https://github.com/simonw/datasette/pull/1043#issuecomment-715585140,https://api.github.com/repos/simonw/datasette/issues/1043,715585140,MDEyOklzc3VlQ29tbWVudDcxNTU4NTE0MA==,9599,simonw,2020-10-23T20:54:29Z,2020-10-23T20:54:29Z,OWNER,Thanks. I'll push a source release of `asgi-csrf`.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727915394,Include LICENSE in sdist,
https://github.com/simonw/datasette/pull/1044#issuecomment-715584579,https://api.github.com/repos/simonw/datasette/issues/1044,715584579,MDEyOklzc3VlQ29tbWVudDcxNTU4NDU3OQ==,9599,simonw,2020-10-23T20:53:01Z,2020-10-23T20:53:01Z,OWNER,Thanks for this!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727916744,Add minimum supported python,
https://github.com/simonw/datasette/issues/745#issuecomment-715556545,https://api.github.com/repos/simonw/datasette/issues/745,715556545,MDEyOklzc3VlQ29tbWVudDcxNTU1NjU0NQ==,9599,simonw,2020-10-23T19:47:10Z,2020-10-23T19:47:10Z,OWNER,Dupe of #647 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608613033,Extract the hash-URL mechanism out into a plugin,
https://github.com/simonw/datasette/issues/1042#issuecomment-715497419,https://api.github.com/repos/simonw/datasette/issues/1042,715497419,MDEyOklzc3VlQ29tbWVudDcxNTQ5NzQxOQ==,9599,simonw,2020-10-23T18:12:40Z,2020-10-23T18:12:40Z,OWNER,Maybe the template loader can optionally return some extra context to pass to the template. That could be used to solve the templates considered comment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715496859,https://api.github.com/repos/simonw/datasette/issues/1042,715496859,MDEyOklzc3VlQ29tbWVudDcxNTQ5Njg1OQ==,9599,simonw,2020-10-23T18:11:27Z,2020-10-23T18:11:27Z,OWNER,"When loading a template the filename is required, but you can optionally also send a set of extra arguments which the template loader can take into consideration.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715490532,https://api.github.com/repos/simonw/datasette/issues/1042,715490532,MDEyOklzc3VlQ29tbWVudDcxNTQ5MDUzMg==,9599,simonw,2020-10-23T17:57:34Z,2020-10-23T17:57:34Z,OWNER,"A better version of this hook would be passed the database, table and query name depending on what was being rendered.

This would require some re-thinking of how core templates are loaded, especially since I would want the templates considered comment to continue working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-714868867,https://api.github.com/repos/simonw/datasette/issues/1042,714868867,MDEyOklzc3VlQ29tbWVudDcxNDg2ODg2Nw==,9599,simonw,2020-10-23T02:31:17Z,2020-10-23T02:31:17Z,OWNER,I'll build this in conjunction with a plugin that supports editing templates stored in SQLite.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-714868624,https://api.github.com/repos/simonw/datasette/issues/1042,714868624,MDEyOklzc3VlQ29tbWVudDcxNDg2ODYyNA==,9599,simonw,2020-10-23T02:30:27Z,2020-10-23T02:30:37Z,OWNER,Maybe `register_template_loader(datasette)` which returns an object which is added in at the beginning of the list passed to `ChoiceLoader` here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-714868207,https://api.github.com/repos/simonw/datasette/issues/1042,714868207,MDEyOklzc3VlQ29tbWVudDcxNDg2ODIwNw==,9599,simonw,2020-10-23T02:29:12Z,2020-10-23T02:29:12Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/d0cc6f4c32e1f89238ddec782086b3122f445bd4/datasette/app.py#L288-L311,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-714758139,https://api.github.com/repos/simonw/sqlite-utils/issues/173,714758139,MDEyOklzc3VlQ29tbWVudDcxNDc1ODEzOQ==,9599,simonw,2020-10-22T20:57:56Z,2020-10-22T20:57:56Z,OWNER,"I could use `ijson` to provide a progress bar for JSON arrays too. I'd prefer to keep that as an optional dependency though, since `sqlite-utils` is a library dependency for many other projects and it would be using `ijson` purely for the CLI component.

Here's how to iterate through a list of objects being read from a file:

```python
import json
parser = ijson.items(open(
    ""/tmp/list.json""
), ""item"")
for object in parser:
   # ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/datasette/issues/1041#issuecomment-714683801,https://api.github.com/repos/simonw/datasette/issues/1041,714683801,MDEyOklzc3VlQ29tbWVudDcxNDY4MzgwMQ==,9599,simonw,2020-10-22T18:37:47Z,2020-10-22T18:37:47Z,OWNER,"I think I'll do this by looking for URLs that start with `/` - since it's also possible to have full `https://...` URLs in that setting.
```json
{
    ""extra_css_urls"": [
        ""/static/styles.css""
    ],
    ""extra_js_urls"": [
        ""/static/app.js""
    ]
}
```
I need to think about the `extra_css_urls` and `extra_js_urls` plugin hooks too: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-css-urls-template-database-table-columns-view-name-request-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting,
https://github.com/simonw/datasette/issues/1041#issuecomment-714682825,https://api.github.com/repos/simonw/datasette/issues/1041,714682825,MDEyOklzc3VlQ29tbWVudDcxNDY4MjgyNQ==,9599,simonw,2020-10-22T18:36:10Z,2020-10-22T18:36:10Z,OWNER,I'll need to update these docs once there's a solution for this in place: https://docs.datasette.io/en/latest/custom_templates.html#serving-static-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting,
https://github.com/simonw/datasette/issues/1041#issuecomment-714682288,https://api.github.com/repos/simonw/datasette/issues/1041,714682288,MDEyOklzc3VlQ29tbWVudDcxNDY4MjI4OA==,9599,simonw,2020-10-22T18:35:15Z,2020-10-22T18:35:15Z,OWNER,"@psychemedia said: https://github.com/simonw/datasette/issues/1033#issuecomment-714657366

> How does `/-/static` relate to [current guidance docs around `static`](https://docs.datasette.io/en/latest/custom_templates.html?highlight=static#serving-static-files) regarding the `--static option` and metadata formulations such as `""extra_js_urls"": [ ""/static/app.js""]` (I've not managed to get this to work in a Jupyter server proxied set up; the [datasette / jupyter server proxy repo](https://github.com/simonw/jupyterserverproxy-datasette-demo) may provide a useful test example, eg via MyBinder, for folk to crib from?)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting,
https://github.com/simonw/datasette/issues/1033#issuecomment-714681365,https://api.github.com/repos/simonw/datasette/issues/1033,714681365,MDEyOklzc3VlQ29tbWVudDcxNDY4MTM2NQ==,9599,simonw,2020-10-22T18:33:48Z,2020-10-22T18:33:48Z,OWNER,"That's a good question - I hadn't considered that. I'm going to open a new issue to have `extra_js_urls` respect the `base_url` setting, since the static files will be served from a different location.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method,
https://github.com/simonw/sqlite-utils/issues/171#issuecomment-714208848,https://api.github.com/repos/simonw/sqlite-utils/issues/171,714208848,MDEyOklzc3VlQ29tbWVudDcxNDIwODg0OA==,9599,simonw,2020-10-22T04:07:14Z,2020-10-22T04:07:14Z,OWNER,"I made the `--load-extension` command much more widely supported in #137 - which should be useful for anyone who wants to use this extension.

It's a bit too obscure for me to want to add direct Python library support relating to that extension though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707407567,Idea: transitive closure tables for tree structures,
https://github.com/simonw/datasette/pull/1031#issuecomment-714206875,https://api.github.com/repos/simonw/datasette/issues/1031,714206875,MDEyOklzc3VlQ29tbWVudDcxNDIwNjg3NQ==,9599,simonw,2020-10-22T04:01:19Z,2020-10-22T04:01:19Z,OWNER,I don't fully understand the bug you're fixing here. Could you provide a bit more explanation?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724369025,Fallback to databases in inspect-data.json when no -i options are passed,
https://github.com/simonw/datasette/pull/1040#issuecomment-714206533,https://api.github.com/repos/simonw/datasette/issues/1040,714206533,MDEyOklzc3VlQ29tbWVudDcxNDIwNjUzMw==,9599,simonw,2020-10-22T04:00:25Z,2020-10-22T04:00:25Z,OWNER,I've decided not to offer a configuration option to turn this off. I'll reconsider if someone asks for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726910999,/db/table/-/blob/pk/column.blob download URL,
https://github.com/simonw/datasette/issues/998#issuecomment-714205783,https://api.github.com/repos/simonw/datasette/issues/998,714205783,MDEyOklzc3VlQ29tbWVudDcxNDIwNTc4Mw==,9599,simonw,2020-10-22T03:58:13Z,2020-10-22T03:58:13Z,OWNER,This is now live here: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/datasette/issues/998#issuecomment-714117534,https://api.github.com/repos/simonw/datasette/issues/998,714117534,MDEyOklzc3VlQ29tbWVudDcxNDExNzUzNA==,9599,simonw,2020-10-22T01:12:06Z,2020-10-22T01:12:06Z,OWNER,"Demo:
![table-scroll](https://user-images.githubusercontent.com/9599/96806421-e74e7e00-13c8-11eb-95fe-44d01e4c2eb3.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/datasette/issues/998#issuecomment-714092002,https://api.github.com/repos/simonw/datasette/issues/998,714092002,MDEyOklzc3VlQ29tbWVudDcxNDA5MjAwMg==,9599,simonw,2020-10-22T00:55:10Z,2020-10-22T00:55:10Z,OWNER,This isn't blocked on #987 - it just means that `datasette-cluster-map` will need to learn to look for `.table-wrapper` first and fall back on the table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/datasette/issues/998#issuecomment-714090965,https://api.github.com/repos/simonw/datasette/issues/998,714090965,MDEyOklzc3VlQ29tbWVudDcxNDA5MDk2NQ==,9599,simonw,2020-10-22T00:54:30Z,2020-10-22T00:54:30Z,OWNER,"Easiest fix for the column action menu positioning - hide them when the user scrolls the containing div:
```javascript
document.querySelector('.table-wrapper').addEventListener(
  'scroll', () => document.querySelector('.dropdown-menu').style.display = 'none'
);                                            
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/datasette/pull/1038#issuecomment-713920461,https://api.github.com/repos/simonw/datasette/issues/1038,713920461,MDEyOklzc3VlQ29tbWVudDcxMzkyMDQ2MQ==,9599,simonw,2020-10-21T22:43:51Z,2020-10-21T22:43:51Z,OWNER,Thanks for spotting this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726154220,DOC: Fix syntax error,
https://github.com/simonw/datasette/issues/1036#issuecomment-713830842,https://api.github.com/repos/simonw/datasette/issues/1036,713830842,MDEyOklzc3VlQ29tbWVudDcxMzgzMDg0Mg==,9599,simonw,2020-10-21T19:41:20Z,2020-10-21T19:41:20Z,OWNER,Another useful demo database: https://datasette-render-images-demo.datasette.io/favicons/favicons - see https://datasette-render-images-demo.datasette.io/favicons/favicons.csv,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713829629,https://api.github.com/repos/simonw/datasette/issues/1036,713829629,MDEyOklzc3VlQ29tbWVudDcxMzgyOTYyOQ==,9599,simonw,2020-10-21T19:38:43Z,2020-10-21T19:38:43Z,OWNER,"Should this work just for BLOB columns, or should it work for other columns too?

For the moment I'm going to restrict it to BLOBs, since data from other columns is available through the UI whereas BLOB columns are not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713821656,https://api.github.com/repos/simonw/datasette/issues/1036,713821656,MDEyOklzc3VlQ29tbWVudDcxMzgyMTY1Ng==,9599,simonw,2020-10-21T19:22:45Z,2020-10-21T19:41:48Z,OWNER,"So for https://latest.datasette.io/fixtures/binary_data the BLOB download URLs would be:

`https://latest.datasette.io/fixtures/-/blob/binary_data/1/data.blob` - that last bit after the primary key is to indicate the `data` column

With these headers:

- `Content-Disposition: attachment; filename=""binary_data-1-data.blob""`
- `X-Content-Type-Options: nosniff`
- `Content-Type: application/binary`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713818817,https://api.github.com/repos/simonw/datasette/issues/1036,713818817,MDEyOklzc3VlQ29tbWVudDcxMzgxODgxNw==,9599,simonw,2020-10-21T19:17:01Z,2020-10-21T19:17:01Z,OWNER,Actually I like `.blob`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713818178,https://api.github.com/repos/simonw/datasette/issues/1036,713818178,MDEyOklzc3VlQ29tbWVudDcxMzgxODE3OA==,9599,simonw,2020-10-21T19:15:38Z,2020-10-21T19:16:34Z,OWNER,"What should the suggested filename be?

I think something that includes the table name, primary key and the name of the column would work.

How about a file extension? I guess `.binary`, then let the user rename it? Or `.raw`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713899530,https://api.github.com/repos/simonw/datasette/issues/1036,713899530,MDEyOklzc3VlQ29tbWVudDcxMzg5OTUzMA==,9599,simonw,2020-10-21T21:55:00Z,2020-10-21T21:55:00Z,OWNER,"This code needs these permission checks:
https://github.com/simonw/datasette/blob/bf82b3d6a605c9ddadd5fb739249dfe6defaf635/datasette/views/table.py#L911-L913","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1039#issuecomment-713754844,https://api.github.com/repos/simonw/datasette/issues/1039,713754844,MDEyOklzc3VlQ29tbWVudDcxMzc1NDg0NA==,9599,simonw,2020-10-21T17:58:27Z,2020-10-21T17:58:27Z,OWNER,"Now live: https://latest.datasette.io/fixtures/roadside_attraction_characteristics

![anim](https://user-images.githubusercontent.com/9599/96759016-55288480-138c-11eb-8ba0-d8e0f6dd8b1f.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726687572,Add an animation to the column actions menu,
https://github.com/simonw/datasette/issues/1036#issuecomment-713226726,https://api.github.com/repos/simonw/datasette/issues/1036,713226726,MDEyOklzc3VlQ29tbWVudDcxMzIyNjcyNg==,9599,simonw,2020-10-21T01:04:25Z,2020-10-21T01:04:25Z,OWNER,"Extra security idea: a `blob_download_host` setting which can be used to indicate a host that should be used for downloads - for example `datasettestatic.com`. If this setting is populated then binary downloads are served from paths on that host only, and no other Datasette URLs from that host will be served.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/262#issuecomment-713208667,https://api.github.com/repos/simonw/datasette/issues/262,713208667,MDEyOklzc3VlQ29tbWVudDcxMzIwODY2Nw==,9599,simonw,2020-10-21T00:03:18Z,2020-10-21T00:03:18Z,OWNER,"I think I should prioritize the facets component of this, since that could have significant performance wins while also supporting `datasette-graphql`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/262#issuecomment-713200782,https://api.github.com/repos/simonw/datasette/issues/262,713200782,MDEyOklzc3VlQ29tbWVudDcxMzIwMDc4Mg==,9599,simonw,2020-10-20T23:41:30Z,2020-10-20T23:41:30Z,OWNER,This is now blocking https://github.com/simonw/datasette-graphql/issues/61 because that issue needs a way to turn off suggested facets when retrieving the results of a table query.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/1036#issuecomment-713278349,https://api.github.com/repos/simonw/datasette/issues/1036,713278349,MDEyOklzc3VlQ29tbWVudDcxMzI3ODM0OQ==,9599,simonw,2020-10-21T03:42:29Z,2020-10-21T03:42:29Z,OWNER,Possible URL for this: `/db/table/-/blob/primary-keys` - this would use the `/db/table/-/` namespace proposed in #296.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1034#issuecomment-713277810,https://api.github.com/repos/simonw/datasette/issues/1034,713277810,MDEyOklzc3VlQ29tbWVudDcxMzI3NzgxMA==,9599,simonw,2020-10-21T03:40:50Z,2020-10-25T01:01:23Z,OWNER,Blocked awaiting #1036 (update: now unblocked),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/998#issuecomment-713269155,https://api.github.com/repos/simonw/datasette/issues/998,713269155,MDEyOklzc3VlQ29tbWVudDcxMzI2OTE1NQ==,9599,simonw,2020-10-21T03:17:07Z,2020-10-21T03:17:07Z,OWNER,"This may require updates to the column action menu JavaScript too, since it was not built with scrolling sideways in mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/datasette/issues/1037#issuecomment-713268905,https://api.github.com/repos/simonw/datasette/issues/1037,713268905,MDEyOklzc3VlQ29tbWVudDcxMzI2ODkwNQ==,9599,simonw,2020-10-21T03:16:36Z,2020-10-21T03:16:36Z,OWNER,Dupe of #998.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726094754,Add horizontal scrollbar to tables,
https://github.com/simonw/datasette/issues/1037#issuecomment-713268498,https://api.github.com/repos/simonw/datasette/issues/1037,713268498,MDEyOklzc3VlQ29tbWVudDcxMzI2ODQ5OA==,9599,simonw,2020-10-21T03:15:44Z,2020-10-21T03:15:44Z,OWNER,"This may require updates to the column action menu JavaScript too, since it was not built with scrolling sideways in mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726094754,Add horizontal scrollbar to tables,
https://github.com/simonw/datasette/issues/1037#issuecomment-713267989,https://api.github.com/repos/simonw/datasette/issues/1037,713267989,MDEyOklzc3VlQ29tbWVudDcxMzI2Nzk4OQ==,9599,simonw,2020-10-21T03:14:34Z,2020-10-21T03:14:34Z,OWNER,"This is particularly relevant to the `datasette-cluster-map` plugin - the map is much nicer to use if the table itself can be scrolled.

That plugin also makes this harder to build, because the plugin inserts the map as the direct predecessor of the `<table>` element and hence breaks if you try to wrap that in a `<div>`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726094754,Add horizontal scrollbar to tables,
https://github.com/simonw/datasette/issues/1034#issuecomment-713191819,https://api.github.com/repos/simonw/datasette/issues/1034,713191819,MDEyOklzc3VlQ29tbWVudDcxMzE5MTgxOQ==,9599,simonw,2020-10-20T23:12:58Z,2020-10-20T23:12:58Z,OWNER,"Enzo has a great solution here: https://twitter.com/enzo_mdd/status/1318685442976436226

> Or maybe an option for a url. This keeps the CSV small but allows scripts to download binary data as needed.

In #1036 I'm planning on adding a way for users to access BLOB data. I can include that URL in the CSV output.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1036#issuecomment-713186189,https://api.github.com/repos/simonw/datasette/issues/1036,713186189,MDEyOklzc3VlQ29tbWVudDcxMzE4NjE4OQ==,9599,simonw,2020-10-20T22:56:33Z,2020-10-20T22:56:33Z,OWNER,I think this plus the binary-CSV stuff in #1034 will justify a dedicated section of the documentation to talk about how Datasette handles binary BLOB columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713185871,https://api.github.com/repos/simonw/datasette/issues/1036,713185871,MDEyOklzc3VlQ29tbWVudDcxMzE4NTg3MQ==,9599,simonw,2020-10-20T22:55:36Z,2020-10-20T22:55:36Z,OWNER,I can also use a `Content-Disposition` header to force a download. I'm reasonably confident that the combination of `Content-Disposition` and `X-Content-Type-Options: nosniff` and `application/binary` will let me allow users to download the contents of arbitrary BLOB columns without any XSS risk.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713185173,https://api.github.com/repos/simonw/datasette/issues/1036,713185173,MDEyOklzc3VlQ29tbWVudDcxMzE4NTE3Mw==,9599,simonw,2020-10-20T22:53:41Z,2020-10-20T22:53:41Z,OWNER,"https://security.stackexchange.com/questions/12896/does-x-content-type-options-really-prevent-content-sniffing-attacks says:

> In Tangled Web Michal Zalewski says:
> 
> > Refrain from using Content-Type: application/octet-stream and use application/binary instead, especially for unknown document types. Refrain from returning Content-Type: text/plain.
> > 
> > For example, any code-hosting platform must exercise caution when returning executables or source archives as application/octet-stream, because there is a risk they may be misinterpreted as HTML and displayed inline.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713184374,https://api.github.com/repos/simonw/datasette/issues/1036,713184374,MDEyOklzc3VlQ29tbWVudDcxMzE4NDM3NA==,9599,simonw,2020-10-20T22:51:22Z,2020-10-20T22:51:22Z,OWNER,"From https://hackerone.com/reports/126197:

> archive.uber.com mirrors pypi. When downloading `.tar.gz` files from archive.uber.com, the MIME type is `application/octet-stream`. Injecting `<html><script>alert(0)</script>` into the start of the `.tar.gz` causes an XSS in Internet Explorer due to MIME sniffing.

So you do have to be careful not to open accidental XSS holes with `application/octet-stream` thanks to (presumably older) versions of IE.

From that thread it looks like the solution is to add a `X-Content-Type-Options: nosniff` header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713183306,https://api.github.com/repos/simonw/datasette/issues/1036,713183306,MDEyOklzc3VlQ29tbWVudDcxMzE4MzMwNg==,9599,simonw,2020-10-20T22:48:10Z,2020-10-20T22:48:10Z,OWNER,Twitter thread: https://twitter.com/dancow/status/1318681053347840005,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1034#issuecomment-713176082,https://api.github.com/repos/simonw/datasette/issues/1034,713176082,MDEyOklzc3VlQ29tbWVudDcxMzE3NjA4Mg==,9599,simonw,2020-10-20T22:27:33Z,2020-10-20T22:27:33Z,OWNER,"This feels good to me - it's consistent with how other features in Datasette work, and it means users who need the binary data in CSV (for whatever reason) can get it if they want to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713175741,https://api.github.com/repos/simonw/datasette/issues/1034,713175741,MDEyOklzc3VlQ29tbWVudDcxMzE3NTc0MQ==,9599,simonw,2020-10-20T22:26:45Z,2020-10-20T22:26:45Z,OWNER,"> New idea: since binary in CSV doesn't make sense anyway, emulate Datasette's HTML UI default and output this:
> 
>     id,title,data
>     1,Some title,<Binary data: 14 bytes>
>     2,Other title,<Binary data: 57 bytes>
> 
> Then allow users to add ?_base64=1 to the URL to get base64 instead
> https://twitter.com/simonw/status/1318679950635888641","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713174690,https://api.github.com/repos/simonw/datasette/issues/1034,713174690,MDEyOklzc3VlQ29tbWVudDcxMzE3NDY5MA==,9599,simonw,2020-10-20T22:23:50Z,2020-10-20T22:23:50Z,OWNER,Or... default to `<Binary data: 7 bytes>` and support a `?_base64=1` option which outputs in base64 instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713174341,https://api.github.com/repos/simonw/datasette/issues/1034,713174341,MDEyOklzc3VlQ29tbWVudDcxMzE3NDM0MQ==,9599,simonw,2020-10-20T22:22:53Z,2020-10-20T22:23:14Z,OWNER,"An even easier option: do what the Datasette UI does and output `<Binary data: 7 bytes>` for that CSV cell, as seen on https://latest.datasette.io/fixtures/binary_data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713172901,https://api.github.com/repos/simonw/datasette/issues/1034,713172901,MDEyOklzc3VlQ29tbWVudDcxMzE3MjkwMQ==,9599,simonw,2020-10-20T22:19:10Z,2020-10-20T22:20:28Z,OWNER,"I could go with the same format as `datasette-render-binary` but using `0x00` as the format for the hex bytes.

    0x15 0x1C 0x02 0xC7 JFIF 0x00 0x01

Problem with this is that it's ambiguous: if the ASCII characters `0x15` occur in the text they will be indistinguishable from those hex bytes.

But since representing binary data in CSV fundamentally doesn't make sense I'm not sure if that really matters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/741#issuecomment-713171742,https://api.github.com/repos/simonw/datasette/issues/741,713171742,MDEyOklzc3VlQ29tbWVudDcxMzE3MTc0Mg==,9599,simonw,2020-10-20T22:16:25Z,2020-10-20T22:16:25Z,OWNER,See also #992 which will rename `--config` to `--setting`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/262#issuecomment-713170979,https://api.github.com/repos/simonw/datasette/issues/262,713170979,MDEyOklzc3VlQ29tbWVudDcxMzE3MDk3OQ==,9599,simonw,2020-10-20T22:14:37Z,2020-10-20T22:14:37Z,OWNER,"I think it's worth having a plugin hook for this - it can be same hook that is used internally. Maybe `register_extra` - it lets you return one or more `extra` implementations, each with a name and an async function that gets called.

Things like suggested facets will become `register_extra` hooks. Maybe actual facets too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/262#issuecomment-713170284,https://api.github.com/repos/simonw/datasette/issues/262,713170284,MDEyOklzc3VlQ29tbWVudDcxMzE3MDI4NA==,9599,simonw,2020-10-20T22:13:01Z,2020-10-20T22:13:01Z,OWNER,In the documentation for `?_extra=` I think I'll emphasize the comma-separated version of it. Also: there will be `?_extra=` values which act as aliases for collection combinations - e.g. `?_extra=full` will toggle everything.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/262#issuecomment-712988146,https://api.github.com/repos/simonw/datasette/issues/262,712988146,MDEyOklzc3VlQ29tbWVudDcxMjk4ODE0Ng==,9599,simonw,2020-10-20T16:32:02Z,2023-01-17T01:54:13Z,OWNER,"Just realized I added an undocumented `?_extras=` option to the row view years ago and forgot about it - it's not even documented. Added in a30c5b220c15360d575e94b0e67f3255e120b916 - https://latest.datasette.io/fixtures/attraction_characteristic/2.json?_extras=foreign_key_tables

That will need to be made consistent with the new mechanism. I think `?_extra=a&_extra=b` is more consistent with other Datasette features (like `?_facet=col1&_facet=col2`) but potentially quite verbose.

So I could support `?_extra=a,b,c` as an alternative allowed syntax, or I could allow `?_extra=single` and `?_extras=comma,separated`.

I think I prefer allowing commas in `?_extra=`.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/782#issuecomment-712986115,https://api.github.com/repos/simonw/datasette/issues/782,712986115,MDEyOklzc3VlQ29tbWVudDcxMjk4NjExNQ==,9599,simonw,2020-10-20T16:28:46Z,2020-10-20T16:29:51Z,OWNER,"I think this all comes down to how the `?_extras=` mechanism works (see #262), as first hinted at in a30c5b220c15360d575e94b0e67f3255e120b916 (see commit message) when I added this long-forgotten undocumented feature: https://latest.datasette.io/fixtures/attraction_characteristic/2.json?_extras=foreign_key_tables

Extras need to be able to execute additional SQL, since that would solve the problem we have now where the expensive ""suggested facets"" code runs on all `.json` output even when its results are not being shown.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1035#issuecomment-712976314,https://api.github.com/repos/simonw/datasette/issues/1035,712976314,MDEyOklzc3VlQ29tbWVudDcxMjk3NjMxNA==,9599,simonw,2020-10-20T16:21:42Z,2020-10-20T16:21:42Z,OWNER,"Makes me question if `datasette.urls` should grow functionality equivalent to the other path and querystring manipulation methods in `datasette.utils`:

https://github.com/simonw/datasette/blob/66120a7a1cb592e8a21164cf537f62a4d7ab1dfc/datasette/utils/__init__.py#L216-L279","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument",
https://github.com/simonw/datasette/issues/1035#issuecomment-712965574,https://api.github.com/repos/simonw/datasette/issues/1035,712965574,MDEyOklzc3VlQ29tbWVudDcxMjk2NTU3NA==,9599,simonw,2020-10-20T16:13:57Z,2020-10-20T16:13:57Z,OWNER,"That `renderers[key] = path_with_format(` is in a base class which can be used for both arbitrary queries, canned queries and the table view. I think that's OK, but it means that the `format=""json""` argument on `datasette.urls.table()` won't be used by Datasette internally, it will just be available for plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument",
https://github.com/simonw/datasette/issues/1035#issuecomment-712963959,https://api.github.com/repos/simonw/datasette/issues/1035,712963959,MDEyOklzc3VlQ29tbWVudDcxMjk2Mzk1OQ==,9599,simonw,2020-10-20T16:11:25Z,2020-10-20T16:11:25Z,OWNER,"Relevant code:

https://github.com/simonw/datasette/blob/091441a4449beae559a8c0d007376dc85d3aa624/datasette/utils/__init__.py#L681-L696

Only used here:

https://github.com/simonw/datasette/blob/091441a4449beae559a8c0d007376dc85d3aa624/datasette/views/base.py#L498-L502","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument",
https://github.com/simonw/datasette/issues/1026#issuecomment-712962517,https://api.github.com/repos/simonw/datasette/issues/1026,712962517,MDEyOklzc3VlQ29tbWVudDcxMjk2MjUxNw==,9599,simonw,2020-10-20T16:09:12Z,2020-10-20T16:09:12Z,OWNER,"That `datasette.urls.table(""db"", ""table"") + "".json""` example is bad because if the table name contains a `.` it should be `?_format=json` instead.

Maybe `.table()` should have a `format=""json""` option that knows how to do this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1026#issuecomment-712959034,https://api.github.com/repos/simonw/datasette/issues/1026,712959034,MDEyOklzc3VlQ29tbWVudDcxMjk1OTAzNA==,9599,simonw,2020-10-20T16:03:33Z,2020-10-20T16:03:55Z,OWNER,"Reconsidering this: I think the `.get()` etc methods should automatically add the `base_url` prefix for you, since these APIs are only intended to make internal calls.

The clincher on this is when I went to add a section to the `datasette.client` documentation recommending you use `datasette.urls.path()` for every call to them that you make.

But there's a problem: to handle table name escaping users are likely to want to use `datasette.urls.table()` anyway, like this:

    response = await datasette.client.get(datasette.urls.table(""db"", ""table"") + "".json"")

This risks adding the `base_url` prefix twice. Maybe the `.table()` method could return a string-like object that is marked as already having the `base_url` prefix added, so the `client.get()` method knows not to add it again.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1023#issuecomment-712607608,https://api.github.com/repos/simonw/datasette/issues/1023,712607608,MDEyOklzc3VlQ29tbWVudDcxMjYwNzYwOA==,9599,simonw,2020-10-20T05:47:42Z,2020-10-20T05:47:42Z,OWNER,Requested alpha testers in https://github.com/simonw/datasette/issues/838#issuecomment-712604364,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url,
https://github.com/simonw/datasette/issues/1026#issuecomment-712607227,https://api.github.com/repos/simonw/datasette/issues/1026,712607227,MDEyOklzc3VlQ29tbWVudDcxMjYwNzIyNw==,9599,simonw,2020-10-20T05:46:44Z,2020-10-20T05:46:44Z,OWNER,We have a solution for this now: `datasette.urls` from #1033 can be used by plugins to assemble the correct URLs to pass to `.get()` and friends.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1023#issuecomment-712604541,https://api.github.com/repos/simonw/datasette/issues/1023,712604541,MDEyOklzc3VlQ29tbWVudDcxMjYwNDU0MQ==,9599,simonw,2020-10-20T05:39:44Z,2020-10-20T05:39:44Z,OWNER,Here's the alpha with most of this work ready for people to preview: https://github.com/simonw/datasette/releases/tag/0.51a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url,
https://github.com/simonw/datasette/issues/838#issuecomment-712604364,https://api.github.com/repos/simonw/datasette/issues/838,712604364,MDEyOklzc3VlQ29tbWVudDcxMjYwNDM2NA==,9599,simonw,2020-10-20T05:39:15Z,2020-10-20T05:39:15Z,OWNER,"OK, I've made a ton of improvements to how the `base_url` setting works - see tickets linked from #1023. I've just pushed out an alpha release with those changes in it: https://github.com/simonw/datasette/releases/tag/0.51a0

@tsibley @tballison @ChristopherWilks I'd really appreciate your help testing this alpha! 

You can install it with:

    pip install datasette==0.51a0

It should work with just `ProxyPass`, without needing the `ProxyPassReverse` setting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/1033#issuecomment-712529413,https://api.github.com/repos/simonw/datasette/issues/1033,712529413,MDEyOklzc3VlQ29tbWVudDcxMjUyOTQxMw==,9599,simonw,2020-10-20T01:21:12Z,2020-10-20T01:21:12Z,OWNER,Also refs #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method,
https://github.com/simonw/datasette/issues/1025#issuecomment-712525557,https://api.github.com/repos/simonw/datasette/issues/1025,712525557,MDEyOklzc3VlQ29tbWVudDcxMjUyNTU1Nw==,9599,simonw,2020-10-20T01:07:02Z,2020-10-20T01:07:02Z,OWNER,"I fixed the `queries.html` one. I'm not going to fix these two:
```
datasette/templates/error.html:        <a href=""/"">home</a>
datasette/templates/patterns.html:        <a href=""/"">home</a> /
```
Because the `error.html` one does not get passed a context (which makes sense since an error has occurred) and the pattern portfolio doesn't need to link to anywhere in particular.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/904#issuecomment-712524699,https://api.github.com/repos/simonw/datasette/issues/904,712524699,MDEyOklzc3VlQ29tbWVudDcxMjUyNDY5OQ==,9599,simonw,2020-10-20T01:04:12Z,2020-10-20T01:04:12Z,OWNER,Documentation is https://docs.datasette.io/en/latest/writing_plugins.html#building-urls-within-plugins and https://docs.datasette.io/en/latest/internals.html#internals-datasette-urls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/865#issuecomment-712597762,https://api.github.com/repos/simonw/datasette/issues/865,712597762,MDEyOklzc3VlQ29tbWVudDcxMjU5Nzc2Mg==,9599,simonw,2020-10-20T05:22:59Z,2020-10-20T05:22:59Z,OWNER,"OK, this is definitely working now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/1025#issuecomment-712593790,https://api.github.com/repos/simonw/datasette/issues/1025,712593790,MDEyOklzc3VlQ29tbWVudDcxMjU5Mzc5MA==,9599,simonw,2020-10-20T05:12:36Z,2020-10-20T05:12:36Z,OWNER,"I'm going to leave the cookies code setting cookies to default to the `""/""` top level.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/782#issuecomment-712590398,https://api.github.com/repos/simonw/datasette/issues/782,712590398,MDEyOklzc3VlQ29tbWVudDcxMjU5MDM5OA==,9599,simonw,2020-10-20T05:03:46Z,2020-10-20T05:04:09Z,OWNER,"OK, https://latest-with-plugins.datasette.io/ is running that now - e.g. https://latest-with-plugins.datasette.io/fixtures/roadside_attractions.json-preview or https://latest-with-plugins.datasette.io/fixtures/compound_three_primary_keys.json-preview

```json
{
  ""rows"": [
    {
      ""pk"": 1,
      ""name"": ""The Mystery Spot"",
      ""address"": ""465 Mystery Spot Road, Santa Cruz, CA 95065"",
      ""latitude"": 37.0167,
      ""longitude"": -122.0024
    },
    {
      ""pk"": 2,
      ""name"": ""Winchester Mystery House"",
      ""address"": ""525 South Winchester Boulevard, San Jose, CA 95128"",
      ""latitude"": 37.3184,
      ""longitude"": -121.9511
    },
    {
      ""pk"": 3,
      ""name"": ""Burlingame Museum of PEZ Memorabilia"",
      ""address"": ""214 California Drive, Burlingame, CA 94010"",
      ""latitude"": 37.5793,
      ""longitude"": -122.3442
    },
    {
      ""pk"": 4,
      ""name"": ""Bigfoot Discovery Museum"",
      ""address"": ""5497 Highway 9, Felton, CA 95018"",
      ""latitude"": 37.0414,
      ""longitude"": -122.0725
    }
  ],
  ""total"": 4,
  ""next_url"": null
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-712585921,https://api.github.com/repos/simonw/datasette/issues/782,712585921,MDEyOklzc3VlQ29tbWVudDcxMjU4NTkyMQ==,9599,simonw,2020-10-20T04:48:01Z,2020-10-20T04:48:01Z,OWNER,I'll update `datasette-json-preview` with that now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-712585687,https://api.github.com/repos/simonw/datasette/issues/782,712585687,MDEyOklzc3VlQ29tbWVudDcxMjU4NTY4Nw==,9599,simonw,2020-10-20T04:47:02Z,2020-10-20T04:47:12Z,OWNER,"Great point about CORS, I hadn't considered that.

I think I'm going to keep the `Link:` header (added in #1014) because I quite enjoy using it with GitHub and WordPress, but I'm not going to have it be the default way of doing pagination. For the default shape I'm now leaning towards this:

```json
{
  ""total"": 36,
  ""rows"": [{""id"": 1, ""name"": ""Cleo""}],
  ""next_url"": ""https://latest-with-plugins.datasette.io/fixtures/facetable.json?_next=5""
}
```

So three keys: `total`, `rows` and `next_url`. Then extra keys can be added using `?_extra=` with various named bundles.","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1034#issuecomment-712582699,https://api.github.com/repos/simonw/datasette/issues/1034,712582699,MDEyOklzc3VlQ29tbWVudDcxMjU4MjY5OQ==,9599,simonw,2020-10-20T04:36:04Z,2020-10-20T04:36:14Z,OWNER,Asked for ideas on Twitter: https://twitter.com/simonw/status/1318409558805467136,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-712581994,https://api.github.com/repos/simonw/datasette/issues/1034,712581994,MDEyOklzc3VlQ29tbWVudDcxMjU4MTk5NA==,9599,simonw,2020-10-20T04:33:28Z,2020-10-20T04:33:28Z,OWNER,"The [datasette-render-binary](https://github.com/simonw/datasette-render-binary) plugin does this, which I really like - but without the different coloured fonts I'm not sure how readable it would be as just plain text:

![image](https://user-images.githubusercontent.com/9599/96540435-9c125f00-1252-11eb-85aa-5fc8d0e63728.png)

Really the goal here is to find the most human-friendly option, so that people looking at the output have a vague idea what's going on. That's why I'm not leaping at the chance to use base64.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-712580976,https://api.github.com/repos/simonw/datasette/issues/1034,712580976,MDEyOklzc3VlQ29tbWVudDcxMjU4MDk3Ng==,9599,simonw,2020-10-20T04:29:23Z,2020-10-20T04:29:23Z,OWNER,Most obvious option is base64. Any other potential solutions I'm missing?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1025#issuecomment-712579674,https://api.github.com/repos/simonw/datasette/issues/1025,712579674,MDEyOklzc3VlQ29tbWVudDcxMjU3OTY3NA==,9599,simonw,2020-10-20T04:24:10Z,2020-10-20T04:24:10Z,OWNER,"Changed my mind, `error.html` needs access to `urls` in order to link to its CSS file. Passing it after all (it already got passed `ds.config(""base_url"")` so `ds` was available previously).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/904#issuecomment-712483066,https://api.github.com/repos/simonw/datasette/issues/904,712483066,MDEyOklzc3VlQ29tbWVudDcxMjQ4MzA2Ng==,9599,simonw,2020-10-19T22:46:48Z,2020-10-19T22:46:48Z,OWNER,"OK, I'm committing to `datasette.urls.table()` and friends, which will be available to (and documented for use in) plugins and will be exposed to templates as `{{ urls.table(...) }}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/1020#issuecomment-712482504,https://api.github.com/repos/simonw/datasette/issues/1020,712482504,MDEyOklzc3VlQ29tbWVudDcxMjQ4MjUwNA==,9599,simonw,2020-10-19T22:45:01Z,2020-10-19T22:45:01Z,OWNER,"I'm having trouble coming up with the syntax for this. Here's one option:
```python
response = await datasette.client.get(path, request=request)
```
But this feels confusing to me. We're not using the `request=` argument as a request - we're using it as a source of some default request values (the cookies and incoming headers, but not the path).

We're essentially combining that request with the other arguments passed to `.get()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-712482015,https://api.github.com/repos/simonw/datasette/issues/1020,712482015,MDEyOklzc3VlQ29tbWVudDcxMjQ4MjAxNQ==,9599,simonw,2020-10-19T22:43:24Z,2020-10-19T22:43:24Z,OWNER,"... unless I want to support authentication mechanisms that work based on incoming IP address instead, in which case there's an argument for copying more over from the incoming request.

Tailscale is a good example of a system where authentication based on IP address can actually work well, so this is worth doing. Also, there might be authentication mechanisms which work by setting a custom header on the incoming request (not to mention the `Authorization` header).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-712481568,https://api.github.com/repos/simonw/datasette/issues/1020,712481568,MDEyOklzc3VlQ29tbWVudDcxMjQ4MTU2OA==,9599,simonw,2020-10-19T22:41:59Z,2020-10-19T22:41:59Z,OWNER,"It turns out this works just fine:
```python
response = await datasette.client.get(path, cookies=request.cookies)
```
So I don't need a mechanism for this. I'm going to add this to the documentation instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1025#issuecomment-712481127,https://api.github.com/repos/simonw/datasette/issues/1025,712481127,MDEyOklzc3VlQ29tbWVudDcxMjQ4MTEyNw==,9599,simonw,2020-10-19T22:40:37Z,2020-10-20T01:21:36Z,OWNER,Was blocked on #904 - now unblocked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1028#issuecomment-712480866,https://api.github.com/repos/simonw/datasette/issues/1028,712480866,MDEyOklzc3VlQ29tbWVudDcxMjQ4MDg2Ng==,9599,simonw,2020-10-19T22:39:51Z,2020-10-19T22:39:51Z,OWNER,Documentation: https://docs.datasette.io/en/latest/spatialite.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723803777,--load-extension=spatialite shortcut,
https://github.com/simonw/datasette/issues/904#issuecomment-712324077,https://api.github.com/repos/simonw/datasette/issues/904,712324077,MDEyOklzc3VlQ29tbWVudDcxMjMyNDA3Nw==,9599,simonw,2020-10-19T17:39:38Z,2020-10-19T17:39:38Z,OWNER,"If I do these methods I think this should be available on the `datasette` object too, as an internal API for plugins to use to construct redirects and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/1027#issuecomment-712320103,https://api.github.com/repos/simonw/datasette/issues/1027,712320103,MDEyOklzc3VlQ29tbWVudDcxMjMyMDEwMw==,9599,simonw,2020-10-19T17:35:04Z,2020-10-19T17:35:04Z,OWNER,Still need to configure proxying though. https://www.netnea.com/cms/apache-tutorial-9_setting-up-a-reverse-proxy/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/991#issuecomment-712317638,https://api.github.com/repos/simonw/datasette/issues/991,712317638,MDEyOklzc3VlQ29tbWVudDcxMjMxNzYzOA==,9599,simonw,2020-10-19T17:30:56Z,2020-10-19T17:30:56Z,OWNER,https://biglocal.datasettes.com/ is one of my larger Datasettes in terms of number of databases.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/issues/1032#issuecomment-712367285,https://api.github.com/repos/simonw/datasette/issues/1032,712367285,MDEyOklzc3VlQ29tbWVudDcxMjM2NzI4NQ==,9599,simonw,2020-10-19T18:39:32Z,2020-10-19T18:39:32Z,OWNER,https://github.com/digital-land/brownfield-land-collection/blob/a09ddf9960a6af59e72dc02448f7b645e59bf227/bin/harmonise.py#L217-L247 is a beautiful example of this problem.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712365439,https://api.github.com/repos/simonw/datasette/issues/1032,712365439,MDEyOklzc3VlQ29tbWVudDcxMjM2NTQzOQ==,9599,simonw,2020-10-19T18:35:50Z,2020-10-19T18:37:57Z,OWNER,"Maybe I don't need to add `dateutil` as a dependency here?

`pendulum` and `arrow` both depend on it so it's pretty deeply embedded in the Python date ecosystem. Adding it as a dependency seems reasonable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712365236,https://api.github.com/repos/simonw/datasette/issues/1032,712365236,MDEyOklzc3VlQ29tbWVudDcxMjM2NTIzNg==,9599,simonw,2020-10-19T18:35:25Z,2020-10-19T18:35:25Z,OWNER,"Since I'm dealing with tables of data I usually have a whole column of examples, so heuristics that check for numbers-greater-than-12 could actually work well for many cases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712364532,https://api.github.com/repos/simonw/datasette/issues/1032,712364532,MDEyOklzc3VlQ29tbWVudDcxMjM2NDUzMg==,9599,simonw,2020-10-19T18:34:10Z,2020-10-19T18:34:10Z,OWNER,Lots of great example date data in https://biglocal.datasettes.com/COVID_WARN_Notices,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712364317,https://api.github.com/repos/simonw/datasette/issues/1032,712364317,MDEyOklzc3VlQ29tbWVudDcxMjM2NDMxNw==,9599,simonw,2020-10-19T18:33:42Z,2020-10-19T18:33:42Z,OWNER,"Related challenge: timezones. I think I'll punt on those for the moment and just concentrate on dates, but I should keep them in mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712363825,https://api.github.com/repos/simonw/datasette/issues/1032,712363825,MDEyOklzc3VlQ29tbWVudDcxMjM2MzgyNQ==,9599,simonw,2020-10-19T18:32:43Z,2020-10-19T18:32:43Z,OWNER,"Horrible thought: I bet there is data out there that uses more than one date format in the same table! So this needs to be exposed as a visible per-column setting.

Column action menu can help here, although it's not yet the full solution because it isn't yet visible on mobile.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712363344,https://api.github.com/repos/simonw/datasette/issues/1032,712363344,MDEyOklzc3VlQ29tbWVudDcxMjM2MzM0NA==,9599,simonw,2020-10-19T18:31:46Z,2020-10-19T18:31:46Z,OWNER,"As always with dates, the challenge is America. `mm/dd/yy` is indistinguishable from the more sensible `dd/mm/yy`. It's crucially important to visibly tell people which format you assumed, otherwise all kinds of weird invisible bugs can manifest.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/904#issuecomment-712355877,https://api.github.com/repos/simonw/datasette/issues/904,712355877,MDEyOklzc3VlQ29tbWVudDcxMjM1NTg3Nw==,9599,simonw,2020-10-19T18:17:22Z,2020-10-19T18:17:22Z,OWNER,I quite like `datasette.urls.table()` but I'm not sure why.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-712355706,https://api.github.com/repos/simonw/datasette/issues/904,712355706,MDEyOklzc3VlQ29tbWVudDcxMjM1NTcwNg==,9599,simonw,2020-10-19T18:17:03Z,2020-10-19T18:17:03Z,OWNER,"Options:
- `datasette.urls.instance()` (or `datasette.urls.root()`), `datasette.urls.table()` etc
- `datasette.url_for.instance()`...
- `datasette.url.instance()`...
- `datasette.root_url()`, `datasette.table_url()`...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-712354600,https://api.github.com/repos/simonw/datasette/issues/904,712354600,MDEyOklzc3VlQ29tbWVudDcxMjM1NDYwMA==,9599,simonw,2020-10-19T18:15:03Z,2020-10-19T18:15:39Z,OWNER,"Related: #1026 (How should datasette.client interact with base_url)

Also this comment from https://github.com/simonw/datasette/issues/943#issuecomment-675752436

> One thing to consider here: Datasette's table and database name escaping rules can be a little bit convoluted.
> 
> If a plugin wants to get back the first five rows of a table, it will need to construct a URL `/dbname/tablename?_size=5` - but it will need to know how to turn the database and table names into the correctly escaped `dbname` and `tablename` values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/pull/1029#issuecomment-711072903,https://api.github.com/repos/simonw/datasette/issues/1029,711072903,MDEyOklzc3VlQ29tbWVudDcxMTA3MjkwMw==,9599,simonw,2020-10-17T20:04:31Z,2020-10-17T20:04:31Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723837704,fix(docs): broken link,
https://github.com/simonw/datasette/issues/1027#issuecomment-710750038,https://api.github.com/repos/simonw/datasette/issues/1027,710750038,MDEyOklzc3VlQ29tbWVudDcxMDc1MDAzOA==,9599,simonw,2020-10-17T05:00:19Z,2020-10-17T05:00:19Z,OWNER,"Running `apachectl -X` appears to run Apache in the foreground so I can hit Ctrl+C to quit it again, using the config file at `/usr/local/etc/httpd/httpd.conf` and serving the `index.html` file from `/usr/local/var/www` (which confusingly says ""Welcome to nginx"", presumably because I installed nginx via Homebrew earlier).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/991#issuecomment-710694711,https://api.github.com/repos/simonw/datasette/issues/991,710694711,MDEyOklzc3VlQ29tbWVudDcxMDY5NDcxMQ==,9599,simonw,2020-10-16T23:24:03Z,2020-10-16T23:24:03Z,OWNER,I'm really interested in exploring how this page could work with hundreds of database files attached and thousands of total tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/issues/991#issuecomment-710694607,https://api.github.com/repos/simonw/datasette/issues/991,710694607,MDEyOklzc3VlQ29tbWVudDcxMDY5NDYwNw==,9599,simonw,2020-10-16T23:23:34Z,2020-10-16T23:23:34Z,OWNER,Thinking more about pagination and search from #461: if the job of the homepage is to showcase the data that is available in the instance - data that is mostly in tables - maybe it's the tables themselves (and the ability to paginate and search through them) that becomes key to the design of the page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/issues/991#issuecomment-710694144,https://api.github.com/repos/simonw/datasette/issues/991,710694144,MDEyOklzc3VlQ29tbWVudDcxMDY5NDE0NA==,9599,simonw,2020-10-16T23:21:41Z,2020-10-16T23:21:41Z,OWNER,Relevant: https://github.com/simonw/datasette/discussions/1021 asks about controlling the order of databases on that page that have been loaded using configuration directory mode.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/issues/461#issuecomment-710693927,https://api.github.com/repos/simonw/datasette/issues/461,710693927,MDEyOklzc3VlQ29tbWVudDcxMDY5MzkyNw==,9599,simonw,2020-10-16T23:20:43Z,2020-10-16T23:20:43Z,OWNER,Related: redesign homepage entirely #991.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",443021509,Paginate + search for databases/tables on the homepage,
https://github.com/simonw/datasette/issues/468#issuecomment-710693818,https://api.github.com/repos/simonw/datasette/issues/468,710693818,MDEyOklzc3VlQ29tbWVudDcxMDY5MzgxOA==,9599,simonw,2020-10-16T23:20:22Z,2020-10-16T23:20:22Z,OWNER,This is a duplicate of #461.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",444746021,Pagination for the database index page,
https://github.com/simonw/sqlite-utils/issues/136#issuecomment-710428802,https://api.github.com/repos/simonw/sqlite-utils/issues/136,710428802,MDEyOklzc3VlQ29tbWVudDcxMDQyODgwMg==,9599,simonw,2020-10-16T18:56:47Z,2020-10-16T18:56:47Z,OWNER,"To keep the code cleaner, I'm tempted to support this instead:

    --load-extension=spatialite

Where `spatialite` is a special shortcut value that triggers a search for that module in known locations.

Users could still load a module in a file called `spatialite` in the current directory using:

    --load-extension=./spatialite

In fact, `--load-extension=spatialite` could handle that case too by always checking for a file called `spatialite` before attempting to search for it in known locations.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683812642,--load-extension=spatialite shortcut option,
https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710405658,https://api.github.com/repos/simonw/sqlite-utils/issues/69,710405658,MDEyOklzc3VlQ29tbWVudDcxMDQwNTY1OA==,9599,simonw,2020-10-16T18:42:48Z,2020-10-16T18:42:48Z,OWNER,"Did some work on this for #134 in 7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d

I still need to add `--load-extension` to other CLI methods, see #137. Closing this issue in favour of that one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534507142,Feature request: enable extensions loading,
https://github.com/simonw/sqlite-utils/issues/48#issuecomment-710402331,https://api.github.com/repos/simonw/sqlite-utils/issues/48,710402331,MDEyOklzc3VlQ29tbWVudDcxMDQwMjMzMQ==,9599,simonw,2020-10-16T18:41:06Z,2020-10-16T18:41:06Z,OWNER,I could use this demo from JupyterCon 2020 https://gist.github.com/simonw/656c21b5800d5e4624dec9930f00e093,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471818939,"Jupyter notebook demo of the library, launchable on Binder",
https://github.com/simonw/datasette/issues/904#issuecomment-710487083,https://api.github.com/repos/simonw/datasette/issues/904,710487083,MDEyOklzc3VlQ29tbWVudDcxMDQ4NzA4Mw==,9599,simonw,2020-10-16T19:34:10Z,2020-10-19T17:39:04Z,OWNER,"Alternatively, I could expose a single object that knows how to construct all kinds of URLs. Something like this:
```
{{ urls.instance() }}
{{ urls.database(database_name) }}
{{ urls.table(database_name, table_name) }}
etc
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710461468,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710461468,MDEyOklzc3VlQ29tbWVudDcxMDQ2MTQ2OA==,9599,simonw,2020-10-16T19:18:19Z,2020-10-16T19:18:19Z,OWNER,"Reconsidering: #89 was a feature request that relates to this, so maybe this is worth implementing after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/89#issuecomment-710460242,https://api.github.com/repos/simonw/sqlite-utils/issues/89,710460242,MDEyOklzc3VlQ29tbWVudDcxMDQ2MDI0Mg==,9599,simonw,2020-10-16T19:17:27Z,2020-10-16T19:17:50Z,OWNER,"I came up with potential syntax for that here: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550 - based on how `table.extract(...)` works:
```python
fresh_db.table(""tree"", extracts=[Extract(
    columns=(""CommonName"", ""LatinName""),
    table=""Species"",
    fk_column=""species_id"",
    rename={""CommonName"": ""name"", ""LatinName"": ""latin""}
)])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548,Ability to customize columns used by extracts= feature,
https://github.com/simonw/sqlite-utils/issues/187#issuecomment-710456981,https://api.github.com/repos/simonw/sqlite-utils/issues/187,710456981,MDEyOklzc3VlQ29tbWVudDcxMDQ1Njk4MQ==,9599,simonw,2020-10-16T19:15:13Z,2020-10-16T19:15:13Z,OWNER,This is a duplicate of #79.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723460107,Maybe: Utility method / CLI tool for initializing SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/187#issuecomment-710440853,https://api.github.com/repos/simonw/sqlite-utils/issues/187,710440853,MDEyOklzc3VlQ29tbWVudDcxMDQ0MDg1Mw==,9599,simonw,2020-10-16T19:04:19Z,2020-10-16T19:04:19Z,OWNER,I split this off from #136.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723460107,Maybe: Utility method / CLI tool for initializing SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/58#issuecomment-710399593,https://api.github.com/repos/simonw/sqlite-utils/issues/58,710399593,MDEyOklzc3VlQ29tbWVudDcxMDM5OTU5Mw==,9599,simonw,2020-10-16T18:39:31Z,2020-10-16T18:39:31Z,OWNER,I don't think this is valuable enough to justify adding to the library - especially since you can execute FTS search against views by joining to an FTS table built against an underlying table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488293926,Support enabling FTS on views,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710397574,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710397574,MDEyOklzc3VlQ29tbWVudDcxMDM5NzU3NA==,9599,simonw,2020-10-16T18:38:21Z,2020-10-16T18:38:21Z,OWNER,"I'm not going to implement this. I'll leave `extract=...` as it is right now, suitable for quick simple single-column operations on input, but if users want to do something more complicated involving multiple columns they should use the `table.extract()` method after the initial insert instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710395444,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710395444,MDEyOklzc3VlQ29tbWVudDcxMDM5NTQ0NA==,9599,simonw,2020-10-16T18:37:10Z,2020-10-16T18:37:10Z,OWNER,"But this begins to feel too complicated, given that `table.extract()` can already be used to achieve the same thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710393550,MDEyOklzc3VlQ29tbWVudDcxMDM5MzU1MA==,9599,simonw,2020-10-16T18:35:57Z,2020-10-16T18:36:39Z,OWNER,"If I want to support that most complicated example, I think the option to pass a `Extracts()` object to `extracts=` is the best way to do it:
```python
fresh_db.table(""tree"", extracts=[Extract(
    columns=(""CommonName"", ""LatinName""),
    table=""Species"",
    fk_column=""species_id"",
    rename={""CommonName"": ""name"", ""LatinName"": ""latin""}
)])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710390915,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710390915,MDEyOklzc3VlQ29tbWVudDcxMDM5MDkxNQ==,9599,simonw,2020-10-16T18:34:26Z,2020-10-16T18:34:50Z,OWNER,"Here's the most complex example of `.extracts()`:
```python
db[""Trees""].extract(
    [""CommonName"", ""LatinName""],
    table=""Species"",
    fk_column=""species_id"",
    rename={""CommonName"": ""name"", ""LatinName"": ""latin""}
)
```
Resulting in:
```sql
CREATE TABLE [Species] (
    [id] INTEGER PRIMARY KEY,
    [name] TEXT,
    [latin] TEXT
)
```
From https://sqlite-utils.readthedocs.io/en/stable/python-api.html#extracting-columns-into-a-separate-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710364942,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710364942,MDEyOklzc3VlQ29tbWVudDcxMDM2NDk0Mg==,9599,simonw,2020-10-16T18:18:48Z,2020-10-16T18:18:48Z,OWNER,"I think there is. It's a nice existing feature, and I don't think adding tuple support to it would be a huge lift.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710363789,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710363789,MDEyOklzc3VlQ29tbWVudDcxMDM2Mzc4OQ==,9599,simonw,2020-10-16T18:18:05Z,2020-10-16T18:18:05Z,OWNER,I wonder if there's value in extending the `extracts=` option at all given the existence of `table.extract()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710359724,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710359724,MDEyOklzc3VlQ29tbWVudDcxMDM1OTcyNA==,9599,simonw,2020-10-16T18:15:31Z,2020-10-16T18:15:31Z,OWNER,"Using a tuple would work:
```python
fresh_db.table(""tree"", extracts=[(""common_name"", ""latin_name"")])
```
Or to define a custom name:
```python
fresh_db.table(""tree"", extracts={(""common_name"", ""latin_name""): ""names""})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710346830,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710346830,MDEyOklzc3VlQ29tbWVudDcxMDM0NjgzMA==,9599,simonw,2020-10-16T18:08:52Z,2020-10-16T18:09:21Z,OWNER,"The new `.extract()` method can handle multiple columns:

https://github.com/simonw/sqlite-utils/blob/2c541fac352632e23e40b0d21e3f233f7a744a57/tests/test_extract.py#L70-L87","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710258736,https://api.github.com/repos/simonw/sqlite-utils/issues/182,710258736,MDEyOklzc3VlQ29tbWVudDcxMDI1ODczNg==,9599,simonw,2020-10-16T17:20:41Z,2020-10-16T17:20:41Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#inserting-csv-or-tsv-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""",
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-710198162,https://api.github.com/repos/simonw/sqlite-utils/issues/186,710198162,MDEyOklzc3VlQ29tbWVudDcxMDE5ODE2Mg==,9599,simonw,2020-10-16T16:41:00Z,2020-10-16T16:41:00Z,OWNER,"Failing test:
```python
def test_extract_null_values(fresh_db):
    fresh_db[""species""].insert({""id"": 1, ""species"": ""Wolf""}, pk=""id"")
    fresh_db[""individuals""].insert_all(
        [
            {""id"": 10, ""name"": ""Terriana"", ""species"": ""Fox""},
            {""id"": 11, ""name"": ""Spenidorm"", ""species"": None},
            {""id"": 12, ""name"": ""Grantheim"", ""species"": ""Wolf""},
            {""id"": 13, ""name"": ""Turnutopia"", ""species"": None},
            {""id"": 14, ""name"": ""Wargal"", ""species"": ""Wolf""},
        ],
        pk=""id"",
    )
    fresh_db[""individuals""].extract(""species"")
    assert fresh_db[""species""].schema == (
        ""CREATE TABLE [species] (\n""
        ""   [id] INTEGER PRIMARY KEY,\n""
        ""   [species] TEXT\n""
        "")""
    )
    assert fresh_db[""individuals""].schema == (
        'CREATE TABLE ""individuals"" (\n'
        ""   [id] INTEGER PRIMARY KEY,\n""
        ""   [name] TEXT,\n""
        ""   [species_id] INTEGER,\n""
        ""   FOREIGN KEY(species_id) REFERENCES species(id)\n""
        "")""
    )
    assert list(fresh_db[""species""].rows) == [
        {""id"": 1, ""species"": ""Wolf""},
        {""id"": 2, ""species"": ""Fox""},
    ]
    assert list(fresh_db[""individuals""].rows) == [
        {""id"": 10, ""name"": ""Terriana"", ""species_id"": 2},
        {""id"": 11, ""name"": ""Spenidorm"", ""species_id"": None},
        {""id"": 12, ""name"": ""Grantheim"", ""species_id"": 1},
        {""id"": 13, ""name"": ""Turnutopia"", ""species_id"": None},
        {""id"": 14, ""name"": ""Wargal"", ""species_id"": 1},
    ]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710178871,https://api.github.com/repos/simonw/sqlite-utils/issues/182,710178871,MDEyOklzc3VlQ29tbWVudDcxMDE3ODg3MQ==,9599,simonw,2020-10-16T16:27:39Z,2020-10-16T16:28:14Z,OWNER,"The file is opened for me by `click.File()`, which also handles things like `-` for stdin. But i neee to be able to switch the encoding used to read from that based on the `--encoding` option.

I think the way to do that is to open the file in binary mode and then wrap it in a codec reader:
```python
fp = codecs.getreader(encoding)(fp)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""",
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706260,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709706260,MDEyOklzc3VlQ29tbWVudDcwOTcwNjI2MA==,9599,simonw,2020-10-16T03:17:02Z,2020-10-16T03:17:17Z,OWNER,Actually I think this should be an option to `.extract()` which controls if nulls are extracted or left alone. Maybe called `extract_nulls=True/False`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706065,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709706065,MDEyOklzc3VlQ29tbWVudDcwOTcwNjA2NQ==,9599,simonw,2020-10-16T03:16:22Z,2020-10-16T03:16:22Z,OWNER,Either way I think I'm going to need to add some SQL which uses `where a = b or (a is null and b is null)`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705885,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709705885,MDEyOklzc3VlQ29tbWVudDcwOTcwNTg4NQ==,9599,simonw,2020-10-16T03:15:39Z,2020-10-16T03:15:39Z,OWNER,The alternative solution here would be that a single `null` value DOES get extracted. To implement this I would need to add some logic that uses `is null` instead of `=`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705624,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709705624,MDEyOklzc3VlQ29tbWVudDcwOTcwNTYyNA==,9599,simonw,2020-10-16T03:14:39Z,2020-10-16T03:14:39Z,OWNER,"How should this work with extractions covering multiple columns?

If there's a single column then it makes sense that a `null` value would not be extracted into the lookup table, but would instead become stay as `null`.

For a multiple column extraction, provided at least one of those columns is not null It should map to a record in the lookup table. Only if ALL of the extracted columns are null should the lookup value stay null.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/datasette/issues/1026#issuecomment-709636372,https://api.github.com/repos/simonw/datasette/issues/1026,709636372,MDEyOklzc3VlQ29tbWVudDcwOTYzNjM3Mg==,9599,simonw,2020-10-15T23:09:34Z,2020-10-15T23:09:34Z,OWNER,"I'm inclined to say that internal requests should ignore `base_url` - since that seems like the right thing for plugins that need to access default Datasette APIs.

The one catch here is plugins that might want to proxy the current incoming URL for some reason - where that incoming `request.path` could include the `base_url`.

Actually those should be fine - because it will have been stripped off earlier:

https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/app.py#L963-L968","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/904#issuecomment-709635276,https://api.github.com/repos/simonw/datasette/issues/904,709635276,MDEyOklzc3VlQ29tbWVudDcwOTYzNTI3Ng==,9599,simonw,2020-10-15T23:05:54Z,2020-10-15T23:05:54Z,OWNER,Could have `instance_url()` take an optional path argument which is then turned into the correct path.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-709635021,https://api.github.com/repos/simonw/datasette/issues/904,709635021,MDEyOklzc3VlQ29tbWVudDcwOTYzNTAyMQ==,9599,simonw,2020-10-15T23:05:11Z,2020-10-15T23:05:11Z,OWNER,"I think this should be a family of functions:

- `instance_url()` - the root URL of the instance (usually `/` unless `base_url` is set)
- `database_url(database_name)` - already got this
- `table_url(database_name, table_name)`
- `row_url(database_name, table_name, row)` - not sure about this one. The idea would be for `row` to be correctly turned into a URL by introspecting the primary keys for that table, then pulling those values out of the SQLite `row` object. Might not be necessary though.

I also need a way for plugins to link to e.g. `/-/configure-fts` - or even `/-/configure-fts/database-name/table-name`. What should that look like?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-709634261,https://api.github.com/repos/simonw/datasette/issues/904,709634261,MDEyOklzc3VlQ29tbWVudDcwOTYzNDI2MQ==,9599,simonw,2020-10-15T23:02:43Z,2020-10-15T23:02:43Z,OWNER,"Here's the current implementation of `database_url` - on the `BaseView` class, but only because it needs access to a `datasette` instance (to read `base_url`):
https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/datasette/views/base.py#L102-L108","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-709633823,https://api.github.com/repos/simonw/datasette/issues/904,709633823,MDEyOklzc3VlQ29tbWVudDcwOTYzMzgyMw==,9599,simonw,2020-10-15T23:01:13Z,2020-10-15T23:01:13Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/988#issuecomment-709633762,https://api.github.com/repos/simonw/datasette/issues/988,709633762,MDEyOklzc3VlQ29tbWVudDcwOTYzMzc2Mg==,9599,simonw,2020-10-15T23:01:01Z,2020-10-15T23:01:01Z,OWNER,This is a dupe of https://github.com/simonw/datasette/issues/904,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",713209404,Mechanism for plugins to construct URLs that respect base_url,
https://github.com/simonw/datasette/issues/865#issuecomment-709633080,https://api.github.com/repos/simonw/datasette/issues/865,709633080,MDEyOklzc3VlQ29tbWVudDcwOTYzMzA4MA==,9599,simonw,2020-10-15T22:58:51Z,2020-10-15T22:58:51Z,OWNER,"It looks like there are places where Datasette might return a redirect that doesn't take `base_url` into account - I'm planning on fixing those here, after which I think `ProxyPassReverse` should no longer be necessary. https://github.com/simonw/datasette/issues/1025#issuecomment-709632136","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/900#issuecomment-709632765,https://api.github.com/repos/simonw/datasette/issues/900,709632765,MDEyOklzc3VlQ29tbWVudDcwOTYzMjc2NQ==,9599,simonw,2020-10-15T22:57:55Z,2020-10-15T22:57:55Z,OWNER,"I believe this particular bug has been fixed, based on my testing here: https://github.com/simonw/datasette/issues/1024#issuecomment-709622973

Please re-open the ticket if you are still experiencing it.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",661605489,Some links don't honor base_url,
https://github.com/simonw/datasette/issues/1025#issuecomment-709632314,https://api.github.com/repos/simonw/datasette/issues/1025,709632314,MDEyOklzc3VlQ29tbWVudDcwOTYzMjMxNA==,9599,simonw,2020-10-15T22:56:25Z,2020-10-15T22:56:34Z,OWNER,"That `utils/asgi.py` line is the default path for setting cookies. That should likely take `base_url` into account too:
https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/utils/asgi.py#L331-L342","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-709632136,https://api.github.com/repos/simonw/datasette/issues/1025,709632136,MDEyOklzc3VlQ29tbWVudDcwOTYzMjEzNg==,9599,simonw,2020-10-15T22:55:44Z,2020-10-15T22:55:44Z,OWNER,"It looks like there are also some generated redirect responses that don't take `base_url` into account:
```
datasette % git grep '""/' -- '*.py' ':(exclude)*test_*.py' ':(exclude)datasette/app.py'
datasette/_version.py:        for i in cfg.versionfile_source.split(""/""):
datasette/utils/asgi.py:        path=""/"",
datasette/views/base.py:            should_redirect = ""/{}-{}"".format(name, expected)
datasette/views/base.py:                should_redirect += ""/"" + urllib.parse.quote_plus(kwargs[""table""])
datasette/views/base.py:                should_redirect += ""/"" + kwargs[""pk_path""]
datasette/views/special.py:            response = Response.redirect(""/"")
datasette/views/special.py:            return Response.redirect(""/"")
datasette/views/special.py:        response = Response.redirect(""/"")
datasette/views/special.py:        return Response.redirect(""/"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-709629920,https://api.github.com/repos/simonw/datasette/issues/1025,709629920,MDEyOklzc3VlQ29tbWVudDcwOTYyOTkyMA==,9599,simonw,2020-10-15T22:48:20Z,2020-10-15T22:48:20Z,OWNER,"Also these:
```
datasette % git grep '""/' -- '*.html' ':(exclude)*/patterns.html'
datasette/templates/allow_debug.html:<form action=""/-/allow-debug"" method=""get"">
datasette/templates/base.html:        <form action=""/-/logout"" method=""post"">
datasette/templates/error.html:        <a href=""/"">home</a>
datasette/templates/logout.html:<form action=""/-/logout"" method=""post"">
datasette/templates/messages_debug.html:<form action=""/-/messages"" method=""post"">
datasette/templates/query.html:        <a href=""/"">home</a> /
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/865#issuecomment-709626786,https://api.github.com/repos/simonw/datasette/issues/865,709626786,MDEyOklzc3VlQ29tbWVudDcwOTYyNjc4Ng==,9599,simonw,2020-10-15T22:38:38Z,2020-10-15T22:38:38Z,OWNER,"I managed to recreate proxying using `nginx` in #1024 - but I could not replicate this bug. I did NOT use `ProxyPassReverse` though. I think that may be what caused the problem.

I'll add a section to the documentation about this shortly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/1024#issuecomment-709625063,https://api.github.com/repos/simonw/datasette/issues/1024,709625063,MDEyOklzc3VlQ29tbWVudDcwOTYyNTA2Mw==,9599,simonw,2020-10-15T22:33:22Z,2020-10-15T22:33:22Z,OWNER,"Of those errors...

`http://localhost:8000/robots.txt` 404 is fine.

`http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22` looks to me like a `wget` parsing bug where it got confused by this JavaScript:

```
window.DATASETTE_CLUSTER_MAP_TILE_LAYER_OPTIONS = {""maxZoom"": 19, ""detectRetina"": true, ""attribution"": ""&copy; <a href=\""https://www.openstreetmap.org/copyright\"">OpenStreetMap</a> contributors""};</script>
```

`http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js` is a real bug. It's a bug in `datasette-cluster-map` but also requires me to solve #988 - mechanism for plugins to construct URLs that obey `base_url`.

I'm not sure why I'm getting a hit to `http://localhost:8000/` since I wouldn't expect to link to `/` anywhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709622973,https://api.github.com/repos/simonw/datasette/issues/1024,709622973,MDEyOklzc3VlQ29tbWVudDcwOTYyMjk3Mw==,9599,simonw,2020-10-15T22:27:31Z,2020-10-15T22:27:31Z,OWNER,"Here's how I tested it:
```
time wget -r 'http://localhost:8000/datasette/' 2>&1 | grep -i -C 5 ""failed\|error"" > /tmp/errors.txt
```
This wrote out any errors (plus context) to the `errors.txt` log - and reported that the full crawl took 33s.

Here's what I got in `errors.txt`:
```

     0K .                                                      71.6M=0s

2020-10-15 15:23:09 (71.6 MB/s) - ‘localhost:8000/datasette/index.html’ saved [1276]

Loading robots.txt; please ignore errors.
--2020-10-15 15:23:09--  http://localhost:8000/robots.txt
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
--
--2020-10-15 15:23:09--  http://localhost:8000/robots.txt
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:09 ERROR 404: Not Found.

--2020-10-15 15:23:09--  http://localhost:8000/datasette/-/static/app.css?b576be
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: 8563 (8.4K) [text/css]
--
--
2020-10-15 15:23:13 (7.90 MB/s) - ‘localhost:8000/datasette/fixtures/primary_key_multiple_columns_explicit_label.json?_shape=object’ saved [58]

--2020-10-15 15:23:13--  http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:13 ERROR 404: Not Found.

--2020-10-15 15:23:13--  http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+limit+101
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
--
--
2020-10-15 15:23:13 (84.3 MB/s) - ‘localhost:8000/datasette/fixtures/roadside_attractions.json?_shape=object’ saved [619]

--2020-10-15 15:23:13--  http://localhost:8000/datasette/fixtures/%5C%22https://www.openstreetmap.org/copyright%5C%22
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:13 ERROR 404: Not Found.

--2020-10-15 15:23:13--  http://localhost:8000/datasette/fixtures?sql=select+pk%2C+text1%2C+text2%2C+%5Bname+with+.+and+spaces%5D+from+searchable+order+by+pk+limit+101
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
--
--
2020-10-15 15:23:14 (28.6 MB/s) - ‘localhost:8000/datasette/fixtures/searchable_view_configured_by_metadata.json?_shape=array&_nl=on’ saved [180]

--2020-10-15 15:23:14--  http://localhost:8000/
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:14 ERROR 404: Not Found.

--2020-10-15 15:23:14--  http://localhost:8000/datasette/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101&_hide_sql=1
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
--
--
2020-10-15 15:23:21 (64.1 MB/s) - ‘localhost:8000/datasette/fixtures.csv?sql=select+pk,+name,+address,+latitude,+longitude+from+roadside_attractions+order+by+pk+limit+101&_size=max’ saved [403]

--2020-10-15 15:23:21--  http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:21 ERROR 404: Not Found.

--2020-10-15 15:23:21--  http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+desc+limit+101
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709600335,https://api.github.com/repos/simonw/datasette/issues/1024,709600335,MDEyOklzc3VlQ29tbWVudDcwOTYwMDMzNQ==,9599,simonw,2020-10-15T21:28:02Z,2020-10-15T22:25:43Z,OWNER,"This is working OK so far:

<img width=""897"" alt=""fixtures__compound_three_primary_keys__1_001_rows_where_sorted_by_pk2"" src=""https://user-images.githubusercontent.com/9599/96187878-96371980-0ef2-11eb-875e-5b1ae9b445fd.png"">

I'll try crawling it with `wget -r` to see if I get any errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1027#issuecomment-709647525,https://api.github.com/repos/simonw/datasette/issues/1027,709647525,MDEyOklzc3VlQ29tbWVudDcwOTY0NzUyNQ==,9599,simonw,2020-10-15T23:49:51Z,2020-10-15T23:51:39Z,OWNER,"I'll install Apache on macOS to figure this out using https://formulae.brew.sh/formula/httpd

`brew install httpd` output this at the end:

```
==> httpd
DocumentRoot is /usr/local/var/www.

The default ports have been set in /usr/local/etc/httpd/httpd.conf to 8080 and in
/usr/local/etc/httpd/extra/httpd-ssl.conf to 8443 so that httpd can run without sudo.

To have launchd start httpd now and restart at login:
  brew services start httpd
Or, if you don't want/need a background service you can just run:
  apachectl start
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/1027#issuecomment-709646865,https://api.github.com/repos/simonw/datasette/issues/1027,709646865,MDEyOklzc3VlQ29tbWVudDcwOTY0Njg2NQ==,9599,simonw,2020-10-15T23:47:08Z,2020-10-15T23:47:08Z,OWNER,It should cover both nginx and Apache. nginx config is here: https://github.com/simonw/datasette/issues/1024#issuecomment-709598324,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/894#issuecomment-709539257,https://api.github.com/repos/simonw/datasette/issues/894,709539257,MDEyOklzc3VlQ29tbWVudDcwOTUzOTI1Nw==,9599,simonw,2020-10-15T19:19:29Z,2020-10-15T19:34:07Z,OWNER,"Urgh this isn't going to work. `%7E~%7E` gets decoded as `~~~` so I wouldn't be able to tell the difference.

I could use double-percentage-encoding here instead. I feel like there's a simpler solution that I'm missing (and that may well be in use within Datasette already, I'm not doing great thinking this morning).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709534197,https://api.github.com/repos/simonw/datasette/issues/894,709534197,MDEyOklzc3VlQ29tbWVudDcwOTUzNDE5Nw==,9599,simonw,2020-10-15T19:08:53Z,2020-10-15T19:17:55Z,OWNER,"Even better solution: use URL encoding in the parameter details. This is consistent with how `?_next=` tokens work, e.g. `?_next=0.291861560261786%2Ce%2Cj`.

So the format can be:

- `mycolumn`
- `urlencoded-mycolumn$castname`

For most columns this will look like: `?_sort=score$numeric`

For columns with a `$` in their name it will be `?_sort=score%24hasdollar$numeric`

Problem: both `$` and `,` are usually URL encoded anyway. I need a character which isn't encoded by default, so that I can use its encoded form to show it is part of the column name and its un-encoded form to split the cast indicator.

`_` is a candidate here - not encoded by default, but can be encoded as `%5F`.

The other unreserved non-alphanumeric characters are `-`, `.`, `_`, `~`.

Of these, `~` is least likely to show up in a column name. So I'll use that.

- `mycolumn`
- `mycolumn~numeric`
- `mycolumn%7Ewith%7Etildes~numeric`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709532369,https://api.github.com/repos/simonw/datasette/issues/894,709532369,MDEyOklzc3VlQ29tbWVudDcwOTUzMjM2OQ==,9599,simonw,2020-10-15T19:05:07Z,2020-10-15T19:07:35Z,OWNER,"Simpler option: `?_sort=` column values look like this:

- `mycolumn` - for sort by column
- `mycolumn$numeric` - for sort by column after cast to float
- `mycolumn$morename$default` - for the edge case where the column name itself contains a $ symbol","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709531343,https://api.github.com/repos/simonw/datasette/issues/894,709531343,MDEyOklzc3VlQ29tbWVudDcwOTUzMTM0Mw==,9599,simonw,2020-10-15T19:03:12Z,2020-10-15T19:03:12Z,OWNER,"The Sort by `<select>` menu needs a rethink: https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/templates/table.html#L80-L87

If it's going to include sort by numeric options it needs a different format - since `?_sort=colname` needs to also support `?_sort=numeric_colname` - but shouldn't clash with a weird column already named `numeric_colname`.

Can I come up with a value syntax for this that is guaranteed not to clash with a weirdly named existing column?

I think so. I could use `{""type"": ""numeric"", ""column"": ""mycolumn""}` as the value - then if a column has a name that is itself valid JSON (weird but possible) the `_sort=` value would be `{""type"": ""default"", ""column"": ""{\""type\"": \""numeric\"", \""name\"": \""column\""}""}`.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709525082,https://api.github.com/repos/simonw/datasette/issues/894,709525082,MDEyOklzc3VlQ29tbWVudDcwOTUyNTA4Mg==,9599,simonw,2020-10-15T18:51:59Z,2020-10-15T18:51:59Z,OWNER,This is enough of a design to build a working prototype.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709524123,https://api.github.com/repos/simonw/datasette/issues/894,709524123,MDEyOklzc3VlQ29tbWVudDcwOTUyNDEyMw==,9599,simonw,2020-10-15T18:50:25Z,2020-10-15T18:50:25Z,OWNER,"For the ""Sort by X"" select menu case... I could automatically expand that menu to contain extra options for ""Sort numerically by X"" for each TEXT column in the table. That's a pretty good option.

For the action cog menu, I can add the extra options to the cog menu - and rely on the fact that the title of the page will say ""Sorted numerically by colname descending"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709513483,https://api.github.com/repos/simonw/datasette/issues/894,709513483,MDEyOklzc3VlQ29tbWVudDcwOTUxMzQ4Mw==,9599,simonw,2020-10-15T18:31:56Z,2020-10-15T18:31:56Z,OWNER,"I think the first version of this feature involves implementing `?_sort_numeric=col` and `?_sort_numeric_desc=col` plus the JavaScript to detect if those values should be shown in the column actions menu.

One question: how to reflect that this is happening in the current sort UI. This menu here for example:

<img width=""440"" alt=""fixtures__compound_three_primary_keys__1_001_rows_where_sorted_by_pk3_descending"" src=""https://user-images.githubusercontent.com/9599/96171519-e35ac180-0ed9-11eb-9945-6780054e4ad5.png"">

And this interface: how should it indicate that a text is currently sorted numerically v.s. sorted alphabetically, and allow the user to switch from one to the other?

<img width=""794"" alt=""fixtures__sortable__201_rows_where_sorted_by_sortable_with_nulls_descending"" src=""https://user-images.githubusercontent.com/9599/96171618-04bbad80-0eda-11eb-946c-ed3b17a8c749.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709511399,https://api.github.com/repos/simonw/datasette/issues/894,709511399,MDEyOklzc3VlQ29tbWVudDcwOTUxMTM5OQ==,9599,simonw,2020-10-15T18:28:09Z,2020-10-15T18:28:09Z,OWNER,"The simplest solution would be for Python code to scan all of the visible values on the current page and show the column action for ""sort by this numeric"" based purely on that. I already do that in the JavaScript for ""are there any blank values in the first page?"" here:

https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/static/table.js#L106-L118

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709510422,https://api.github.com/repos/simonw/datasette/issues/894,709510422,MDEyOklzc3VlQ29tbWVudDcwOTUxMDQyMg==,9599,simonw,2020-10-15T18:26:25Z,2020-10-15T18:26:25Z,OWNER,"There's something interesting about figuring out which sort options should be offered in the column actions menu.

Two options:

- Try to detect if a text column is all-integers or all-floats, either by scanning the entire table (if it's small enough) or by scanning the first X rows, where X might be the size of the first page or maybe the first 1,000 or similar.
- Could also let users define this in `metadata.yml` for the table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709509635,https://api.github.com/repos/simonw/datasette/issues/894,709509635,MDEyOklzc3VlQ29tbWVudDcwOTUwOTYzNQ==,9599,simonw,2020-10-15T18:24:57Z,2020-10-15T18:24:57Z,OWNER,"This does feel like a weird plugin hook just because there aren't really THAT many different use-cases that plugins could solve. The ones I can think of are:

- Sort numeric
- Sort by parsed date
- Sort by FTS rank

Could this work if I just allow `_sort_clause=`?

One possible solution for the no-arbitrary-SQL case: users can define sort orders in `metadata.json/yml`. So if you want to enable sort-by-distance without enabling arbitrary SQL you could add something like this:

```yaml
databases:
  mydb:
    tables:
      museums:
        sort_clause: bm25(fts)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709505953,https://api.github.com/repos/simonw/datasette/issues/894,709505953,MDEyOklzc3VlQ29tbWVudDcwOTUwNTk1Mw==,9599,simonw,2020-10-15T18:20:13Z,2020-10-15T18:20:13Z,OWNER,Sorting by date when the column has a junk date format in it is such a column need it should maybe ship in Datasette by default - though I've been trying to avoid adding heavy dependencies like `dateutil` if I can get away with it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709505608,https://api.github.com/repos/simonw/datasette/issues/894,709505608,MDEyOklzc3VlQ29tbWVudDcwOTUwNTYwOA==,9599,simonw,2020-10-15T18:19:35Z,2020-10-15T18:19:35Z,OWNER,"I could even let plugins define new sort types. Imagine a plugin that enables this:

`?_sort_date_desc=mycol` - where it knows how to handle specific date formats, or even uses `dateutil.parser.parse`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709505147,https://api.github.com/repos/simonw/datasette/issues/894,709505147,MDEyOklzc3VlQ29tbWVudDcwOTUwNTE0Nw==,9599,simonw,2020-10-15T18:18:45Z,2020-10-15T18:50:50Z,OWNER,"This needs querystring parameter design. Some options:

* `?_sort_clause=cast(mycol+as+integer)` and `?_sort_clause_desc=cast(mycol+as+integer)` - allowing any expression. This would need to be disabled if arbitrary SQL was turned off, similar to the restrictions on `?_where=`.
* `?_sort_numeric=mycol` and `?_sort_numeric_desc=mycol` - this would cast to `real` which would work on integer values as well. It could be allowed even when arbitrary SQL was disabled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/878#issuecomment-709503359,https://api.github.com/repos/simonw/datasette/issues/878,709503359,MDEyOklzc3VlQ29tbWVudDcwOTUwMzM1OQ==,9599,simonw,2020-10-15T18:15:28Z,2020-10-15T18:15:28Z,OWNER,I think this is blocking #619,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-709502889,https://api.github.com/repos/simonw/datasette/issues/878,709502889,MDEyOklzc3VlQ29tbWVudDcwOTUwMjg4OQ==,9599,simonw,2020-10-15T18:14:34Z,2020-10-15T18:14:34Z,OWNER,"The `BaseView` class does this for Datasette internals at the moment, but I'm not convinced it works as well as it could.

I'd like to turn this into a class that is documented and available to plugins as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/690#issuecomment-709500715,https://api.github.com/repos/simonw/datasette/issues/690,709500715,MDEyOklzc3VlQ29tbWVudDcwOTUwMDcxNQ==,9599,simonw,2020-10-15T18:10:32Z,2020-10-15T18:10:32Z,OWNER,"Row action menus are a question mark for me. Adding them to the table page itself could get really noisy - though maybe they're a vertical ... menu at the end of the row? They could also go on the `row.html` template, though that page is rarely used at the moment.

I'm going to skip row actions for the moment and concentrate on the other four, which I know I have use-cases for.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/issues/1024#issuecomment-709598324,https://api.github.com/repos/simonw/datasette/issues/1024,709598324,MDEyOklzc3VlQ29tbWVudDcwOTU5ODMyNA==,9599,simonw,2020-10-15T21:23:33Z,2020-10-15T21:26:55Z,OWNER,"Combining these two examples, here's the config file I am going to use for this. I'll save this as `nginx.conf`:
```
daemon off;

events {
  worker_connections  1024;
}

http {
  server {
    listen 8000;

    location /datasette {
        proxy_pass              http://127.0.0.1:8001;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    }
  }
}
```
Then start the server with:
```
nginx -p `pwd` -c `pwd`/nginx.conf
```
And start Datasette like this:
```
datasette fixtures.db --config base_url:/datasette/
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709597589,https://api.github.com/repos/simonw/datasette/issues/1024,709597589,MDEyOklzc3VlQ29tbWVudDcwOTU5NzU4OQ==,9599,simonw,2020-10-15T21:21:53Z,2020-10-15T21:23:25Z,OWNER,"Here's a recipe for running nginx against a custom config file: https://gist.github.com/simonw/35f0ebf9c1d6df158759

```
daemon off;

events {
  worker_connections  1024;
}

http {
  access_log /dev/stdout;
  error_log /dev/stderr;

  types {
    text/html                             html htm shtml;
    text/css                              css;
    image/gif                             gif;
    image/jpeg                            jpeg jpg;
    application/javascript                js;
  }

  server {
    listen 8002;
    index index.html;
    root  app;
  }
}
```
```
nginx -p `pwd` -c `pwd`/nginx.conf
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709595960,https://api.github.com/repos/simonw/datasette/issues/1024,709595960,MDEyOklzc3VlQ29tbWVudDcwOTU5NTk2MA==,9599,simonw,2020-10-15T21:18:14Z,2020-10-15T21:18:14Z,OWNER,Typing `nginx` starts it running as a daemon listening on port `http-alt` aka 8080. It uses the config file from ` /usr/local/etc/nginx/nginx.conf`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709590941,https://api.github.com/repos/simonw/datasette/issues/1024,709590941,MDEyOklzc3VlQ29tbWVudDcwOTU5MDk0MQ==,9599,simonw,2020-10-15T21:07:47Z,2020-10-15T21:07:47Z,OWNER,On macOS I ran `brew install nginx`. I'm going to try running it on port 8000 so I don't have to run it as root.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709590337,https://api.github.com/repos/simonw/datasette/issues/1024,709590337,MDEyOklzc3VlQ29tbWVudDcwOTU5MDMzNw==,9599,simonw,2020-10-15T21:06:24Z,2020-10-15T21:07:19Z,OWNER,"From https://stackoverflow.com/questions/32549684/nginx-proxy-and-remove-proxy-pass-prefix/32550251 it looks like the config I should use is:
```
server {
    listen 80;
    server_name example.com;

    location /datasette/ {
        proxy_pass              http://127.0.0.1:8001;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        proxy_read_timeout      90;
    }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709589297,https://api.github.com/repos/simonw/datasette/issues/1024,709589297,MDEyOklzc3VlQ29tbWVudDcwOTU4OTI5Nw==,9599,simonw,2020-10-15T21:04:31Z,2020-10-15T21:04:31Z,OWNER,I think nginx or Apache would be the best tools for this. I'm inclined to try with nginx first since I know it better.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/838#issuecomment-709588425,https://api.github.com/repos/simonw/datasette/issues/838,709588425,MDEyOklzc3VlQ29tbWVudDcwOTU4ODQyNQ==,9599,simonw,2020-10-15T21:02:37Z,2020-10-15T21:02:37Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/900#issuecomment-709588322,https://api.github.com/repos/simonw/datasette/issues/900,709588322,MDEyOklzc3VlQ29tbWVudDcwOTU4ODMyMg==,9599,simonw,2020-10-15T21:02:26Z,2020-10-15T21:02:26Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",661605489,Some links don't honor base_url,
https://github.com/simonw/datasette/issues/865#issuecomment-709588373,https://api.github.com/repos/simonw/datasette/issues/865,709588373,MDEyOklzc3VlQ29tbWVudDcwOTU4ODM3Mw==,9599,simonw,2020-10-15T21:02:31Z,2020-10-15T21:02:31Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/988#issuecomment-709588290,https://api.github.com/repos/simonw/datasette/issues/988,709588290,MDEyOklzc3VlQ29tbWVudDcwOTU4ODI5MA==,9599,simonw,2020-10-15T21:02:21Z,2020-10-15T21:02:21Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",713209404,Mechanism for plugins to construct URLs that respect base_url,
https://github.com/simonw/datasette/issues/894#issuecomment-709575818,https://api.github.com/repos/simonw/datasette/issues/894,709575818,MDEyOklzc3VlQ29tbWVudDcwOTU3NTgxOA==,9599,simonw,2020-10-15T20:35:03Z,2020-10-15T20:35:03Z,OWNER,"Prototype so far:
```diff
diff --git a/datasette/views/table.py b/datasette/views/table.py
index ea11a51..d61f8bd 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -497,17 +497,32 @@ class TableView(RowTableShared):
         if sort and sort_desc:
             raise DatasetteError(""Cannot use _sort and _sort_desc at the same time"")
 
+        def parse_sort(sort):
+            if ""~"" in sort:
+                if sort.endswith(""~default""):
+                    col = sort.rsplit(""~"", 1)[0]
+                    return col, escape_sqlite(col)
+                elif sort.endswith(""~numeric""):
+                    col = sort.rsplit(""~"", 1)[0]
+                    return col, ""cast(nullif({}, '') as real)"".format(escape_sqlite(col))
+                else:
+                    return sort, escape_sqlite(sort)
+            else:
+                return sort, escape_sqlite(sort)
+
         if sort:
-            if sort not in sortable_columns:
-                raise DatasetteError(""Cannot sort table by {}"".format(sort))
+            sort_column, sort_clause = parse_sort(sort)
+            if sort_column not in sortable_columns:
+                raise DatasetteError(""Cannot sort table by {}"".format(sort_column))
 
-            order_by = escape_sqlite(sort)
+            order_by = sort_clause
 
         if sort_desc:
-            if sort_desc not in sortable_columns:
-                raise DatasetteError(""Cannot sort table by {}"".format(sort_desc))
+            sort_column, sort_clause = parse_sort(sort_desc)
+            if sort_column not in sortable_columns:
+                raise DatasetteError(""Cannot sort table by {}"".format(sort_column))
 
-            order_by = ""{} desc"".format(escape_sqlite(sort_desc))
+            order_by = ""{} desc"".format(sort_clause)
 
         from_sql = ""from {table_name} {where}"".format(
             table_name=escape_sqlite(table),
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709572425,https://api.github.com/repos/simonw/datasette/issues/894,709572425,MDEyOklzc3VlQ29tbWVudDcwOTU3MjQyNQ==,9599,simonw,2020-10-15T20:28:18Z,2020-10-15T20:28:18Z,OWNER,"Also need to rethink this template logic that decides if to show a column as sorted or not:
https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/templates/_table.html#L10-L14","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709571143,https://api.github.com/repos/simonw/datasette/issues/894,709571143,MDEyOklzc3VlQ29tbWVudDcwOTU3MTE0Mw==,9599,simonw,2020-10-15T20:25:35Z,2020-10-15T20:25:35Z,OWNER,"`cast(nullif(colname, '') as real)` can fix this - it will treat `''` the same as `null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709569951,https://api.github.com/repos/simonw/datasette/issues/894,709569951,MDEyOklzc3VlQ29tbWVudDcwOTU2OTk1MQ==,9599,simonw,2020-10-15T20:23:02Z,2020-10-15T20:23:02Z,OWNER,"Something to watch out for: `""""` empty strings cast to `0.0`:

`select cast(""100"" as real), ""100"", cast(null as real), cast("""" as real)`

cast(""100"" as real) | ""100"" | cast(null as real) | cast("""" as real)
-- | -- | -- | --
100.0 | 100 |   | 0.0

https://latest.datasette.io/fixtures?sql=select+cast%28%22100%22+as+real%29%2C+%22100%22%2C+cast%28null+as+real%29%2C+cast%28%22%22+as+real%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709562940,https://api.github.com/repos/simonw/datasette/issues/894,709562940,MDEyOklzc3VlQ29tbWVudDcwOTU2Mjk0MA==,9599,simonw,2020-10-15T20:08:16Z,2020-10-15T20:08:16Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/views/table.py#L485-L510,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709546976,https://api.github.com/repos/simonw/datasette/issues/894,709546976,MDEyOklzc3VlQ29tbWVudDcwOTU0Njk3Ng==,9599,simonw,2020-10-15T19:35:55Z,2020-10-15T19:36:38Z,OWNER,"Much easier solution: if the suffix is `~numeric` then treat it as the column name sorted numerically. If the suffix is missing OR the suffix is `~default`, sort without casting. Only add the `~default` suffix if the column name itself contains at least one `~` symbol.

Using `~` because it doesn't need to be URL-encoded.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/690#issuecomment-709499944,https://api.github.com/repos/simonw/datasette/issues/690,709499944,MDEyOklzc3VlQ29tbWVudDcwOTQ5OTk0NA==,9599,simonw,2020-10-15T18:09:06Z,2020-10-15T18:09:06Z,OWNER,"I'm going to dedicate this issue to thinking about action menus. #981 added column action menus, others to build could be:

- Table action menus (for things like configure FTS, edit schema)
- Database action menus (import CSV file into this table)
- Instance action menus (manage users, import CSV into this instance)
- Column action menus (already there, plugins could include extract this column)
- Row action menus?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/issues/690#issuecomment-709498425,https://api.github.com/repos/simonw/datasette/issues/690,709498425,MDEyOklzc3VlQ29tbWVudDcwOTQ5ODQyNQ==,9599,simonw,2020-10-15T18:06:08Z,2020-10-15T18:06:08Z,OWNER,And for instance-level actions (linking to `datasette-import-csv` for example) an actions menu anchored against a burger-bar menu icon in the navigation bar.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/issues/690#issuecomment-709497595,https://api.github.com/repos/simonw/datasette/issues/690,709497595,MDEyOklzc3VlQ29tbWVudDcwOTQ5NzU5NQ==,9599,simonw,2020-10-15T18:04:35Z,2020-10-15T18:12:15Z,OWNER,"For the table actions: attaching it to a cog icon next to the table name could make sense.

<img width=""503"" alt=""data__faa-wildlife-strikes__26_rows_where_where_AIRPORT____OIL_RIG__sorted_by_rowid_and_Fix_table_name_in_spatialite_example_command_by_jsfenfen_·_Pull_Request__1022_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/96168896-45192c80-0ed6-11eb-9d90-6172a8346468.png"">

This is the column action icon at twice the size, color `#666`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/pull/1022#issuecomment-708718475,https://api.github.com/repos/simonw/datasette/issues/1022,708718475,MDEyOklzc3VlQ29tbWVudDcwODcxODQ3NQ==,9599,simonw,2020-10-14T23:46:38Z,2020-10-14T23:46:38Z,OWNER,"Good catch, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721830990,Fix table name in spatialite example command,
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-708695761,https://api.github.com/repos/simonw/sqlite-utils/issues/182,708695761,MDEyOklzc3VlQ29tbWVudDcwODY5NTc2MQ==,9599,simonw,2020-10-14T22:33:55Z,2020-10-14T22:35:17Z,OWNER,"For the moment I'm going to add a `--encoding` option and some code that catches `UnicodeDecodeError` and shows an error message that suggests using `--encoding`.

That error message could detect if the `file` command is available and, if it is, suggest running `file filename.txt` to detect the character encoding.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""",
https://github.com/simonw/sqlite-utils/issues/163#issuecomment-708695369,https://api.github.com/repos/simonw/sqlite-utils/issues/163,708695369,MDEyOklzc3VlQ29tbWVudDcwODY5NTM2OQ==,9599,simonw,2020-10-14T22:32:39Z,2020-10-14T22:32:39Z,OWNER,`sqlite-utils convert my.db mytable mycolumn` could become the new `sqlite-transform`. The sub-command could take `--code` but could also take `--parsedate` and `--parsedatetime` options.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706001517,Idea: conversions= could take Python functions,
https://github.com/simonw/sqlite-utils/issues/163#issuecomment-708693290,https://api.github.com/repos/simonw/sqlite-utils/issues/163,708693290,MDEyOklzc3VlQ29tbWVudDcwODY5MzI5MA==,9599,simonw,2020-10-14T22:26:50Z,2020-10-14T22:27:48Z,OWNER,"This could even be exposed as a feature of the CLI tool - similar to how `sqlite-transform` works: https://github.com/simonw/sqlite-transform

    $ sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'

Maybe `sqlite-utils` should absorb all of the functionality from `sqlite-transform` - having two separate tools doesn't necessarily make sense.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706001517,Idea: conversions= could take Python functions,
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-708693133,https://api.github.com/repos/simonw/sqlite-utils/issues/182,708693133,MDEyOklzc3VlQ29tbWVudDcwODY5MzEzMw==,9599,simonw,2020-10-14T22:26:26Z,2020-10-14T22:26:26Z,OWNER,"I could use https://github.com/chardet/chardet to help here, though I'd rather not add it as a dependency (`sqlite-utils` has very few dependencies at the moment). I could add it as an optional dependency though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""",
https://github.com/simonw/datasette/issues/1020#issuecomment-708670392,https://api.github.com/repos/simonw/datasette/issues/1020,708670392,MDEyOklzc3VlQ29tbWVudDcwODY3MDM5Mg==,9599,simonw,2020-10-14T21:29:22Z,2020-10-14T21:29:22Z,OWNER,"I should also verify (and probably unit-test) that things like the `?_trace=1` mechanism work across the internal request boundary.

`/-/permissions` appears to work across this boundary, but again a test would be useful confirmation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-708669778,https://api.github.com/repos/simonw/datasette/issues/1020,708669778,MDEyOklzc3VlQ29tbWVudDcwODY2OTc3OA==,9599,simonw,2020-10-14T21:27:58Z,2020-10-14T21:27:58Z,OWNER,"Maybe these internal requests should have some kind of flag that lets the underlying code tell that it's being called internally.

One option: add a `x-internal: 1` request header - and ensure that any requests from outside Datasette have that header stripped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-708669178,https://api.github.com/repos/simonw/datasette/issues/1020,708669178,MDEyOklzc3VlQ29tbWVudDcwODY2OTE3OA==,9599,simonw,2020-10-14T21:26:37Z,2020-10-14T21:26:37Z,OWNER,"One option: add an optional request=... parameter which can be passed the current request, and will use that to populate the mock request with the exception of the bits that are passed explicitly (like the path):

```python
response = await datasette.client.get(""/db/table.json"", request=request)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1019#issuecomment-708139822,https://api.github.com/repos/simonw/datasette/issues/1019,708139822,MDEyOklzc3VlQ29tbWVudDcwODEzOTgyMg==,9599,simonw,2020-10-14T03:53:21Z,2020-10-14T03:53:21Z,OWNER,"Demos:
- https://latest.datasette.io/fixtures/neighborhood_search has the link
- https://latest.datasette.io/fixtures/neighborhood_search?text=ber has the link, and the link passes through the parameter that has already been entered
- https://latest.datasette.io/fixtures/%F0%9D%90%9C%F0%9D%90%A2%F0%9D%90%AD%F0%9D%90%A2%F0%9D%90%9E%F0%9D%90%AC has the link
- https://latest.datasette.io/fixtures/pragma_cache_size does not have the link (`PRAGMA` is not allowed in regular editable queries)
- https://latest.datasette.io/fixtures/magic_parameters does not have the link (magic parameters)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708130290,https://api.github.com/repos/simonw/datasette/issues/1019,708130290,MDEyOklzc3VlQ29tbWVudDcwODEzMDI5MA==,9599,simonw,2020-10-14T03:17:57Z,2020-10-14T03:17:57Z,OWNER,"One last edge-case (I think) - magic parameters aren't supported for non-canned-queries, so if a query includes those it shouldn't show an Edit SQL link: https://latest.datasette.io/fixtures/magic_parameters

I don't have a clean way of detecting if a query contains defined magic parameters, but I can instead do a dumb substring match for `:_` and skip showing the Edit SQL link if that is present.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708128286,https://api.github.com/repos/simonw/datasette/issues/1019,708128286,MDEyOklzc3VlQ29tbWVudDcwODEyODI4Ng==,9599,simonw,2020-10-14T03:11:33Z,2020-10-14T03:14:07Z,OWNER,"Another edge-case: https://latest.datasette.io/fixtures/pragma_cache_size (`PRAGMA cache_size;`) isn't an allowed query usually, so linking to ""Edit SQL"" for it would link to an error page: https://latest.datasette.io/fixtures?sql=PRAGMA+cache_size%3B

Can use `datasette.utils.validate_sql_select(sql)` to check for that - it raises `datasette.utils.InvalidSql` if there's a problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708127294,https://api.github.com/repos/simonw/datasette/issues/1019,708127294,MDEyOklzc3VlQ29tbWVudDcwODEyNzI5NA==,9599,simonw,2020-10-14T03:08:13Z,2020-10-14T03:08:13Z,OWNER,"Maybe like this:

<img width=""1385"" alt=""fixtures__select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________order_by_neighborhood_"" src=""https://user-images.githubusercontent.com/9599/95938930-ce721700-0d8f-11eb-87c8-f67ea4e57d55.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708114636,https://api.github.com/repos/simonw/datasette/issues/1019,708114636,MDEyOklzc3VlQ29tbWVudDcwODExNDYzNg==,9599,simonw,2020-10-14T02:24:56Z,2020-10-14T02:24:56Z,OWNER,"So it would appear on this page: https://latest.datasette.io/fixtures/neighborhood_search#fragment-goes-here

Probably as a blue ""Edit SQL"" link next to those buttons.

![575FB160-4F75-43B2-B2AB-D75E60F10477](https://user-images.githubusercontent.com/9599/95936143-c2835680-0d89-11eb-9cf3-06f86d923103.jpeg)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708113989,https://api.github.com/repos/simonw/datasette/issues/1019,708113989,MDEyOklzc3VlQ29tbWVudDcwODExMzk4OQ==,9599,simonw,2020-10-14T02:22:40Z,2020-10-14T02:22:40Z,OWNER,"I've found myself wanting this too.

It should respect the allow-sql permission so it doesn't display if the user isn't able to run custom SQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1016#issuecomment-706828375,https://api.github.com/repos/simonw/datasette/issues/1016,706828375,MDEyOklzc3VlQ29tbWVudDcwNjgyODM3NQ==,9599,simonw,2020-10-12T03:01:58Z,2020-10-12T03:01:58Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet=created&created=2019-01-17+08%3A00%3A00&_facet=state&state=MI

![x-button](https://user-images.githubusercontent.com/9599/95700744-9b4b4e80-0bfc-11eb-85a3-6da82d9e0cbf.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718953669,"Add a ""delete"" icon next to filters (in addition to ""remove filter"")",
https://github.com/simonw/datasette/issues/1016#issuecomment-706821561,https://api.github.com/repos/simonw/datasette/issues/1016,706821561,MDEyOklzc3VlQ29tbWVudDcwNjgyMTU2MQ==,9599,simonw,2020-10-12T02:33:52Z,2020-10-12T02:33:52Z,OWNER,"Should this do the equivalent of clicking Apply for you? I don't think so - that wouldn't fit with how the rest of that block of controls works.

But if it makes the row vanish it may not be clear to the user that they still need to click Apply.

Easiest thing would be for it to select the ""remove filter"" column and blank out the boxes. I will try that first.

The code can go in `table.js`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718953669,"Add a ""delete"" icon next to filters (in addition to ""remove filter"")",
https://github.com/simonw/datasette/issues/690#issuecomment-706845950,https://api.github.com/repos/simonw/datasette/issues/690,706845950,MDEyOklzc3VlQ29tbWVudDcwNjg0NTk1MA==,9599,simonw,2020-10-12T04:07:28Z,2020-10-12T04:07:28Z,OWNER,Next step is to design where these will go and what they will look like - in particular the navigation links that lead to plugins such as `datasette-upload-csvs` and `datasette-configure-fts` and `datasette-edit-schema`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/issues/782#issuecomment-706738020,https://api.github.com/repos/simonw/datasette/issues/782,706738020,MDEyOklzc3VlQ29tbWVudDcwNjczODAyMA==,9599,simonw,2020-10-11T17:23:18Z,2020-10-11T17:23:48Z,OWNER,"I'm going to prototype what it would look like if the default shape was a list of objects and `?_extra=` turns that into an object with a `rows` key, in a plugin. As a separate extension (maybe `.json-preview`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-706736541,https://api.github.com/repos/simonw/datasette/issues/782,706736541,MDEyOklzc3VlQ29tbWVudDcwNjczNjU0MQ==,9599,simonw,2020-10-11T17:12:27Z,2020-10-11T17:12:27Z,OWNER,"The core issue that I keep reconsidering is whether the default `.json` representation should be an object or a list.

Arguments in favour of a list:

- It's what I always want. Almost all of the code that I've written against the API myself uses `?_shape=array`.
- It's really easy to use. You can pipe it to e.g. `sqlite-utils insert`, you can load it into JavaScript without thinking about it.

Arguments against:

- Nowhere to put pagination or total counts. I added pagination to the `link:` HTTP header in #1014 (inspired by the WordPress and GitHub APIs) but I haven't solved for total count, and there's other stuff that's useful like `""truncated"": true` to indicate that more than 1000 results were returned and they were truncated.
- An array is inherently non-extensible: if the root item is an object it's easy to add new features to it in a backwards-compatible way in the future. An array is a fixed format.

But maybe that last point is a positive? It ensures the default `.json` format remains completely predictable forever.

If `.json` DID default to an array of objects, the `?_shape=` argument could still be used to get back alternative formats.

Maybe `.json?_extra=total` changes the shape of that default to be this instead:

```json
{
  ""rows"": [{""id"": 1}, {""id"": 2}],
  ""total"": 104
}
```

The thing I care about most though is `next_url`. That could be provided like so:

`.json?_extra=total&_extra=next` - alternative syntax `.json?_extra=total,next`:

```json
{
  ""rows"": [{""id"": 1}, {""id"": 2}],
  ""total"": 104,
  ""next"": ""2"",
  ""next_url"": ""/db/table.json?_extra=total&_extra=next&_next=2""
}
```
This is feeling a bit verbose for a common combination though.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-706735341,https://api.github.com/repos/simonw/datasette/issues/782,706735341,MDEyOklzc3VlQ29tbWVudDcwNjczNTM0MQ==,9599,simonw,2020-10-11T17:03:29Z,2020-10-11T17:15:34Z,OWNER,"Maybe `.jsonfull` becomes a new renderer that returns ALL of the defined `?_extra=` blocks.

Or... `?_extra=all` turns on ALL of the available information blocks (some of which can come from plugins).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-706735200,https://api.github.com/repos/simonw/datasette/issues/782,706735200,MDEyOklzc3VlQ29tbWVudDcwNjczNTIwMA==,9599,simonw,2020-10-11T17:02:11Z,2020-10-11T17:14:51Z,OWNER,"Since the total count can be expensive to calculate, I'm inclined to make that an opt-in extra - maybe `?_extra=count`.

Based on that, the default JSON shape could look something like this:

```json
{
  ""rows"": [{""id"": 1}, {""id"": 2}],
  ""next"": ""2"",
  ""next_url"": ""/db/table?_next=2""
}
```
And with `?_extra=count`:
```json
{
  ""rows"": [{""id"": 1}, {""id"": 2}],
  ""next"": ""2"",
  ""next_url"": ""/db/table?_next=2"",
  ""count"": 31
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-706735280,https://api.github.com/repos/simonw/datasette/issues/782,706735280,MDEyOklzc3VlQ29tbWVudDcwNjczNTI4MA==,9599,simonw,2020-10-11T17:03:01Z,2020-10-11T17:03:01Z,OWNER,"Should that default also include `""columns""` as a list of strings? That would be duplicate data of the keys in the `""rows""` list of objects, and I've never found myself wanting it in my own code - so I'm going to say no.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1016#issuecomment-706788010,https://api.github.com/repos/simonw/datasette/issues/1016,706788010,MDEyOklzc3VlQ29tbWVudDcwNjc4ODAxMA==,9599,simonw,2020-10-11T23:50:39Z,2020-10-11T23:50:39Z,OWNER,"For consistency can reuse the icon used on selected facets:

<img width=""721"" alt=""evernote__notes__8_rows_where_where_application-data____ACD4B543-4663-40B2-BAF5-666FA54D4204__and_source-url_is_not_blank_sorted_by_id"" src=""https://user-images.githubusercontent.com/9599/95693564-e0ae5280-0be1-11eb-881b-fa0cec8397f5.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718953669,"Add a ""delete"" icon next to filters (in addition to ""remove filter"")",
https://github.com/simonw/datasette/issues/1015#issuecomment-706756879,https://api.github.com/repos/simonw/datasette/issues/1015,706756879,MDEyOklzc3VlQ29tbWVudDcwNjc1Njg3OQ==,9599,simonw,2020-10-11T19:35:03Z,2020-10-11T19:35:03Z,OWNER,"Since plugins are installed via pip this would require Datasette to be restarted. This StackOverflow thread looks relevant to that: https://stackoverflow.com/questions/11329917/restart-python-script-from-within-itself

This recipe looks promising:
```python
import os
import sys
import psutil
import logging

def restart_program():
    """"""Restarts the current program, with file objects and descriptors
       cleanup
    """"""

    try:
        p = psutil.Process(os.getpid())
        for handler in p.get_open_files() + p.connections():
            os.close(handler.fd)
    except Exception, e:
        logging.error(e)

    python = sys.executable
    os.execl(python, python, *sys.argv)
```
https://docs.python.org/3/library/os.html#os.execl says about `os.execl`:

> These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718910318,Research: could Datasette install its own plugins?,
https://github.com/simonw/datasette/issues/782#issuecomment-706745236,https://api.github.com/repos/simonw/datasette/issues/782,706745236,MDEyOklzc3VlQ29tbWVudDcwNjc0NTIzNg==,9599,simonw,2020-10-11T18:16:05Z,2020-10-11T18:16:05Z,OWNER,Here's the `datasette-json-preview` plugin I'll be using to experiment with different formats: https://github.com/simonw/datasette-json-preview,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-706740250,https://api.github.com/repos/simonw/datasette/issues/782,706740250,MDEyOklzc3VlQ29tbWVudDcwNjc0MDI1MA==,9599,simonw,2020-10-11T17:40:48Z,2020-10-11T17:43:07Z,OWNER,"Building this plugin reminded me of an oddity of the `register_output_renderer()` plugin hook: one of the arguments that can be passed to it is `data`, which is the default internal data structure created by Datasette - but I deliberately avoided documenting that on https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette because it's not a stable interface.

That's not ideal. I'd like custom renderers to be able to access this data to get at things like suggested facets, on an opt-in basis.

So maybe that kind of stuff is re-implemented as ""extras"" which are awaitable callables - then renderer plugins can call the extras that they need to as part of their execution.

To illustrate the problem (in this case the need to access `next_url`) here's my first prototype of the plugin:
```python
from datasette import hookimpl
from datasette.utils.asgi import Response


@hookimpl
def register_output_renderer(datasette):
    return {
        ""extension"": ""json-preview"",
        ""render"": json_preview,
    }


def json_preview(data, columns, rows):
    next_url = data.get(""next_url"")
    headers = {}
    if next_url:
        headers[""link""] = '<{}>; rel=""next""'.format(next_url)
    return Response.json([dict(zip(columns, row)) for row in rows], headers=headers)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1014#issuecomment-706631006,https://api.github.com/repos/simonw/datasette/issues/1014,706631006,MDEyOklzc3VlQ29tbWVudDcwNjYzMTAwNg==,9599,simonw,2020-10-11T00:36:43Z,2020-10-11T00:36:43Z,OWNER,"Demo using [paginate-json](https://github.com/simonw/paginate-json):
```
% paginate-json 'https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array' | jq '. | length' 
https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cd%2Cv
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ch%2Cr
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cl%2Cn
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cp%2Cj
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ct%2Cf
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cx%2Cb
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ca%2Cx
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ce%2Ct
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ci%2Cp
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Cm%2Cl
1001
```
New documentation: https://docs.datasette.io/en/latest/json_api.html#pagination","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706627035,https://api.github.com/repos/simonw/datasette/issues/1014,706627035,MDEyOklzc3VlQ29tbWVudDcwNjYyNzAzNQ==,9599,simonw,2020-10-10T23:50:40Z,2020-10-10T23:50:40Z,OWNER,"First example page to add this to: `?_shape=array`, e.g. https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706626934,https://api.github.com/repos/simonw/datasette/issues/1014,706626934,MDEyOklzc3VlQ29tbWVudDcwNjYyNjkzNA==,9599,simonw,2020-10-10T23:49:25Z,2020-10-10T23:49:25Z,OWNER,I'm just going to implement the 'next' one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706626913,https://api.github.com/repos/simonw/datasette/issues/1014,706626913,MDEyOklzc3VlQ29tbWVudDcwNjYyNjkxMw==,9599,simonw,2020-10-10T23:49:08Z,2020-10-10T23:49:08Z,OWNER,This should be documented on https://docs.datasette.io/en/stable/json_api.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706626971,https://api.github.com/repos/simonw/datasette/issues/1014,706626971,MDEyOklzc3VlQ29tbWVudDcwNjYyNjk3MQ==,9599,simonw,2020-10-10T23:49:57Z,2020-10-10T23:49:57Z,OWNER,I won't offer a total-number header. That's expensive to calculate - no need to calculate it unless it's explicitly asked for.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706626796,https://api.github.com/repos/simonw/datasette/issues/1014,706626796,MDEyOklzc3VlQ29tbWVudDcwNjYyNjc5Ng==,9599,simonw,2020-10-10T23:47:23Z,2020-10-10T23:48:26Z,OWNER,"The GitHub API has these:

`link: <https://api.github.com/repositories/107914493/commits?page=2>; rel=""next"", <https://api.github.com/repositories/107914493/commits?page=46>; rel=""last""`

WordPress API: https://css-tricks.com/wp-json/wp/v2/posts

`link: <https://css-tricks.com/wp-json/wp/v2/posts?page=2>; rel=""next""`

It also has these:

```
x-wp-total: 5162
x-wp-totalpages: 517
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1011#issuecomment-706483680,https://api.github.com/repos/simonw/datasette/issues/1011,706483680,MDEyOklzc3VlQ29tbWVudDcwNjQ4MzY4MA==,9599,simonw,2020-10-10T04:09:32Z,2020-10-10T04:09:32Z,OWNER,"OK, Datasette 0.50.2 is live on PyPI now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718521469,column name links broken in 0.50.1,