(impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c73af5d...8f00c31](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",855446829,
https://github.com/simonw/datasette/issues/1295#issuecomment-817301355,https://api.github.com/repos/simonw/datasette/issues/1295,817301355,MDEyOklzc3VlQ29tbWVudDgxNzMwMTM1NQ==,9599,2021-04-11T12:40:25Z,2021-04-11T12:41:06Z,OWNER,"I could have a page about error codes in the docs, then have `https://datasette.io/E123` style URLs for each error core which are shown when that error occurs and redirect to the corresponding documentation section.
Can enforce these with a documentation unit test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",855296937,
https://github.com/simonw/datasette/issues/1286#issuecomment-815978405,https://api.github.com/repos/simonw/datasette/issues/1286,815978405,MDEyOklzc3VlQ29tbWVudDgxNTk3ODQwNQ==,192568,2021-04-08T16:47:29Z,2021-04-10T03:59:00Z,CONTRIBUTOR,"This worked for me:
`| {{ cell.value | replace('"", ""','; ') | replace('[\""','') | replace('\""]','')}} | `
I'm sure there is a prettier (and more flexible) way, but for now, this is ever-so-much more pleasant to look at.
------ AFTER:
![]()
------ BEFORE:
![]()
(Note: I didn't figure out how to have one item have no semicolon, while multi-items close with a semicolon, but this is good enough for now. I also didn't figure out how to set up a new jinja filter. I don't want to add to /datasette/utils/__init__.py as I assume that would get overwritten when upgrading datasette. Having a starter guide on creating jinja filters in datasette would be helpful. (The jinja documentation isn't datasette-specific enough for me to quite nail it.)
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849220154,
https://github.com/simonw/datasette/issues/1293#issuecomment-813438771,https://api.github.com/repos/simonw/datasette/issues/1293,813438771,MDEyOklzc3VlQ29tbWVudDgxMzQzODc3MQ==,9599,2021-04-05T14:58:48Z,2021-04-05T14:58:48Z,OWNER,I may need to do something special for rowid columns - there is a `RowId` opcode that might come into play here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813480043,https://api.github.com/repos/simonw/datasette/issues/1293,813480043,MDEyOklzc3VlQ29tbWVudDgxMzQ4MDA0Mw==,9599,2021-04-05T16:16:17Z,2021-04-05T16:16:17Z,OWNER,"https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because `paginated_view` is defined like this:
```sql
CREATE VIEW paginated_view AS
SELECT
content,
'- ' || content || ' -' AS content_extra
FROM no_primary_key;
```
So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813445512,https://api.github.com/repos/simonw/datasette/issues/1293,813445512,MDEyOklzc3VlQ29tbWVudDgxMzQ0NTUxMg==,9599,2021-04-05T15:11:40Z,2021-04-05T15:11:40Z,OWNER,"Here's some older example code that works with opcodes from Python, in this case to output indexes used by a query: https://github.com/plasticityai/supersqlite/blob/master/supersqlite/idxchk.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/dogsheep/dogsheep-photos/issues/35#issuecomment-813249000,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/35,813249000,MDEyOklzc3VlQ29tbWVudDgxMzI0OTAwMA==,1151557,2021-04-05T07:37:57Z,2021-04-05T07:37:57Z,NONE,"There are trained ML models used in Photoprism:
- https://dl.photoprism.org/tensorflow/nasnet.zip
- https://dl.photoprism.org/tensorflow/nsfw.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842695374,
https://github.com/simonw/datasette/issues/1293#issuecomment-813134637,https://api.github.com/repos/simonw/datasette/issues/1293,813134637,MDEyOklzc3VlQ29tbWVudDgxMzEzNDYzNw==,9599,2021-04-05T01:21:59Z,2021-04-05T01:21:59Z,OWNER,"http://www.sqlite.org/draft/lang_explain.html says:
> Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.
I'm going to keep exploring this though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813134386,https://api.github.com/repos/simonw/datasette/issues/1293,813134386,MDEyOklzc3VlQ29tbWVudDgxMzEzNDM4Ng==,9599,2021-04-05T01:20:28Z,2021-08-13T00:42:30Z,OWNER,"... that output might also provide a better way to extract variables than the current mechanism using a regular expression, by looking for the `Variable` opcodes.
[UPDATE: it did indeed do that, see #1421]","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813134227,https://api.github.com/repos/simonw/datasette/issues/1293,813134227,MDEyOklzc3VlQ29tbWVudDgxMzEzNDIyNw==,9599,2021-04-05T01:19:31Z,2021-04-05T01:19:31Z,OWNER,"| addr | opcode | p1 | p2 | p3 | p4 | p5 | comment |
|--------|---------------|------|------|------|-----------------------|------|-----------|
| 0 | Init | 0 | 47 | 0 | | 00 | |
| 1 | OpenRead | 0 | 51 | 0 | 15 | 00 | |
| 2 | Integer | 15 | 2 | 0 | | 00 | |
| 3 | Once | 0 | 15 | 0 | | 00 | |
| 4 | OpenEphemeral | 2 | 1 | 0 | k(1,) | 00 | |
| 5 | VOpen | 1 | 0 | 0 | vtab:3E692C362158 | 00 | |
| 6 | String8 | 0 | 5 | 0 | CPAD_2020a_SuperUnits | 00 | |
| 7 | SCopy | 7 | 6 | 0 | | 00 | |
| 8 | Integer | 2 | 3 | 0 | | 00 | |
| 9 | Integer | 2 | 4 | 0 | | 00 | |
| 10 | VFilter | 1 | 15 | 3 | | 00 | |
| 11 | Rowid | 1 | 8 | 0 | | 00 | |
| 12 | MakeRecord | 8 | 1 | 9 | C | 00 | |
| 13 | IdxInsert | 2 | 9 | 8 | 1 | 00 | |
| 14 | VNext | 1 | 11 | 0 | | 00 | |
| 15 | Return | 2 | 0 | 0 | | 00 | |
| 16 | Rewind | 2 | 46 | 0 | | 00 | |
| 17 | Column | 2 | 0 | 1 | | 00 | |
| 18 | IsNull | 1 | 45 | 0 | | 00 | |
| 19 | SeekRowid | 0 | 45 | 1 | | 00 | |
| 20 | Column | 0 | 2 | 11 | | 00 | |
| 21 | Function0 | 1 | 10 | 9 | like(2) | 02 | |
| 22 | IfNot | 9 | 45 | 1 | | 00 | |
| 23 | Column | 0 | 14 | 13 | | 00 | |
| 24 | Function0 | 1 | 12 | 9 | intersects(2) | 02 | |
| 25 | Ne | 14 | 45 | 9 | | 51 | |
| 26 | Column | 0 | 14 | 9 | | 00 | |
| 27 | Function0 | 0 | 9 | 15 | asgeojson(1) | 01 | |
| 28 | Rowid | 0 | 16 | 0 | | 00 | |
| 29 | Column | 0 | 1 | 17 | | 00 | |
| 30 | Column | 0 | 2 | 18 | | 00 | |
| 31 | Column | 0 | 3 | 19 | | 00 | |
| 32 | Column | 0 | 4 | 20 | | 00 | |
| 33 | Column | 0 | 5 | 21 | | 00 | |
| 34 | Column | 0 | 6 | 22 | | 00 | |
| 35 | Column | 0 | 7 | 23 | | 00 | |
| 36 | Column | 0 | 8 | 24 | | 00 | |
| 37 | Column | 0 | 9 | 25 | | 00 | |
| 38 | Column | 0 | 10 | 26 | | 00 | |
| 39 | Column | 0 | 11 | 27 | | 00 | |
| 40 | RealAffinity | 27 | 0 | 0 | | 00 | |
| 41 | Column | 0 | 12 | 28 | | 00 | |
| 42 | Column | 0 | 13 | 29 | | 00 | |
| 43 | Column | 0 | 14 | 30 | | 00 | |
| 44 | ResultRow | 15 | 16 | 0 | | 00 | |
| 45 | Next | 2 | 17 | 0 | | 00 | |
| 46 | Halt | 0 | 0 | 0 | | 00 | |
| 47 | Transaction | 0 | 0 | 265 | 0 | 01 | |
| 48 | Variable | 1 | 31 | 0 | :freedraw | 00 | |
| 49 | Function0 | 1 | 31 | 7 | geomfromgeojson(1) | 01 | |
| 50 | String8 | 0 | 10 | 0 | %mini% | 00 | |
| 51 | Variable | 1 | 32 | 0 | :freedraw | 00 | |
| 52 | Function0 | 1 | 32 | 12 | geomfromgeojson(1) | 01 | |
| 53 | Integer | 1 | 14 | 0 | | 00 | |
| 54 | Goto | 0 | 1 | 0 | | 00 | |
Essential documentation for understanding that output: https://www.sqlite.org/opcode.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813134072,https://api.github.com/repos/simonw/datasette/issues/1293,813134072,MDEyOklzc3VlQ29tbWVudDgxMzEzNDA3Mg==,9599,2021-04-05T01:18:37Z,2021-04-05T01:18:37Z,OWNER,"Had a fantastic suggestion on the SQLite forum: it might be possible to get what I want by interpreting the opcodes output by `explain select ...`.
Copying the reply I posted to this thread:
That's really useful, thanks! It looks like it _might_ be possible for me to reconstruct where each column came from using the `explain select` output.
Here's a complex example:
It looks like the opcodes I need to inspect are `OpenRead`, `Column` and `ResultRow`.
`OpenRead` tells me which tables are being opened - the `p2` value (in this case 51) corresponds to the `rootpage` column in `sqlite_master` here: - it gets assigned to the register in `p1`.
The `Column` opcodes tell me which columns are being read - `p1` is that table reference, and `p2` is the `cid` of the column within that table.
The `ResultRow` opcode then tells me which columns are used in the results. `15 16` means start at the 15th and then read the next 16 columns.
I think this might work!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813116177,https://api.github.com/repos/simonw/datasette/issues/1293,813116177,MDEyOklzc3VlQ29tbWVudDgxMzExNjE3Nw==,9599,2021-04-04T23:31:00Z,2021-04-04T23:31:00Z,OWNER,"Sadly it doesn't do what I need. This query should only return one column, but instead I get back every column that was consulted by the query:
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813115607,https://api.github.com/repos/simonw/datasette/issues/1293,813115607,MDEyOklzc3VlQ29tbWVudDgxMzExNTYwNw==,9599,2021-04-04T23:25:15Z,2021-04-04T23:25:15Z,OWNER,"Oh wow, I just spotted https://github.com/macbre/sql-metadata
> Uses tokenized query returned by python-sqlparse and generates query metadata. Extracts column names and tables used by the query. Provides a helper for normalization of SQL queries and tables aliases resolving.
It's for MySQL, PostgreSQL and Hive right now but maybe getting it working with SQLite wouldn't be too hard?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813115414,https://api.github.com/repos/simonw/datasette/issues/1293,813115414,MDEyOklzc3VlQ29tbWVudDgxMzExNTQxNA==,9599,2021-04-04T23:23:34Z,2021-04-04T23:23:34Z,OWNER,"The other approach I considered for this was to have my own SQL query parser running in Python, which could pick apart a complex query and figure out which column was sourced from which table. I dropped this idea because it felt that the moment `select *` came into play a pure parsing approach wouldn't work - I'd need knowledge of the schema in order to resolve the `*`.
A Python parser approach might be good enough to handle a subset of queries - those that don't use `select *` for example - and maybe that would be worth shipping? The feature doesn't have to be perfect for it to be useful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813114933,https://api.github.com/repos/simonw/datasette/issues/1293,813114933,MDEyOklzc3VlQ29tbWVudDgxMzExNDkzMw==,9599,2021-04-04T23:19:22Z,2021-04-04T23:19:22Z,OWNER,I asked about this on the SQLite forum: https://sqlite.org/forum/forumpost/0180277fb7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813113653,https://api.github.com/repos/simonw/datasette/issues/1293,813113653,MDEyOklzc3VlQ29tbWVudDgxMzExMzY1Mw==,9599,2021-04-04T23:10:49Z,2021-04-04T23:10:49Z,OWNER,"One option I've not fully explored yet: could I write my own custom SQLite C extension which exposes this functionality as a callable function?
Then I could load that extension and run a SQL query something like this:
```
select database, table, column from analyze_query(:sql_query)
```
Where `analyze_query(...)` would be a fancy virtual table function of some sort that uses the underlying `sqlite3_column_database_name()` C functions to analyze the SQL query and return details of what it would return.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813113403,https://api.github.com/repos/simonw/datasette/issues/1293,813113403,MDEyOklzc3VlQ29tbWVudDgxMzExMzQwMw==,9599,2021-04-04T23:08:48Z,2021-04-04T23:08:48Z,OWNER,"Worth noting that adding `limit 0` to the query still causes it to conduct the permission checks, hopefully while avoiding doing any of the actual work of executing the query:
```pycon
In [20]: db.execute('select * from compound_primary_key join facetable on facetable.rowid = compound_primary_key.rowid limit 0').fetchall()
...:
args (21, None, None, None, None) kwargs {}
args (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'content', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'facetable', 'created', 'main', None) kwargs {}
args (20, 'facetable', 'planet_int', 'main', None) kwargs {}
args (20, 'facetable', 'on_earth', 'main', None) kwargs {}
args (20, 'facetable', 'state', 'main', None) kwargs {}
args (20, 'facetable', 'city_id', 'main', None) kwargs {}
args (20, 'facetable', 'neighborhood', 'main', None) kwargs {}
args (20, 'facetable', 'tags', 'main', None) kwargs {}
args (20, 'facetable', 'complex_array', 'main', None) kwargs {}
args (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'ROWID', 'main', None) kwargs {}
Out[20]: []
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813113218,https://api.github.com/repos/simonw/datasette/issues/1293,813113218,MDEyOklzc3VlQ29tbWVudDgxMzExMzIxOA==,9599,2021-04-04T23:07:25Z,2021-04-04T23:07:25Z,OWNER,"Here are all of the available constants:
```pycon
In [3]: for k in dir(sqlite3):
...: if k.startswith(""SQLITE_""):
...: print(k, getattr(sqlite3, k))
...:
SQLITE_ALTER_TABLE 26
SQLITE_ANALYZE 28
SQLITE_ATTACH 24
SQLITE_CREATE_INDEX 1
SQLITE_CREATE_TABLE 2
SQLITE_CREATE_TEMP_INDEX 3
SQLITE_CREATE_TEMP_TABLE 4
SQLITE_CREATE_TEMP_TRIGGER 5
SQLITE_CREATE_TEMP_VIEW 6
SQLITE_CREATE_TRIGGER 7
SQLITE_CREATE_VIEW 8
SQLITE_CREATE_VTABLE 29
SQLITE_DELETE 9
SQLITE_DENY 1
SQLITE_DETACH 25
SQLITE_DONE 101
SQLITE_DROP_INDEX 10
SQLITE_DROP_TABLE 11
SQLITE_DROP_TEMP_INDEX 12
SQLITE_DROP_TEMP_TABLE 13
SQLITE_DROP_TEMP_TRIGGER 14
SQLITE_DROP_TEMP_VIEW 15
SQLITE_DROP_TRIGGER 16
SQLITE_DROP_VIEW 17
SQLITE_DROP_VTABLE 30
SQLITE_FUNCTION 31
SQLITE_IGNORE 2
SQLITE_INSERT 18
SQLITE_OK 0
SQLITE_PRAGMA 19
SQLITE_READ 20
SQLITE_RECURSIVE 33
SQLITE_REINDEX 27
SQLITE_SAVEPOINT 32
SQLITE_SELECT 21
SQLITE_TRANSACTION 22
SQLITE_UPDATE 23
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813113175,https://api.github.com/repos/simonw/datasette/issues/1293,813113175,MDEyOklzc3VlQ29tbWVudDgxMzExMzE3NQ==,9599,2021-04-04T23:07:01Z,2021-04-04T23:07:01Z,OWNER,"A more promising route I found involved the `db.set_authorizer` method. This can be used to log the permission checks that SQLite uses, including checks for permission to access specific columns of specific tables. For a while I thought this could work!
```pycon
>>> def print_args(*args, **kwargs):
... print(""args"", args, ""kwargs"", kwargs)
... return sqlite3.SQLITE_OK
>>> db = sqlite3.connect(""fixtures.db"")
>>> db.execute('select * from compound_primary_key join facetable on rowid').fetchall()
args (21, None, None, None, None) kwargs {}
args (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'content', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'facetable', 'created', 'main', None) kwargs {}
args (20, 'facetable', 'planet_int', 'main', None) kwargs {}
args (20, 'facetable', 'on_earth', 'main', None) kwargs {}
args (20, 'facetable', 'state', 'main', None) kwargs {}
args (20, 'facetable', 'city_id', 'main', None) kwargs {}
args (20, 'facetable', 'neighborhood', 'main', None) kwargs {}
args (20, 'facetable', 'tags', 'main', None) kwargs {}
args (20, 'facetable', 'complex_array', 'main', None) kwargs {}
args (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}
```
Those `20` values (where 20 is `SQLITE_READ`) looked like they were checking permissions for the columns in the order they would be returned!
Then I found a snag:
```pycon
In [18]: db.execute('select 1 + 1 + (select max(rowid) from facetable)')
args (21, None, None, None, None) kwargs {}
args (31, None, 'max', None, None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (21, None, None, None, None) kwargs {}
args (20, 'facetable', '', None, None) kwargs {}
```
Once a subselect is involved the order of the `20` checks no longer matches the order in which the columns are returned from the query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813112546,https://api.github.com/repos/simonw/datasette/issues/1293,813112546,MDEyOklzc3VlQ29tbWVudDgxMzExMjU0Ng==,9599,2021-04-04T23:02:45Z,2021-04-04T23:02:45Z,OWNER,"I've done various pieces of research into this over the past few years. Capturing what I've discovered in this ticket.
The SQLite C API has functions that can help with this: https://www.sqlite.org/c3ref/column_database_name.html details those. But they're not exposed in the Python SQLite library.
Maybe it would be possible to use them via `ctypes`? My hunch is that I would have to re-implement the full `sqlite3` module with `ctypes`, which sounds daunting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1292#issuecomment-813109789,https://api.github.com/repos/simonw/datasette/issues/1292,813109789,MDEyOklzc3VlQ29tbWVudDgxMzEwOTc4OQ==,9599,2021-04-04T22:37:47Z,2021-04-04T22:37:47Z,OWNER,Could maybe replace this code: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/utils/__init__.py#L1021-L1026,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849975810,
https://github.com/simonw/datasette/issues/620#issuecomment-813167335,https://api.github.com/repos/simonw/datasette/issues/620,813167335,MDEyOklzc3VlQ29tbWVudDgxMzE2NzMzNQ==,9599,2021-04-05T03:57:22Z,2021-04-05T03:57:22Z,OWNER,This may be obsoleted by #1293 - it looks like I may be able to auto-detect these foreign keys for arbitrary queries after all.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520667773,
https://github.com/simonw/datasette/issues/1293#issuecomment-813164282,https://api.github.com/repos/simonw/datasette/issues/1293,813164282,MDEyOklzc3VlQ29tbWVudDgxMzE2NDI4Mg==,9599,2021-04-05T03:42:26Z,2021-04-05T03:42:36Z,OWNER,"Extracting variables with this trick appears to work OK, but you have to pass the correct variables to the `explain select...` query. Using `defaultdict` seems to work there:
```pycon
>>> rows = conn.execute('explain select * from repos where id = :id', defaultdict(int))
>>> [dict(r) for r in rows if r['opcode'] == 'Variable']
[{'addr': 2,
'opcode': 'Variable',
'p1': 1,
'p2': 1,
'p3': 0,
'p4': ':id',
'p5': 0,
'comment': None}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813162622,https://api.github.com/repos/simonw/datasette/issues/1293,813162622,MDEyOklzc3VlQ29tbWVudDgxMzE2MjYyMg==,9599,2021-04-05T03:34:24Z,2021-04-05T03:40:35Z,OWNER,"This almost works, but throws errors with some queries (anything with a `rowid` column for example) - it needs a bunch of test coverage.
```python
def columns_for_query(conn, sql):
rows = conn.execute('explain ' + sql).fetchall()
table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'}
names_by_rootpage = dict(
conn.execute(
'select rootpage, name from sqlite_master where rootpage in ({})'.format(
', '.join(map(str, table_rootpage_by_register.values()))
)
)
)
columns_by_column_register = {}
for row in rows:
if row['opcode'] == 'Column':
addr, opcode, table_id, cid, column_register, p4, p5, comment = row
table = names_by_rootpage[table_rootpage_by_register[table_id]]
columns_by_column_register[column_register] = (table, cid)
result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0]
registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""] - 1))
all_column_names = {}
for table in names_by_rootpage.values():
table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall()
for row in table_xinfo:
all_column_names[(table, row[""cid""])] = row[""name""]
final_output = []
for r in registers:
try:
table, cid = columns_by_column_register[r]
final_output.append((table, all_column_names[table, cid]))
except KeyError:
final_output.append((None, None))
return final_output
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1273#issuecomment-813061516,https://api.github.com/repos/simonw/datasette/issues/1273,813061516,MDEyOklzc3VlQ29tbWVudDgxMzA2MTUxNg==,9599,2021-04-04T16:32:40Z,2021-04-04T16:32:40Z,OWNER,Useful tutorial series from 2012: https://northredoubt.com/n/2012/01/20/spatialite-speed-test/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838382890,
https://github.com/simonw/datasette/issues/1287#issuecomment-812935384,https://api.github.com/repos/simonw/datasette/issues/1287,812935384,MDEyOklzc3VlQ29tbWVudDgxMjkzNTM4NA==,9599,2021-04-03T22:38:33Z,2021-04-03T22:38:33Z,OWNER,"https://twitter.com/llanga/status/1378431719934681094 looks like I should wait for 3.9.4, out in a few days.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849396758,
https://github.com/simonw/datasette/issues/916#issuecomment-812941818,https://api.github.com/repos/simonw/datasette/issues/916,812941818,MDEyOklzc3VlQ29tbWVudDgxMjk0MTgxOA==,9599,2021-04-03T23:43:11Z,2021-04-03T23:43:11Z,OWNER,"Relevant code is some of the most complex in all of Datasette.
https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L530-L594
And
https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L743-L771
I'll need to think hard about how to refactor this out into something more understandable before implementing previous links.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,
https://github.com/simonw/datasette/issues/916#issuecomment-812941340,https://api.github.com/repos/simonw/datasette/issues/916,812941340,MDEyOklzc3VlQ29tbWVudDgxMjk0MTM0MA==,9599,2021-04-03T23:38:37Z,2021-04-03T23:38:37Z,OWNER,"Same query again with `a, d, v` returns 0 results, which is also as we would want: it signifies that we are back to the very first page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=d&p2=v","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,
https://github.com/simonw/datasette/issues/916#issuecomment-812941112,https://api.github.com/repos/simonw/datasette/issues/916,812941112,MDEyOklzc3VlQ29tbWVudDgxMjk0MTExMg==,9599,2021-04-03T23:35:55Z,2021-04-03T23:35:55Z,OWNER,"I tried flipping the direction of the sort and the comparison operators and got this: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=h&p2=r
```sql
select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 < :p0)
or
(pk1 = :p0 and pk2 < :p1)
or
(pk1 = :p0 and pk2 = :p1 and pk3 < :p2)) order by pk1 desc, pk2 desc, pk3 desc limit 1 offset 99
```
Which returned `a-d-v` as desired. I messed around with it to find the `limit 1 offset 99` values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,
https://github.com/simonw/datasette/issues/916#issuecomment-812940907,https://api.github.com/repos/simonw/datasette/issues/916,812940907,MDEyOklzc3VlQ29tbWVudDgxMjk0MDkwNw==,9599,2021-04-03T23:33:41Z,2021-04-03T23:33:41Z,OWNER,"Let's figure out the SQL for this. The most complex case is probably this one: https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Ch%2Cr
Here's the SQL for that page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3E+%3Ap2%29%29+order+by+pk1%2C+pk2%2C+pk3+limit+101&p0=a&p1=h&p2=r
```sql
select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 > :p0)
or
(pk1 = :p0 and pk2 > :p1)
or
(pk1 = :p0 and pk2 = :p1 and pk3 > :p2)) order by pk1, pk2, pk3 limit 101
```
Where `p0` is `a`, `p1` is `h` and `p2` is `r`.
Given the above, how would I figure out the correct previous link? It should be https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv - `a`, `d`, `v`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,
https://github.com/simonw/datasette/issues/916#issuecomment-812940457,https://api.github.com/repos/simonw/datasette/issues/916,812940457,MDEyOklzc3VlQ29tbWVudDgxMjk0MDQ1Nw==,9599,2021-04-03T23:28:40Z,2021-04-03T23:28:40Z,OWNER,"I think my ideal implementation for this would be to reverse the order, grab the previous page-size-plus-one items, then return a `?_next=x` token that would provide the previous page sorted back in the expected default order.
The alternative would be to have a `?_previous=x` token which can be used to paginate backwards in reverse order, but I think this would be confusing as it would result in ""hit next page, then hit previous page"" returning you to a new state which features rows in the reverse order.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,
https://github.com/simonw/datasette/pull/1291#issuecomment-812815358,https://api.github.com/repos/simonw/datasette/issues/1291,812815358,MDEyOklzc3VlQ29tbWVudDgxMjgxNTM1OA==,22429695,2021-04-03T05:32:50Z,2021-06-05T19:48:30Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1291](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8789157) into [main](https://codecov.io/gh/simonw/datasette/commit/0a7621f96f8ad14da17e7172e8a7bce24ef78966?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0a7621f) will **not change** coverage.
> The diff coverage is `n/a`.
> :exclamation: Current head 8789157 differs from pull request most recent head 9bf089f. Consider uploading reports for the commit 9bf089f to get more accurate results
[](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #1291 +/- ##
=======================================
Coverage 91.51% 91.51%
=======================================
Files 34 34
Lines 4255 4255
=======================================
Hits 3894 3894
Misses 361 361
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0a7621f...9bf089f](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849582643,
https://github.com/simonw/datasette/issues/502#issuecomment-812813732,https://api.github.com/repos/simonw/datasette/issues/502,812813732,MDEyOklzc3VlQ29tbWVudDgxMjgxMzczMg==,5413548,2021-04-03T05:16:54Z,2021-04-03T05:16:54Z,CONTRIBUTOR,"For what it's worth, if anyone finds this in the future, I was having the same issue.
After digging through the code, it turned out that the database download is only available if it the db served in immutable mode, so `datasette serve -i xyz.db` rather than the doc's quickstart recommendation of `datasette serve xyz.db`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",453131917,
https://github.com/simonw/datasette/issues/916#issuecomment-812804998,https://api.github.com/repos/simonw/datasette/issues/916,812804998,MDEyOklzc3VlQ29tbWVudDgxMjgwNDk5OA==,9599,2021-04-03T03:47:45Z,2021-04-03T03:47:45Z,OWNER,I found one example of an implementation of reversed keyset pagination here: https://github.com/tvainika/objection-keyset-pagination/blob/cb21a493c96daa6e63c302efae6718d09aa11661/index.js#L74-L79,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,
https://github.com/simonw/datasette/pull/1290#issuecomment-812804178,https://api.github.com/repos/simonw/datasette/issues/1290,812804178,MDEyOklzc3VlQ29tbWVudDgxMjgwNDE3OA==,22429695,2021-04-03T03:39:16Z,2021-04-03T03:41:29Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1290?src=pr&el=h1) Report
> Merging [#1290](https://codecov.io/gh/simonw/datasette/pull/1290?src=pr&el=desc) (2fb1e42) into [main](https://codecov.io/gh/simonw/datasette/commit/87b583a128986982552421d2510e467e74ac5046?el=desc) (87b583a) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/datasette/pull/1290?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #1290 +/- ##
=======================================
Coverage 91.51% 91.51%
=======================================
Files 34 34
Lines 4255 4255
=======================================
Hits 3894 3894
Misses 361 361
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1290?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1290?src=pr&el=footer). Last update [87b583a...2fb1e42](https://codecov.io/gh/simonw/datasette/pull/1290?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849568079,
https://github.com/simonw/datasette/issues/1289#issuecomment-812803256,https://api.github.com/repos/simonw/datasette/issues/1289,812803256,MDEyOklzc3VlQ29tbWVudDgxMjgwMzI1Ng==,9599,2021-04-03T03:29:25Z,2021-04-03T03:29:25Z,OWNER,"https://github.com/simonw/datasette/actions/runs/713207828 ran with `pytest-xdist` in 4m22s:
![]()
Here's the test suite running on regular `pytest` in 5m13s:
![]()
Not a huge speed-up because there are only 2 available cores in the GitHub Actions environment, but still worthwhile - especially since this lets people run in parallel on their own laptops.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849543502,
https://github.com/simonw/datasette/issues/1245#issuecomment-812711365,https://api.github.com/repos/simonw/datasette/issues/1245,812711365,MDEyOklzc3VlQ29tbWVudDgxMjcxMTM2NQ==,1111743,2021-04-02T20:53:35Z,2021-04-02T20:53:35Z,NONE,"Yes, I agree.
Alternatively, maybe the header could be at the top and bottom, above the next page button.
Maybe even have the header 50 records down?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817544251,
https://github.com/simonw/datasette/issues/1255#issuecomment-812710120,https://api.github.com/repos/simonw/datasette/issues/1255,812710120,MDEyOklzc3VlQ29tbWVudDgxMjcxMDEyMA==,1111743,2021-04-02T20:50:08Z,2021-04-02T20:50:08Z,NONE,"Hello again,
I was able to get my facets running with this `settings.json`, which was lifted from one of Simon's datasette's and slightly modified.
```
{
""default_page_size"": 100,
""max_returned_rows"": 1000,
""num_sql_threads"": 3,
""sql_time_limit_ms"": 9000,
""default_facet_size"": 10,
""facet_time_limit_ms"": 9000,
""facet_suggest_time_limit_ms"": 500,
""hash_urls"": false,
""allow_facet"": true,
""suggest_facets"": false,
""default_cache_ttl"": 5,
""default_cache_ttl_hashed"": 31536000,
""cache_size_kb"": 0,
""allow_csv_stream"": true,
""max_csv_mb"": 100,
""truncate_cells_html"": 2048,
""template_debug"": false,
""base_url"": ""/""
}
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826700095,
https://github.com/simonw/datasette/issues/1289#issuecomment-812768915,https://api.github.com/repos/simonw/datasette/issues/1289,812768915,MDEyOklzc3VlQ29tbWVudDgxMjc2ODkxNQ==,9599,2021-04-03T00:59:15Z,2021-04-03T00:59:26Z,OWNER,"Looks like `-n auto` only detected two cores on GitHub Actions: https://github.com/simonw/datasette/runs/2257597137?check_suite_focus=true
```
============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
SQLite: 3.31.1
rootdir: /home/runner/work/datasette/datasette, configfile: pytest.ini
plugins: xdist-2.2.1, timeout-1.4.2, forked-1.3.0, asyncio-0.14.0
gw0 I / gw1 I
gw0 [878] / gw1 [878]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849543502,
https://github.com/simonw/datasette/issues/1289#issuecomment-812767460,https://api.github.com/repos/simonw/datasette/issues/1289,812767460,MDEyOklzc3VlQ29tbWVudDgxMjc2NzQ2MA==,9599,2021-04-03T00:48:26Z,2021-04-03T00:48:26Z,OWNER,"On my Mac `pytest-xdist` ran the test suite (minus two tests) in 59s, as opposed to 2m23s without xdist.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849543502,
https://github.com/simonw/datasette/issues/916#issuecomment-812742462,https://api.github.com/repos/simonw/datasette/issues/916,812742462,MDEyOklzc3VlQ29tbWVudDgxMjc0MjQ2Mg==,1111743,2021-04-02T22:37:27Z,2021-04-02T22:37:27Z,NONE,"Yes, this would be nice!
I using Datasette v0.56 and don't see a previous page button.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,
https://github.com/simonw/datasette/issues/1255#issuecomment-812680519,https://api.github.com/repos/simonw/datasette/issues/1255,812680519,MDEyOklzc3VlQ29tbWVudDgxMjY4MDUxOQ==,1111743,2021-04-02T19:37:57Z,2021-04-02T19:37:57Z,NONE,"Hello,
I'm also experiencing a timeout in my environment. I don't know if it's because I need more indexes or a more powerful system.
My data has 1,271,111 and when I try to create a facet, there's a time out. I've tried this on two different rows that should significantly filter down data: `CITY` and `PARTY_REG`.
Simon's johns_hopkins_csse_daily_reports has more rows and it setup with two facets on load. He does have four indexes created, though. Do I need more indexes?
I have one simple one so far:
```
CREATE INDEX [idx_party_reg]
ON [county_active] ([PARTY_REG]);
```
I'm running Datasette 0.56 installed via pip with Python 3.7.3.
`4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux`
```
$ cat /etc/os-release
PRETTY_NAME=""Debian GNU/Linux 10 (buster)""
NAME=""Debian GNU/Linux""
VERSION_ID=""10""
VERSION=""10 (buster)""
VERSION_CODENAME=buster
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826700095,
https://github.com/simonw/datasette/issues/1286#issuecomment-812679221,https://api.github.com/repos/simonw/datasette/issues/1286,812679221,MDEyOklzc3VlQ29tbWVudDgxMjY3OTIyMQ==,192568,2021-04-02T19:34:01Z,2021-04-02T19:34:01Z,CONTRIBUTOR,"This shows the city in a different color (and not the comma), but I get the idea, and I like it. (Ooh, could be nice to have the gear have an option in array fields to show as bullets or commas or semicolons...)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849220154,
https://github.com/simonw/datasette/issues/1287#issuecomment-812665092,https://api.github.com/repos/simonw/datasette/issues/1287,812665092,MDEyOklzc3VlQ29tbWVudDgxMjY2NTA5Mg==,9599,2021-04-02T18:54:29Z,2021-04-02T18:54:29Z,OWNER,`python:3.9.3-slim-buster` isn't on Docker Hub yet either.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849396758,
https://github.com/simonw/datasette/issues/1286#issuecomment-812664443,https://api.github.com/repos/simonw/datasette/issues/1286,812664443,MDEyOklzc3VlQ29tbWVudDgxMjY2NDQ0Mw==,9599,2021-04-02T18:52:45Z,2021-04-02T18:52:51Z,OWNER,"Idea: default to displaying single-dimension JSON arrays of strings as a comma-separated list but show the comma in a different colour - something like this:
![]()
I used this HTML for the prototype (re-using `.type-int` just to get the colour):
```html
tag1, tag2 |
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849220154,
https://github.com/simonw/datasette/issues/1286#issuecomment-812663107,https://api.github.com/repos/simonw/datasette/issues/1286,812663107,MDEyOklzc3VlQ29tbWVudDgxMjY2MzEwNw==,9599,2021-04-02T18:49:22Z,2021-04-02T18:49:22Z,OWNER,"This makes senses - showing an array as `[""blah"", ""blah2"", ""blah3""]` isn't particularly human-friendly!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849220154,
https://github.com/simonw/datasette/issues/1287#issuecomment-812662583,https://api.github.com/repos/simonw/datasette/issues/1287,812662583,MDEyOklzc3VlQ29tbWVudDgxMjY2MjU4Mw==,9599,2021-04-02T18:47:59Z,2021-04-02T18:47:59Z,OWNER,Once again having tests for the Dockerfile as seen in #1272 would be useful here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849396758,
https://github.com/simonw/datasette/issues/1287#issuecomment-812662026,https://api.github.com/repos/simonw/datasette/issues/1287,812662026,MDEyOklzc3VlQ29tbWVudDgxMjY2MjAyNg==,9599,2021-04-02T18:46:20Z,2021-04-02T18:46:20Z,OWNER,https://devcenter.heroku.com/articles/python-support#supported-runtimes looks like Heroku still only have 3.9.2 for the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849396758,
https://github.com/simonw/datasette/issues/1287#issuecomment-812661269,https://api.github.com/repos/simonw/datasette/issues/1287,812661269,MDEyOklzc3VlQ29tbWVudDgxMjY2MTI2OQ==,9599,2021-04-02T18:45:08Z,2021-04-02T18:45:19Z,OWNER,"A few places:
https://github.com/simonw/datasette/blob/7b1a9a1999eb9326ce8ec830d75ac200e5279c46/Dockerfile#L1
https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L350
https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/tests/test_package.py#L14
https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/publish/heroku.py#L177-L178","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849396758,
https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-811362316,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31,811362316,MDEyOklzc3VlQ29tbWVudDgxMTM2MjMxNg==,871250,2021-03-31T19:14:39Z,2021-03-31T19:14:39Z,NONE,👋 could I help somehow for this to be merged? As Big Sur is going to be more used as the time goes I think it would be nice to merge and publish a new version. Nice work!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771511344,
https://github.com/simonw/datasette/issues/1276#issuecomment-811209922,https://api.github.com/repos/simonw/datasette/issues/1276,811209922,MDEyOklzc3VlQ29tbWVudDgxMTIwOTkyMg==,1314318,2021-03-31T16:27:26Z,2021-03-31T16:27:26Z,NONE,Fantastic. Thank you! ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,
https://github.com/simonw/datasette/issues/526#issuecomment-810943882,https://api.github.com/repos/simonw/datasette/issues/526,810943882,MDEyOklzc3VlQ29tbWVudDgxMDk0Mzg4Mg==,701,2021-03-31T10:03:55Z,2021-03-31T10:03:55Z,NONE,"+1 on using nested queries to achieve this! Would be great as streaming CSV is an amazing feature.
Some UX/DX details:
I was expecting it to work to simply add `&_stream=on` to custom SQL queries because the docs say
> Any Datasette table, view or **custom SQL query** can be exported as CSV.
After a bit of testing back and forth I realized streaming only works for full tables.
Would love this feature because I'm using `pandas.read_csv` to paint graphs from custom queries and the graphs are cut off because of the 1000 row limit. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,
https://github.com/simonw/datasette/issues/1284#issuecomment-810779928,https://api.github.com/repos/simonw/datasette/issues/1284,810779928,MDEyOklzc3VlQ29tbWVudDgxMDc3OTkyOA==,192568,2021-03-31T05:40:12Z,2021-03-31T05:40:12Z,CONTRIBUTOR,"Maybe the addition of two template files: 'one_database_index.html' and 'one_table_index.html' would be a better idea than the documentation diff idea. (They could include commented instructions to rename the preferred template 'index.html', along with any other necessary guidance.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",845794436,
https://github.com/simonw/datasette/issues/1284#issuecomment-810740486,https://api.github.com/repos/simonw/datasette/issues/1284,810740486,MDEyOklzc3VlQ29tbWVudDgxMDc0MDQ4Ng==,9599,2021-03-31T03:57:55Z,2021-03-31T03:57:55Z,OWNER,"You're right, doing this is really hard at the moment - I'm not sure I know how I would tackle this either, and it's something I've wanted in the past!
I'll have a think about this one.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",845794436,
https://github.com/simonw/datasette/pull/1282#issuecomment-809670294,https://api.github.com/repos/simonw/datasette/issues/1282,809670294,MDEyOklzc3VlQ29tbWVudDgwOTY3MDI5NA==,9599,2021-03-29T19:57:29Z,2021-03-29T19:57:29Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",843739658,
https://github.com/simonw/datasette/pull/1282#issuecomment-809667320,https://api.github.com/repos/simonw/datasette/issues/1282,809667320,MDEyOklzc3VlQ29tbWVudDgwOTY2NzMyMA==,22429695,2021-03-29T19:52:35Z,2021-03-29T19:52:35Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1282?src=pr&el=h1) Report
> Merging [#1282](https://codecov.io/gh/simonw/datasette/pull/1282?src=pr&el=desc) (08f7427) into [main](https://codecov.io/gh/simonw/datasette/commit/0486303b60ce2784fd2e2ecdbecf304b7d6e6659?el=desc) (0486303) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/simonw/datasette/pull/1282?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #1282 +/- ##
=======================================
Coverage 91.51% 91.51%
=======================================
Files 34 34
Lines 4255 4255
=======================================
Hits 3894 3894
Misses 361 361
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1282?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1282?src=pr&el=footer). Last update [0486303...08f7427](https://codecov.io/gh/simonw/datasette/pull/1282?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",843739658,
https://github.com/simonw/datasette/issues/696#issuecomment-809548363,https://api.github.com/repos/simonw/datasette/issues/696,809548363,MDEyOklzc3VlQ29tbWVudDgwOTU0ODM2Mw==,9599,2021-03-29T17:04:19Z,2021-03-29T17:04:19Z,OWNER,I tried this just now against Datasette 0.56 with the new Dockerfile from #1249 (that uses SQLite and SpatiaLite installed with `apt-get install`) and the tests all passed.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",576722115,
https://github.com/simonw/datasette/pull/1031#issuecomment-809010713,https://api.github.com/repos/simonw/datasette/issues/1031,809010713,MDEyOklzc3VlQ29tbWVudDgwOTAxMDcxMw==,9599,2021-03-29T01:46:45Z,2021-03-29T01:46:45Z,OWNER,Sorry I didn't get to this PR sooner. I've joint-credited you in the release notes for this fix: https://docs.datasette.io/en/stable/changelog.html#v0-56,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724369025,
https://github.com/simonw/datasette/issues/1281#issuecomment-809009580,https://api.github.com/repos/simonw/datasette/issues/1281,809009580,MDEyOklzc3VlQ29tbWVudDgwOTAwOTU4MA==,9599,2021-03-29T01:41:48Z,2021-03-29T01:41:48Z,OWNER,"https://github.com/simonw/datasette/runs/2214871602?check_suite_focus=true worked:
![]()
Here's the 0.56 image on Docker Hub: https://hub.docker.com/layers/datasetteproject/datasette/0.56/images/sha256-701fc0f299a0ea79434a4852c46dab351254b9ac25dbe3c5f36fd5360caf52f9?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,
https://github.com/simonw/datasette/issues/1281#issuecomment-809008760,https://api.github.com/repos/simonw/datasette/issues/1281,809008760,MDEyOklzc3VlQ29tbWVudDgwOTAwODc2MA==,9599,2021-03-29T01:38:21Z,2021-03-29T01:38:21Z,OWNER,"Got this error:
```
""docker tag"" requires exactly 2 arguments.
See 'docker tag --help'.
Usage: docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]
Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,
https://github.com/simonw/datasette/issues/1281#issuecomment-809007255,https://api.github.com/repos/simonw/datasette/issues/1281,809007255,MDEyOklzc3VlQ29tbWVudDgwOTAwNzI1NQ==,9599,2021-03-29T01:32:18Z,2021-03-29T01:32:18Z,OWNER,I'm going to build a new GitHub Actions workflow for this that lets me manually specify a tag to build and push as a Docker image.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,
https://github.com/simonw/datasette/issues/1281#issuecomment-809001653,https://api.github.com/repos/simonw/datasette/issues/1281,809001653,MDEyOklzc3VlQ29tbWVudDgwOTAwMTY1Mw==,9599,2021-03-29T01:08:31Z,2021-03-29T01:08:31Z,OWNER,"I'm going to attempt to fix this manually for the 0.56 release, by building and tagging it by hand and then pushing the 0.56 tag to Docker Hub.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,
https://github.com/simonw/datasette/issues/1281#issuecomment-809001273,https://api.github.com/repos/simonw/datasette/issues/1281,809001273,MDEyOklzc3VlQ29tbWVudDgwOTAwMTI3Mw==,9599,2021-03-29T01:06:45Z,2021-03-29T01:06:45Z,OWNER,"https://docs.docker.com/engine/reference/commandline/push/#push-all-tags-of-an-image
> Use the `-a` (or `--all-tags`) option to push all tags of a local image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,
https://github.com/simonw/datasette/issues/1281#issuecomment-809000903,https://api.github.com/repos/simonw/datasette/issues/1281,809000903,MDEyOklzc3VlQ29tbWVudDgwOTAwMDkwMw==,9599,2021-03-29T01:05:10Z,2021-03-29T01:05:10Z,OWNER,"https://github.com/simonw/datasette/runs/1763835467?check_suite_focus=true for Datasette 0.54 worked, and the output included this:
```
Successfully tagged ***/datasette:0.54
The push refers to repository [docker.io/***/datasette]
aedd33c6b161: Preparing
...
aedd33c6b161: Pushed
0.54: digest: sha256:65c7e579d1c29755dac5c1ca86b1e97fa88c48bd3d724ac3e02988d0da296140 size: 2005
aedd33c6b161: Preparing
...
5dacd731af1b: Layer already exists
latest: digest: sha256:65c7e579d1c29755dac5c1ca86b1e97fa88c48bd3d724ac3e02988d0da296140 size: 2005
```
Here's that same section of output from the 0.56 release:
```
Successfully tagged ***/datasette:0.56
Using default tag: latest
The push refers to repository [docker.io/***/datasette]
4d4a9976adcc: Preparing
...
9b2132a0d5cf: Pushed
latest: digest: sha256:2250d0fbe57b1d615a8d6df0c9d43deb9533532e00bac68854773d8ff8dcf00a size: 1793
```
The difference here is the ""Using default tag: latest"" bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,
https://github.com/simonw/datasette/issues/1281#issuecomment-808999525,https://api.github.com/repos/simonw/datasette/issues/1281,808999525,MDEyOklzc3VlQ29tbWVudDgwODk5OTUyNQ==,9599,2021-03-29T01:00:38Z,2021-03-29T01:00:38Z,OWNER,"Here's the diff between `Dockerfile` in 0.54.1 and 0.56: https://github.com/simonw/datasette/compare/0.54.1...0.56#diff-551d1fcf87f78cc3bc18a7b332a4dc5d8773a512062df881c5aba28a6f5c48d7
![]()
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,
https://github.com/simonw/datasette/issues/1249#issuecomment-808998719,https://api.github.com/repos/simonw/datasette/issues/1249,808998719,MDEyOklzc3VlQ29tbWVudDgwODk5ODcxOQ==,9599,2021-03-29T00:57:13Z,2021-03-29T00:57:13Z,OWNER,"I just shipped Datasette 0.56 - here's the CI run: https://github.com/simonw/datasette/runs/2214701802?check_suite_focus=true
It pushed a new `latest` tag to https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated
docker pull datasetteproject/datasette:latest
And then:
docker run datasetteproject/datasette:latest datasette \
--load-extension=spatialite \
--get /-/versions.json | jq .sqlite
Outputs:
```json
{
""version"": ""3.27.2"",
""fts_versions"": [
""FTS5"",
""FTS4"",
""FTS3""
],
""extensions"": {
""json1"": null,
""spatialite"": ""5.0.1""
},
""compile_options"": [
""COMPILER=gcc-8.3.0"",
""ENABLE_COLUMN_METADATA"",
""ENABLE_DBSTAT_VTAB"",
""ENABLE_FTS3"",
""ENABLE_FTS3_PARENTHESIS"",
""ENABLE_FTS3_TOKENIZER"",
""ENABLE_FTS4"",
""ENABLE_FTS5"",
""ENABLE_JSON1"",
""ENABLE_LOAD_EXTENSION"",
""ENABLE_PREUPDATE_HOOK"",
""ENABLE_RTREE"",
""ENABLE_SESSION"",
""ENABLE_STMTVTAB"",
""ENABLE_UNLOCK_NOTIFY"",
""ENABLE_UPDATE_DELETE_LIMIT"",
""HAVE_ISNAN"",
""LIKE_DOESNT_MATCH_BLOBS"",
""MAX_SCHEMA_RETRY=25"",
""MAX_VARIABLE_NUMBER=250000"",
""OMIT_LOOKASIDE"",
""SECURE_DELETE"",
""SOUNDEX"",
""TEMP_STORE=1"",
""THREADSAFE=1"",
""USE_URI""
]
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,