{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898788262", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898788262, "node_id": "IC_kwDOBm6k_c41kmum", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T01:22:26Z", "updated_at": "2021-08-14T01:51:08Z", "author_association": "OWNER", "body": "Tried a more complicated query:\r\n```sql\r\nexplain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101\r\n```\r\nHere's the explain:\r\n```\r\nsqlite> explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101\r\n   ...> ;\r\naddr  opcode         p1    p2    p3    p4             p5  comment      \r\n----  -------------  ----  ----  ----  -------------  --  -------------\r\n0     Init           0     41    0                    00  Start at 41  \r\n1     OpenEphemeral  2     6     0     k(1,-B)        00  nColumn=6    \r\n2     Integer        101   1     0                    00  r[1]=101; LIMIT counter\r\n3     OpenRead       0     32    0     4              00  root=32 iDb=0; searchable\r\n4     Integer        16    3     0                    00  r[3]=16; return address\r\n5     Once           0     16    0                    00               \r\n6     OpenEphemeral  3     1     0     k(1,)          00  nColumn=1; Result of SELECT 1\r\n7     VOpen          1     0     0     vtab:7FCBCA72BE80  00               \r\n8     Function0      1     7     6     unknown(-1)    01  r[6]=func(r[7])\r\n9     Integer        5     4     0                    00  r[4]=5       \r\n10    Integer        1     5     0                    00  r[5]=1       \r\n11    VFilter        1     16    4                    00  iplan=r[4] zplan=''\r\n12      Rowid          1     8     0                    00  r[8]=rowid   \r\n13      MakeRecord     8     1     9     C              00  r[9]=mkrec(r[8])\r\n14      IdxInsert      3     9     8     1              00  key=r[9]     \r\n15    VNext          1     12    0                    00               \r\n16    Return         3     0     0                    00               \r\n17    Rewind         3     33    0                    00               \r\n18      Column         3     0     2                    00  r[2]=        \r\n19      IsNull         2     32    0                    00  if r[2]==NULL goto 32\r\n20      SeekRowid      0     32    2                    00  intkey=r[2]  \r\n21      Column         0     1     10                   00  r[10]=searchable.text1\r\n22      Sequence       2     11    0                    00  r[11]=cursor[2].ctr++\r\n23      IfNotZero      1     27    0                    00  if r[1]!=0 then r[1]--, goto 27\r\n24      Last           2     0     0                    00               \r\n25      IdxLE          2     32    10    1              00  key=r[10]    \r\n26      Delete         2     0     0                    00               \r\n27      Rowid          0     12    0                    00  r[12]=rowid  \r\n28      Column         0     2     13                   00  r[13]=searchable.text2\r\n29      Column         0     3     14                   00  r[14]=searchable.name with . and spaces\r\n30      MakeRecord     10    5     16                   00  r[16]=mkrec(r[10..14])\r\n31      IdxInsert      2     16    10    5              00  key=r[16]    \r\n32    Next           3     18    0                    00               \r\n33    Sort           2     40    0                    00               \r\n34      Column         2     4     15                   00  r[15]=[name with . and spaces]\r\n35      Column         2     3     14                   00  r[14]=text2  \r\n36      Column         2     0     13                   00  r[13]=text1  \r\n37      Column         2     2     12                   00  r[12]=pk     \r\n38      ResultRow      12    4     0                    00  output=r[12..15]\r\n39    Next           2     34    0                    00               \r\n40    Halt           0     0     0                    00               \r\n41    Transaction    0     0     35    0              01  usesStmtJournal=0\r\n42    Variable       1     7     0     :search        00  r[7]=parameter(1,:search)\r\n43    Goto           0     1     0                    00               \r\n```\r\nHere the `ResultRow` is for registers `12..15` - but those all refer to `Column` records in `2` - where `2` is the first `OpenEphemeral` declared right at the start. I'm having enormous trouble figuring out how that ephemeral table gets populated by the other operations in a way that would let me derive which columns end up in the `ResultRow`.\r\n\r\nFrustratingly SQLite seems to be able to figure that out just fine, see the column of comments on the right hand side - but I only get those in the `sqlite3` CLI shell, they're not available to me with SQLite when called as a library from Python.\r\n\r\nMaybe the key to that is this section:\r\n```\r\n27      Rowid          0     12    0                    00  r[12]=rowid  \r\n28      Column         0     2     13                   00  r[13]=searchable.text2\r\n29      Column         0     3     14                   00  r[14]=searchable.name with . and spaces\r\n30      MakeRecord     10    5     16                   00  r[16]=mkrec(r[10..14])\r\n31      IdxInsert      2     16    10    5              00  key=r[16]    \r\n```\r\nMakeRecord:\r\n\r\n> Convert P2 registers beginning with P1 into the record format use as a data record in a database table or as a key in an index. The Column opcode can decode the record later.\r\n> \r\n> P4 may be a string that is P2 characters long. The N-th character of the string indicates the column affinity that should be used for the N-th field of the index key.\r\n> \r\n> The mapping from character to affinity is given by the SQLITE_AFF_ macros defined in sqliteInt.h.\r\n> \r\n> If P4 is NULL then all index fields have the affinity BLOB.\r\n> \r\n> The meaning of P5 depends on whether or not the SQLITE_ENABLE_NULL_TRIM compile-time option is enabled:\r\n> \r\n> * If SQLITE_ENABLE_NULL_TRIM is enabled, then the P5 is the index of the right-most table that can be null-trimmed.\r\n> \r\n> * If SQLITE_ENABLE_NULL_TRIM is omitted, then P5 has the value OPFLAG_NOCHNG_MAGIC if the MakeRecord opcode is allowed to accept no-change records with serial_type 10. This value is only used inside an assert() and does not affect the end result.\r\n\r\nIdxInsert:\r\n> Register P2 holds an SQL index key made using the MakeRecord instructions. This opcode writes that key into the index P1. Data for the entry is nil.\r\n> \r\n> If P4 is not zero, then it is the number of values in the unpacked key of reg(P2). In that case, P3 is the index of the first register for the unpacked key. The availability of the unpacked key can sometimes be an optimization.\r\n> \r\n> If P5 has the OPFLAG_APPEND bit set, that is a hint to the b-tree layer that this insert is likely to be an append.\r\n> \r\n> If P5 has the OPFLAG_NCHANGE bit set, then the change counter is incremented by this instruction. If the OPFLAG_NCHANGE bit is clear, then the change counter is unchanged.\r\n> \r\n> If the OPFLAG_USESEEKRESULT flag of P5 is set, the implementation might run faster by avoiding an unnecessary seek on cursor P1. However, the OPFLAG_USESEEKRESULT flag must only be set if there have been no prior seeks on the cursor or if the most recent seek used a key equivalent to P2.\r\n>\r\n> This instruction only works for indices. The equivalent instruction for tables is Insert.\r\n\r\nIdxLE:\r\n> The P4 register values beginning with P3 form an unpacked index key that omits the PRIMARY KEY or ROWID. Compare this key value against the index that P1 is currently pointing to, ignoring the PRIMARY KEY or ROWID on the P1 index.\r\n>\r\n> If the P1 index entry is less than or equal to the key value then jump to P2. Otherwise fall through to the next instruction.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898913554", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898913554, "node_id": "IC_kwDOBm6k_c41lFUS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T16:13:40Z", "updated_at": "2021-08-14T16:13:40Z", "author_association": "OWNER", "body": "I think I need to care about the following:\r\n\r\n- `ResultRow` and `Column` for the final result\r\n- `OpenRead` for opening tables\r\n- `OpenEphemeral` then `MakeRecord` and `IdxInsert` for writing records into ephemeral tables\r\n\r\n`Column` may reference either a table (from `OpenRead`) or an ephemeral table (from `OpenEphemeral`).\r\n\r\nThat *might* be enough.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898913629", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898913629, "node_id": "IC_kwDOBm6k_c41lFVd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T16:14:12Z", "updated_at": "2021-08-14T16:14:12Z", "author_association": "OWNER", "body": "I would feel a lot more comfortable about all of this if I had a robust mechanism for running the Datasette test suite against multiple versions of SQLite itself.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898933865", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898933865, "node_id": "IC_kwDOBm6k_c41lKRp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T17:27:16Z", "updated_at": "2021-08-14T17:28:29Z", "author_association": "OWNER", "body": "Maybe I split this out into a separate Python library that gets tested against *every* SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself?\r\n\r\nDatasette could depend on that library. The library could be released independently of Datasette any time a new SQLite version comes out.\r\n\r\nI could even run a separate git scraper repo that checks for new SQLite releases and submits PRs against the library when a new release comes out.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898936068", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898936068, "node_id": "IC_kwDOBm6k_c41lK0E", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T17:44:54Z", "updated_at": "2021-08-14T17:44:54Z", "author_association": "OWNER", "body": "Another interesting query to consider: https://latest.datasette.io/fixtures?sql=explain+select+*+from++pragma_table_info%28+%27123_starts_with_digits%27%29\r\n\r\nThat one shows `VColumn` instead of `Column`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898961535", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898961535, "node_id": "IC_kwDOBm6k_c41lRB_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T21:37:24Z", "updated_at": "2021-08-14T21:37:24Z", "author_association": "OWNER", "body": "Did some more research into building SQLite custom versions via `pysqlite3` - here's what I figured out for macOS (which should hopefully work for Linux too): https://til.simonwillison.net/sqlite/build-specific-sqlite-pysqlite-macos", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/316#issuecomment-898824020", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/316", "id": 898824020, "node_id": "IC_kwDOCGYnMM41kvdU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T05:12:23Z", "updated_at": "2021-08-14T05:12:23Z", "author_association": "OWNER", "body": "No visible backticks on https://sqlite-utils.datasette.io/en/latest/reference.html any more.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 970320615, "label": "Fix visible backticks on reference page"}, "performed_via_github_app": null}