{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813162622", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813162622, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzE2MjYyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T03:34:24Z", "updated_at": "2021-04-05T03:40:35Z", "author_association": "OWNER", "body": "This almost works, but throws errors with some queries (anything with a `rowid` column for example) - it needs a bunch of test coverage.\r\n```python\r\ndef columns_for_query(conn, sql):\r\n    rows = conn.execute('explain ' + sql).fetchall()\r\n    table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'}\r\n    names_by_rootpage = dict(\r\n        conn.execute(\r\n            'select rootpage, name from sqlite_master where rootpage in ({})'.format(\r\n                ', '.join(map(str, table_rootpage_by_register.values()))\r\n            )\r\n        )\r\n    )\r\n    columns_by_column_register = {}\r\n    for row in rows:\r\n        if row['opcode'] == 'Column':\r\n            addr, opcode, table_id, cid, column_register, p4, p5, comment = row\r\n            table = names_by_rootpage[table_rootpage_by_register[table_id]]\r\n            columns_by_column_register[column_register] = (table, cid)\r\n    result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0]\r\n    registers = list(range(result_row[\"p1\"], result_row[\"p1\"] + result_row[\"p2\"] - 1))\r\n    all_column_names = {}\r\n    for table in names_by_rootpage.values():\r\n        table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall()\r\n        for row in table_xinfo:\r\n            all_column_names[(table, row[\"cid\"])] = row[\"name\"]\r\n    final_output = []\r\n    for r in registers:\r\n        try:\r\n            table, cid = columns_by_column_register[r]\r\n            final_output.append((table, all_column_names[table, cid]))\r\n        except KeyError:\r\n            final_output.append((None, None))\r\n    return final_output\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813164282", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813164282, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzE2NDI4Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T03:42:26Z", "updated_at": "2021-04-05T03:42:36Z", "author_association": "OWNER", "body": "Extracting variables with this trick appears to work OK, but you have to pass the correct variables to the `explain select...` query. Using `defaultdict` seems to work there:\r\n\r\n```pycon\r\n>>> rows = conn.execute('explain select * from repos where id = :id', defaultdict(int))\r\n>>> [dict(r) for r in rows if r['opcode'] == 'Variable']\r\n[{'addr': 2,\r\n  'opcode': 'Variable',\r\n  'p1': 1,\r\n  'p2': 1,\r\n  'p3': 0,\r\n  'p4': ':id',\r\n  'p5': 0,\r\n  'comment': None}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/620#issuecomment-813167335", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/620", "id": 813167335, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzE2NzMzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T03:57:22Z", "updated_at": "2021-04-05T03:57:22Z", "author_association": "OWNER", "body": "This may be obsoleted by #1293 - it looks like I may be able to auto-detect these foreign keys for arbitrary queries after all.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 520667773, "label": "Mechanism for indicating foreign key relationships in the table and query page URLs"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813134072", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813134072, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzEzNDA3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T01:18:37Z", "updated_at": "2021-04-05T01:18:37Z", "author_association": "OWNER", "body": "Had a fantastic suggestion on the SQLite forum: it might be possible to get what I want by interpreting the opcodes output by `explain select ...`.\r\n\r\nCopying the reply I posted to this thread:\r\n\r\nThat's really useful, thanks! It looks like it _might_ be possible for me to reconstruct where each column came from using the `explain select` output.\r\n\r\nHere's a complex example: <https://calands.datasettes.com/calands?sql=explain+select%0D%0A++AsGeoJSON%28geometry%29%2C+*%0D%0Afrom%0D%0A++CPAD_2020a_SuperUnits%0D%0Awhere%0D%0A++PARK_NAME+like+%27%25mini%25%27+and%0D%0A++Intersects%28GeomFromGeoJSON%28%3Afreedraw%29%2C+geometry%29+%3D+1%0D%0A++and+CPAD_2020a_SuperUnits.rowid+in+%28%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++SpatialIndex%0D%0A++++where%0D%0A++++++f_table_name+%3D+%27CPAD_2020a_SuperUnits%27%0D%0A++++++and+search_frame+%3D+GeomFromGeoJSON%28%3Afreedraw%29%0D%0A++%29&freedraw=%7B%22type%22%3A%22MultiPolygon%22%2C%22coordinates%22%3A%5B%5B%5B%5B-122.42202758789064%2C37.82280243352759%5D%2C%5B-122.39868164062501%2C37.823887203271454%5D%2C%5B-122.38220214843751%2C37.81846319511331%5D%2C%5B-122.35061645507814%2C37.77071473849611%5D%2C%5B-122.34924316406251%2C37.74465712069939%5D%2C%5B-122.37258911132814%2C37.703380457832374%5D%2C%5B-122.39044189453125%2C37.690340943717715%5D%2C%5B-122.41241455078126%2C37.680559803205135%5D%2C%5B-122.44262695312501%2C37.67295135774715%5D%2C%5B-122.47283935546876%2C37.67295135774715%5D%2C%5B-122.52502441406251%2C37.68382032669382%5D%2C%5B-122.53463745117189%2C37.6892542140253%5D%2C%5B-122.54699707031251%2C37.690340943717715%5D%2C%5B-122.55798339843751%2C37.72945260537781%5D%2C%5B-122.54287719726564%2C37.77831314799672%5D%2C%5B-122.49893188476564%2C37.81303878836991%5D%2C%5B-122.46185302734376%2C37.82822612280363%5D%2C%5B-122.42889404296876%2C37.82822612280363%5D%2C%5B-122.42202758789064%2C37.82280243352759%5D%5D%5D%5D%7D>\r\n\r\nIt looks like the opcodes I need to inspect are `OpenRead`, `Column` and `ResultRow`.\r\n\r\n`OpenRead` tells me which tables are being opened - the `p2` value (in this case 51) corresponds to the `rootpage` column in `sqlite_master` here: <https://calands.datasettes.com/calands?sql=select+*+from+sqlite_master> - it gets assigned to the register in `p1`.\r\n\r\nThe `Column` opcodes tell me which columns are being read - `p1` is that table reference, and `p2` is the `cid` of the column within that table.\r\n\r\nThe `ResultRow` opcode then tells me which columns are used in the results. `15 16` means start at the 15th and then read the next 16 columns.\r\n\r\nI think this might work!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813134227", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813134227, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzEzNDIyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T01:19:31Z", "updated_at": "2021-04-05T01:19:31Z", "author_association": "OWNER", "body": "|   addr | opcode        |   p1 |   p2 |   p3 | p4                    |   p5 | comment   |\r\n|--------|---------------|------|------|------|-----------------------|------|-----------|\r\n|      0 | Init          |    0 |   47 |    0 |                       |   00 |           |\r\n|      1 | OpenRead      |    0 |   51 |    0 | 15                    |   00 |           |\r\n|      2 | Integer       |   15 |    2 |    0 |                       |   00 |           |\r\n|      3 | Once          |    0 |   15 |    0 |                       |   00 |           |\r\n|      4 | OpenEphemeral |    2 |    1 |    0 | k(1,)                 |   00 |           |\r\n|      5 | VOpen         |    1 |    0 |    0 | vtab:3E692C362158     |   00 |           |\r\n|      6 | String8       |    0 |    5 |    0 | CPAD_2020a_SuperUnits |   00 |           |\r\n|      7 | SCopy         |    7 |    6 |    0 |                       |   00 |           |\r\n|      8 | Integer       |    2 |    3 |    0 |                       |   00 |           |\r\n|      9 | Integer       |    2 |    4 |    0 |                       |   00 |           |\r\n|     10 | VFilter       |    1 |   15 |    3 |                       |   00 |           |\r\n|     11 | Rowid         |    1 |    8 |    0 |                       |   00 |           |\r\n|     12 | MakeRecord    |    8 |    1 |    9 | C                     |   00 |           |\r\n|     13 | IdxInsert     |    2 |    9 |    8 | 1                     |   00 |           |\r\n|     14 | VNext         |    1 |   11 |    0 |                       |   00 |           |\r\n|     15 | Return        |    2 |    0 |    0 |                       |   00 |           |\r\n|     16 | Rewind        |    2 |   46 |    0 |                       |   00 |           |\r\n|     17 | Column        |    2 |    0 |    1 |                       |   00 |           |\r\n|     18 | IsNull        |    1 |   45 |    0 |                       |   00 |           |\r\n|     19 | SeekRowid     |    0 |   45 |    1 |                       |   00 |           |\r\n|     20 | Column        |    0 |    2 |   11 |                       |   00 |           |\r\n|     21 | Function0     |    1 |   10 |    9 | like(2)               |   02 |           |\r\n|     22 | IfNot         |    9 |   45 |    1 |                       |   00 |           |\r\n|     23 | Column        |    0 |   14 |   13 |                       |   00 |           |\r\n|     24 | Function0     |    1 |   12 |    9 | intersects(2)         |   02 |           |\r\n|     25 | Ne            |   14 |   45 |    9 |                       |   51 |           |\r\n|     26 | Column        |    0 |   14 |    9 |                       |   00 |           |\r\n|     27 | Function0     |    0 |    9 |   15 | asgeojson(1)          |   01 |           |\r\n|     28 | Rowid         |    0 |   16 |    0 |                       |   00 |           |\r\n|     29 | Column        |    0 |    1 |   17 |                       |   00 |           |\r\n|     30 | Column        |    0 |    2 |   18 |                       |   00 |           |\r\n|     31 | Column        |    0 |    3 |   19 |                       |   00 |           |\r\n|     32 | Column        |    0 |    4 |   20 |                       |   00 |           |\r\n|     33 | Column        |    0 |    5 |   21 |                       |   00 |           |\r\n|     34 | Column        |    0 |    6 |   22 |                       |   00 |           |\r\n|     35 | Column        |    0 |    7 |   23 |                       |   00 |           |\r\n|     36 | Column        |    0 |    8 |   24 |                       |   00 |           |\r\n|     37 | Column        |    0 |    9 |   25 |                       |   00 |           |\r\n|     38 | Column        |    0 |   10 |   26 |                       |   00 |           |\r\n|     39 | Column        |    0 |   11 |   27 |                       |   00 |           |\r\n|     40 | RealAffinity  |   27 |    0 |    0 |                       |   00 |           |\r\n|     41 | Column        |    0 |   12 |   28 |                       |   00 |           |\r\n|     42 | Column        |    0 |   13 |   29 |                       |   00 |           |\r\n|     43 | Column        |    0 |   14 |   30 |                       |   00 |           |\r\n|     44 | ResultRow     |   15 |   16 |    0 |                       |   00 |           |\r\n|     45 | Next          |    2 |   17 |    0 |                       |   00 |           |\r\n|     46 | Halt          |    0 |    0 |    0 |                       |   00 |           |\r\n|     47 | Transaction   |    0 |    0 |  265 | 0                     |   01 |           |\r\n|     48 | Variable      |    1 |   31 |    0 | :freedraw             |   00 |           |\r\n|     49 | Function0     |    1 |   31 |    7 | geomfromgeojson(1)    |   01 |           |\r\n|     50 | String8       |    0 |   10 |    0 | %mini%                |   00 |           |\r\n|     51 | Variable      |    1 |   32 |    0 | :freedraw             |   00 |           |\r\n|     52 | Function0     |    1 |   32 |   12 | geomfromgeojson(1)    |   01 |           |\r\n|     53 | Integer       |    1 |   14 |    0 |                       |   00 |           |\r\n|     54 | Goto          |    0 |    1 |    0 |                       |   00 |           |\r\n\r\nEssential documentation for understanding that output: https://www.sqlite.org/opcode.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813134637", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813134637, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzEzNDYzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T01:21:59Z", "updated_at": "2021-04-05T01:21:59Z", "author_association": "OWNER", "body": "http://www.sqlite.org/draft/lang_explain.html says:\r\n\r\n> Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.\r\n\r\nI'm going to keep exploring this though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/35#issuecomment-813249000", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/35", "id": 813249000, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzI0OTAwMA==", "user": {"value": 1151557, "label": "ligurio"}, "created_at": "2021-04-05T07:37:57Z", "updated_at": "2021-04-05T07:37:57Z", "author_association": "NONE", "body": "There are trained ML models used in Photoprism:\r\n- https://dl.photoprism.org/tensorflow/nasnet.zip\r\n- https://dl.photoprism.org/tensorflow/nsfw.zip", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 842695374, "label": "Support to annotate photos on other than  macOS OSes"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813445512", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813445512, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzQ0NTUxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T15:11:40Z", "updated_at": "2021-04-05T15:11:40Z", "author_association": "OWNER", "body": "Here's some older example code that works with opcodes from Python, in this case to output indexes used by a query: https://github.com/plasticityai/supersqlite/blob/master/supersqlite/idxchk.py", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813480043", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813480043, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzQ4MDA0Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T16:16:17Z", "updated_at": "2021-04-05T16:16:17Z", "author_association": "OWNER", "body": "https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because `paginated_view` is defined like this:\r\n\r\n```sql\r\nCREATE VIEW paginated_view AS\r\n    SELECT\r\n        content,\r\n        '- ' || content || ' -' AS content_extra\r\n    FROM no_primary_key;\r\n```\r\nSo this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813438771", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813438771, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzQzODc3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T14:58:48Z", "updated_at": "2021-04-05T14:58:48Z", "author_association": "OWNER", "body": "I may need to do something special for rowid columns - there is a `RowId` opcode that might come into play here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}