{"html_url": "https://github.com/simonw/datasette/issues/1273#issuecomment-813061516", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1273", "id": 813061516, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzA2MTUxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T16:32:40Z", "updated_at": "2021-04-04T16:32:40Z", "author_association": "OWNER", "body": "Useful tutorial series from 2012: https://northredoubt.com/n/2012/01/20/spatialite-speed-test/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 838382890, "label": "Refresh SpatiaLite documentation"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1292#issuecomment-813109789", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1292", "id": 813109789, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzEwOTc4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T22:37:47Z", "updated_at": "2021-04-04T22:37:47Z", "author_association": "OWNER", "body": "Could maybe replace this code: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/utils/__init__.py#L1021-L1026", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849975810, "label": "Research ctypes.util.find_library('spatialite')"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813112546", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813112546, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExMjU0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:02:45Z", "updated_at": "2021-04-04T23:02:45Z", "author_association": "OWNER", "body": "I've done various pieces of research into this over the past few years. Capturing what I've discovered in this ticket.\r\n\r\nThe SQLite C API has functions that can help with this: https://www.sqlite.org/c3ref/column_database_name.html details those. But they're not exposed in the Python SQLite library.\r\n\r\nMaybe it would be possible to use them via `ctypes`? My hunch is that I would have to re-implement the full `sqlite3` module with `ctypes`, which sounds daunting.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813113175", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813113175, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExMzE3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:07:01Z", "updated_at": "2021-04-04T23:07:01Z", "author_association": "OWNER", "body": "A more promising route I found involved the `db.set_authorizer` method. This can be used to log the permission checks that SQLite uses, including checks for permission to access specific columns of specific tables. For a while I thought this could work!\r\n\r\n```pycon\r\n>>> def print_args(*args, **kwargs):\r\n...    print(\"args\", args, \"kwargs\", kwargs)\r\n...    return sqlite3.SQLITE_OK\r\n\r\n>>> db = sqlite3.connect(\"fixtures.db\")\r\n>>> db.execute('select * from compound_primary_key join facetable on rowid').fetchall()\r\nargs (21, None, None, None, None) kwargs {}\r\nargs (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}\r\nargs (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}\r\nargs (20, 'compound_primary_key', 'content', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'pk', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'created', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'planet_int', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'on_earth', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'state', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'city_id', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'neighborhood', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'tags', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'complex_array', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}\r\n```\r\nThose `20` values (where 20 is `SQLITE_READ`) looked like they were checking permissions for the columns in the order they would be returned!\r\n\r\nThen I found a snag:\r\n\r\n```pycon\r\nIn [18]: db.execute('select 1 + 1 + (select max(rowid) from facetable)')\r\nargs (21, None, None, None, None) kwargs {}\r\nargs (31, None, 'max', None, None) kwargs {}\r\nargs (20, 'facetable', 'pk', 'main', None) kwargs {}\r\nargs (21, None, None, None, None) kwargs {}\r\nargs (20, 'facetable', '', None, None) kwargs {}\r\n```\r\nOnce a subselect is involved the order of the `20` checks no longer matches the order in which the columns are returned from the query.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813113218", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813113218, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExMzIxOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:07:25Z", "updated_at": "2021-04-04T23:07:25Z", "author_association": "OWNER", "body": "Here are all of the available constants:\r\n```pycon\r\nIn [3]: for k in dir(sqlite3):\r\n   ...:     if k.startswith(\"SQLITE_\"):\r\n   ...:         print(k, getattr(sqlite3, k))\r\n   ...: \r\nSQLITE_ALTER_TABLE 26\r\nSQLITE_ANALYZE 28\r\nSQLITE_ATTACH 24\r\nSQLITE_CREATE_INDEX 1\r\nSQLITE_CREATE_TABLE 2\r\nSQLITE_CREATE_TEMP_INDEX 3\r\nSQLITE_CREATE_TEMP_TABLE 4\r\nSQLITE_CREATE_TEMP_TRIGGER 5\r\nSQLITE_CREATE_TEMP_VIEW 6\r\nSQLITE_CREATE_TRIGGER 7\r\nSQLITE_CREATE_VIEW 8\r\nSQLITE_CREATE_VTABLE 29\r\nSQLITE_DELETE 9\r\nSQLITE_DENY 1\r\nSQLITE_DETACH 25\r\nSQLITE_DONE 101\r\nSQLITE_DROP_INDEX 10\r\nSQLITE_DROP_TABLE 11\r\nSQLITE_DROP_TEMP_INDEX 12\r\nSQLITE_DROP_TEMP_TABLE 13\r\nSQLITE_DROP_TEMP_TRIGGER 14\r\nSQLITE_DROP_TEMP_VIEW 15\r\nSQLITE_DROP_TRIGGER 16\r\nSQLITE_DROP_VIEW 17\r\nSQLITE_DROP_VTABLE 30\r\nSQLITE_FUNCTION 31\r\nSQLITE_IGNORE 2\r\nSQLITE_INSERT 18\r\nSQLITE_OK 0\r\nSQLITE_PRAGMA 19\r\nSQLITE_READ 20\r\nSQLITE_RECURSIVE 33\r\nSQLITE_REINDEX 27\r\nSQLITE_SAVEPOINT 32\r\nSQLITE_SELECT 21\r\nSQLITE_TRANSACTION 22\r\nSQLITE_UPDATE 23\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813113403", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813113403, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExMzQwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:08:48Z", "updated_at": "2021-04-04T23:08:48Z", "author_association": "OWNER", "body": "Worth noting that adding `limit 0` to the query still causes it to conduct the permission checks, hopefully while avoiding doing any of the actual work of executing the query:\r\n```pycon\r\nIn [20]: db.execute('select * from compound_primary_key join facetable on facetable.rowid = compound_primary_key.rowid limit 0').fetchall()\r\n    ...: \r\nargs (21, None, None, None, None) kwargs {}\r\nargs (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}\r\nargs (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}\r\nargs (20, 'compound_primary_key', 'content', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'pk', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'created', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'planet_int', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'on_earth', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'state', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'city_id', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'neighborhood', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'tags', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'complex_array', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'pk', 'main', None) kwargs {}\r\nargs (20, 'compound_primary_key', 'ROWID', 'main', None) kwargs {}\r\nOut[20]: []\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813113653", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813113653, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExMzY1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:10:49Z", "updated_at": "2021-04-04T23:10:49Z", "author_association": "OWNER", "body": "One option I've not fully explored yet: could I write my own custom SQLite C extension which exposes this functionality as a callable function?\r\n\r\nThen I could load that extension and run a SQL query something like this:\r\n\r\n```\r\nselect database, table, column from analyze_query(:sql_query)\r\n```\r\nWhere `analyze_query(...)` would be a fancy virtual table function of some sort that uses the underlying `sqlite3_column_database_name()` C functions to analyze the SQL query and return details of what it would return.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813114933", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813114933, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExNDkzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:19:22Z", "updated_at": "2021-04-04T23:19:22Z", "author_association": "OWNER", "body": "I asked about this on the SQLite forum: https://sqlite.org/forum/forumpost/0180277fb7", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813115414", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813115414, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExNTQxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:23:34Z", "updated_at": "2021-04-04T23:23:34Z", "author_association": "OWNER", "body": "The other approach I considered for this was to have my own SQL query parser running in Python, which could pick apart a complex query and figure out which column was sourced from which table. I dropped this idea because it felt that the moment `select *` came into play a pure parsing approach wouldn't work - I'd need knowledge of the schema in order to resolve the `*`.\r\n\r\nA Python parser approach might be good enough to handle a subset of queries - those that don't use `select *` for example - and maybe that would be worth shipping? The feature doesn't have to be perfect for it to be useful.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813115607", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813115607, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExNTYwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:25:15Z", "updated_at": "2021-04-04T23:25:15Z", "author_association": "OWNER", "body": "Oh wow, I just spotted https://github.com/macbre/sql-metadata\r\n\r\n> Uses tokenized query returned by python-sqlparse and generates query metadata. Extracts column names and tables used by the query. Provides a helper for normalization of SQL queries and tables aliases resolving.\r\n\r\nIt's for MySQL, PostgreSQL and Hive right now but maybe getting it working with SQLite wouldn't be too hard?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813116177", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813116177, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExNjE3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:31:00Z", "updated_at": "2021-04-04T23:31:00Z", "author_association": "OWNER", "body": "Sadly it doesn't do what I need. This query should only return one column, but instead I get back every column that was consulted by the query:\r\n\r\n<img width=\"597\" alt=\"sql-metadata_-_Jupyter_Notebook\" src=\"https://user-images.githubusercontent.com/9599/113524385-216ca000-9563-11eb-8a7e-cbbe5dfc02f6.png\">\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}