{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997320824", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997320824, "node_id": "IC_kwDOBm6k_c47ceh4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T02:59:57Z", "updated_at": "2021-12-19T03:00:44Z", "author_association": "OWNER", "body": "To list all indexes: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++index_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_index_list%28sqlite_master.name%29+AS+index_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27\r\n\r\n```sql\r\nSELECT\r\n sqlite_master.name,\r\n index_list.*\r\nFROM\r\n sqlite_master,\r\n pragma_index_list(sqlite_master.name) AS index_list\r\nWHERE\r\n sqlite_master.type = 'table'\r\n```\r\n\r\nForeign keys: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++foreign_key_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_foreign_key_list%28sqlite_master.name%29+AS+foreign_key_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27\r\n\r\n```sql\r\nSELECT\r\n sqlite_master.name,\r\n foreign_key_list.*\r\nFROM\r\n sqlite_master,\r\n pragma_foreign_key_list(sqlite_master.name) AS foreign_key_list\r\nWHERE\r\n sqlite_master.type = 'table'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997321115", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997321115, "node_id": "IC_kwDOBm6k_c47cemb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T03:03:12Z", "updated_at": "2021-12-19T03:03:12Z", "author_association": "OWNER", "body": "Table columns is a bit harder, because `table_xinfo` is only in SQLite 3.26.0 or higher: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/utils/__init__.py#L565-L581\r\n\r\nSo if that function is available: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_xinfo.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_xinfo%28sqlite_master.name%29+AS+table_xinfo%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27\r\n\r\n```sql\r\nSELECT\r\n sqlite_master.name,\r\n table_xinfo.*\r\nFROM\r\n sqlite_master,\r\n pragma_table_xinfo(sqlite_master.name) AS table_xinfo\r\nWHERE\r\n sqlite_master.type = 'table'\r\n```\r\nAnd otherwise, using `table_info`: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_info.*%2C%0D%0A++0+as+hidden%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_info%28sqlite_master.name%29+AS+table_info%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27\r\n\r\n```sql\r\nSELECT\r\n sqlite_master.name,\r\n table_info.*,\r\n 0 as hidden\r\nFROM\r\n sqlite_master,\r\n pragma_table_info(sqlite_master.name) AS table_info\r\nWHERE\r\n sqlite_master.type = 'table'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997321217", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997321217, "node_id": "IC_kwDOBm6k_c47ceoB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T03:04:16Z", "updated_at": "2021-12-19T03:04:16Z", "author_association": "OWNER", "body": "One thing to watch out for though, from https://sqlite.org/pragma.html#pragfunc\r\n\r\n> The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997321327", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997321327, "node_id": "IC_kwDOBm6k_c47cepv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T03:05:39Z", "updated_at": "2021-12-19T03:05:44Z", "author_association": "OWNER", "body": "This caught me out once before in:\r\n- https://github.com/simonw/datasette/issues/1276\r\n\r\nTurns out Glitch was running SQLite 3.11.0 from 2016-02-15.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997321477", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997321477, "node_id": "IC_kwDOBm6k_c47cesF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T03:07:33Z", "updated_at": "2021-12-19T03:07:33Z", "author_association": "OWNER", "body": "If I want to continue supporting SQLite prior to 3.16.0 (2017-01-02) I'll need this optimization to only kick in with versions that support table-valued PRAGMA functions, while keeping the old `PRAGMA foreign_key_list(table)` stuff working for those older versions.\r\n\r\nThat's feasible, but it's a bit more work - and I need to make sure I have robust testing in place for SQLite 3.15.0.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997321653", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997321653, "node_id": "IC_kwDOBm6k_c47ceu1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T03:09:43Z", "updated_at": "2021-12-19T03:09:43Z", "author_association": "OWNER", "body": "On that same documentation page I just spotted this:\r\n\r\n> This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported. \r\n\r\nThis makes me nervous to rely on pragma function optimizations in Datasette itself.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997321767", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997321767, "node_id": "IC_kwDOBm6k_c47cewn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T03:10:58Z", "updated_at": "2021-12-19T03:10:58Z", "author_association": "OWNER", "body": "I wonder how much overhead there is switching between the `async` event loop main code and the thread that runs the SQL queries.\r\n\r\nWould there be a performance boost if I gathered all of the column/index information in a single function run on the thread using `db.execute_fn()` I wonder? It would eliminate a bunch of switching between threads.\r\n\r\nWould be great to understand how much of an impact that would have.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997324156", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997324156, "node_id": "IC_kwDOBm6k_c47cfV8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T03:40:05Z", "updated_at": "2021-12-19T03:40:05Z", "author_association": "OWNER", "body": "Using the prototype of this:\r\n- https://github.com/simonw/datasette-pretty-traces/issues/5\r\n\r\nI'm seeing about 180ms spent running all of these queries on startup!\r\n\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997324666", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997324666, "node_id": "IC_kwDOBm6k_c47cfd6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T03:47:51Z", "updated_at": "2021-12-19T03:48:09Z", "author_association": "OWNER", "body": "Here's a hacked together prototype of running all of that stuff inside a single function passed to `.execute_fn()`:\r\n\r\n```diff\r\ndiff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py\r\nindex 95055d8..58f9982 100644\r\n--- a/datasette/utils/internal_db.py\r\n+++ b/datasette/utils/internal_db.py\r\n@@ -1,4 +1,5 @@\r\n import textwrap\r\n+from datasette.utils import table_column_details\r\n \r\n \r\n async def init_internal_db(db):\r\n@@ -70,49 +71,70 @@ async def populate_schema_tables(internal_db, db):\r\n \"DELETE FROM tables WHERE database_name = ?\", [database_name], block=True\r\n )\r\n tables = (await db.execute(\"select * from sqlite_master WHERE type = 'table'\")).rows\r\n- tables_to_insert = []\r\n- columns_to_delete = []\r\n- columns_to_insert = []\r\n- foreign_keys_to_delete = []\r\n- foreign_keys_to_insert = []\r\n- indexes_to_delete = []\r\n- indexes_to_insert = []\r\n \r\n- for table in tables:\r\n- table_name = table[\"name\"]\r\n- tables_to_insert.append(\r\n- (database_name, table_name, table[\"rootpage\"], table[\"sql\"])\r\n- )\r\n- columns_to_delete.append((database_name, table_name))\r\n- columns = await db.table_column_details(table_name)\r\n- columns_to_insert.extend(\r\n- {\r\n- **{\"database_name\": database_name, \"table_name\": table_name},\r\n- **column._asdict(),\r\n- }\r\n- for column in columns\r\n- )\r\n- foreign_keys_to_delete.append((database_name, table_name))\r\n- foreign_keys = (\r\n- await db.execute(f\"PRAGMA foreign_key_list([{table_name}])\")\r\n- ).rows\r\n- foreign_keys_to_insert.extend(\r\n- {\r\n- **{\"database_name\": database_name, \"table_name\": table_name},\r\n- **dict(foreign_key),\r\n- }\r\n- for foreign_key in foreign_keys\r\n- )\r\n- indexes_to_delete.append((database_name, table_name))\r\n- indexes = (await db.execute(f\"PRAGMA index_list([{table_name}])\")).rows\r\n- indexes_to_insert.extend(\r\n- {\r\n- **{\"database_name\": database_name, \"table_name\": table_name},\r\n- **dict(index),\r\n- }\r\n- for index in indexes\r\n+ def collect_info(conn):\r\n+ tables_to_insert = []\r\n+ columns_to_delete = []\r\n+ columns_to_insert = []\r\n+ foreign_keys_to_delete = []\r\n+ foreign_keys_to_insert = []\r\n+ indexes_to_delete = []\r\n+ indexes_to_insert = []\r\n+\r\n+ for table in tables:\r\n+ table_name = table[\"name\"]\r\n+ tables_to_insert.append(\r\n+ (database_name, table_name, table[\"rootpage\"], table[\"sql\"])\r\n+ )\r\n+ columns_to_delete.append((database_name, table_name))\r\n+ columns = table_column_details(conn, table_name)\r\n+ columns_to_insert.extend(\r\n+ {\r\n+ **{\"database_name\": database_name, \"table_name\": table_name},\r\n+ **column._asdict(),\r\n+ }\r\n+ for column in columns\r\n+ )\r\n+ foreign_keys_to_delete.append((database_name, table_name))\r\n+ foreign_keys = conn.execute(\r\n+ f\"PRAGMA foreign_key_list([{table_name}])\"\r\n+ ).fetchall()\r\n+ foreign_keys_to_insert.extend(\r\n+ {\r\n+ **{\"database_name\": database_name, \"table_name\": table_name},\r\n+ **dict(foreign_key),\r\n+ }\r\n+ for foreign_key in foreign_keys\r\n+ )\r\n+ indexes_to_delete.append((database_name, table_name))\r\n+ indexes = conn.execute(f\"PRAGMA index_list([{table_name}])\").fetchall()\r\n+ indexes_to_insert.extend(\r\n+ {\r\n+ **{\"database_name\": database_name, \"table_name\": table_name},\r\n+ **dict(index),\r\n+ }\r\n+ for index in indexes\r\n+ )\r\n+ return (\r\n+ tables_to_insert,\r\n+ columns_to_delete,\r\n+ columns_to_insert,\r\n+ foreign_keys_to_delete,\r\n+ foreign_keys_to_insert,\r\n+ indexes_to_delete,\r\n+ indexes_to_insert,\r\n )\r\n \r\n+ (\r\n+ tables_to_insert,\r\n+ columns_to_delete,\r\n+ columns_to_insert,\r\n+ foreign_keys_to_delete,\r\n+ foreign_keys_to_insert,\r\n+ indexes_to_delete,\r\n+ indexes_to_insert,\r\n+ ) = await db.execute_fn(collect_info)\r\n+\r\n await internal_db.execute_write_many(\r\n \"\"\"\r\n INSERT INTO tables (database_name, table_name, rootpage, sql)\r\n```\r\nFirst impressions: it looks like this helps **a lot** - as far as I can tell this is now taking around 21ms to get to the point at which all of those internal databases have been populated, where previously it took more than 180ms.\r\n\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997325189", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997325189, "node_id": "IC_kwDOBm6k_c47cfmF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T03:55:01Z", "updated_at": "2021-12-19T20:54:51Z", "author_association": "OWNER", "body": "It's a bit annoying that the queries no longer show up in the trace at all now, thanks to running in `.execute_fn()`. I wonder if there's something smart I can do about that - maybe have `trace()` record that function with a traceback even though it doesn't have the executed SQL string?\r\n\r\n5fac26aa221a111d7633f2dd92014641f7c0ade9 has the same problem.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997342494", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997342494, "node_id": "IC_kwDOBm6k_c47cj0e", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T07:22:04Z", "updated_at": "2021-12-19T07:22:04Z", "author_association": "OWNER", "body": "Another option would be to provide an abstraction that makes it easier to run a group of SQL queries in the same thread at the same time, and have them traced correctly.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1566#issuecomment-997457790", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1566", "id": 997457790, "node_id": "IC_kwDOBm6k_c47c_9-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T20:40:50Z", "updated_at": "2021-12-19T20:40:57Z", "author_association": "OWNER", "body": "Also release new version of `datasette-pretty-traces` with this feature:\r\n- https://github.com/simonw/datasette-pretty-traces/issues/7", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1083669410, "label": "Release Datasette 0.60"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997459637", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997459637, "node_id": "IC_kwDOBm6k_c47dAa1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T20:53:46Z", "updated_at": "2021-12-19T20:53:46Z", "author_association": "OWNER", "body": "Using #1571 showed me that the `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ? and table_name = ?` queries were running way more times than I expected. I came up with a new optimization that just does `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ?` instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997459958", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997459958, "node_id": "IC_kwDOBm6k_c47dAf2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T20:55:59Z", "updated_at": "2021-12-19T20:55:59Z", "author_association": "OWNER", "body": "Closing this issue because I've optimized this a whole bunch, and it's definitely good enough for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1570#issuecomment-997460061", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1570", "id": 997460061, "node_id": "IC_kwDOBm6k_c47dAhd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T20:56:54Z", "updated_at": "2021-12-19T20:56:54Z", "author_association": "OWNER", "body": "Documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1083921371, "label": "Separate db.execute_write() into three methods"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1547#issuecomment-997460731", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1547", "id": 997460731, "node_id": "IC_kwDOBm6k_c47dAr7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T21:02:15Z", "updated_at": "2021-12-19T21:02:15Z", "author_association": "OWNER", "body": "Yes, this is a bug. It looks like the problem is with the `if write:` branch in this code here: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L252-L327\r\n\r\nIs missing this bit of code:\r\n\r\nhttps://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L343-L347", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1076388044, "label": "Writable canned queries fail to load custom templates"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1573#issuecomment-997462117", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1573", "id": 997462117, "node_id": "IC_kwDOBm6k_c47dBBl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T21:13:13Z", "updated_at": "2021-12-19T21:13:13Z", "author_association": "OWNER", "body": "This might also be the impetus I need to bring the https://datasette.io/plugins/datasette-pretty-traces plugin into Datasette core itself.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1084185188, "label": "Make trace() a documented internal API"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1545#issuecomment-997462604", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1545", "id": 997462604, "node_id": "IC_kwDOBm6k_c47dBJM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T21:17:08Z", "updated_at": "2021-12-19T21:17:08Z", "author_association": "OWNER", "body": "Here's the relevant code: https://github.com/simonw/datasette/blob/4094741c2881c2ada3f3f878b532fdaec7914953/datasette/app.py#L1204-L1219\r\n\r\nIt's using `route_path.split(\"/\")` which should be OK because that's the incoming `request.path` path - which I would expect to use `/` even on Windows. Then it uses `os.path.join` which should do the right thing.\r\n\r\nI need to get myself a proper Windows development environment setup to investigate this one.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1075893249, "label": "Custom pages don't work on windows"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1566#issuecomment-997470633", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1566", "id": 997470633, "node_id": "IC_kwDOBm6k_c47dDGp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T22:12:00Z", "updated_at": "2021-12-19T22:12:00Z", "author_association": "OWNER", "body": "Released another alpha, 0.60a1: https://github.com/simonw/datasette/releases/tag/0.60a1", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1083669410, "label": "Release Datasette 0.60"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1547#issuecomment-997471672", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1547", "id": 997471672, "node_id": "IC_kwDOBm6k_c47dDW4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T22:18:26Z", "updated_at": "2021-12-19T22:18:26Z", "author_association": "OWNER", "body": "I released this [in an alpha](https://github.com/simonw/datasette/releases/tag/0.60a1), so you can try out this fix using:\r\n\r\n pip install datasette==0.60a1", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1076388044, "label": "Writable canned queries fail to load custom templates"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-997472214", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 997472214, "node_id": "IC_kwDOBm6k_c47dDfW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T22:22:08Z", "updated_at": "2021-12-19T22:22:08Z", "author_association": "OWNER", "body": "I sketched out a chained SQL builder pattern that might be useful for further tidying up this code - though with the new plugin hook I'm less excited about it than I was:\r\n\r\n```python\r\nclass TableQuery:\r\n def __init__(self, table, columns, pks, is_view=False, prev=None):\r\n self.table = table\r\n self.columns = columns\r\n self.pks = pks\r\n self.is_view = is_view\r\n self.prev = prev\r\n \r\n # These can be changed for different instances in the chain:\r\n self._where_clauses = None\r\n self._order_by = None\r\n self._page_size = None\r\n self._offset = None\r\n self._select_columns = None\r\n\r\n self.select_all_columns = '*'\r\n self.select_specified_columns = '*'\r\n\r\n @property\r\n def where_clauses(self):\r\n wheres = []\r\n current = self\r\n while current:\r\n if current._where_clauses is not None:\r\n wheres.extend(current._where_clauses)\r\n current = current.prev\r\n return list(reversed(wheres))\r\n\r\n def where(self, where):\r\n new_cls = TableQuery(self.table, self.columns, self.pks, self.is_view, self)\r\n new_cls._where_clauses = [where]\r\n return new_cls\r\n \r\n @classmethod\r\n async def introspect(cls, db, table):\r\n return cls(\r\n table,\r\n columns = await db.table_columns(table),\r\n pks = await db.primary_keys(table),\r\n is_view = bool(await db.get_view_definition(table))\r\n )\r\n \r\n @property\r\n def sql_from(self):\r\n return f\"from {self.table}{self.sql_where}\"\r\n\r\n @property\r\n def sql_where(self):\r\n if not self.where_clauses:\r\n return \"\"\r\n else:\r\n return f\" where {' and '.join(self.where_clauses)}\"\r\n\r\n @property\r\n def sql_no_order_no_limit(self):\r\n return f\"select {self.select_all_columns} from {self.table}{self.sql_where}\"\r\n\r\n @property\r\n def sql(self):\r\n return f\"select {self.select_specified_columns} from {self.table} {self.sql_where}{self._order_by} limit {self._page_size}{self._offset}\"\r\n\r\n @property\r\n def sql_count(self):\r\n return f\"select count(*) {self.sql_from}\"\r\n\r\n\r\n def __repr__(self):\r\n return f\"\"\r\n```\r\nUsage:\r\n```python\r\nfrom datasette.app import Datasette\r\nds = Datasette(memory=True, files=[\"/Users/simon/Dropbox/Development/datasette/fixtures.db\"])\r\ndb = ds.get_database(\"fixtures\")\r\nquery = await TableQuery.introspect(db, \"facetable\")\r\nprint(query.where(\"foo = bar\").where(\"baz = 1\").sql_count)\r\n# 'select count(*) from facetable where foo = bar and baz = 1'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1565#issuecomment-997472370", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1565", "id": 997472370, "node_id": "IC_kwDOBm6k_c47dDhy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T22:23:36Z", "updated_at": "2021-12-19T22:23:36Z", "author_association": "OWNER", "body": "This should also expose the JSON API endpoints used to execute SQL against this database.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1083657868, "label": "Documented JavaScript variables on different templates made available for plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1565#issuecomment-997472509", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1565", "id": 997472509, "node_id": "IC_kwDOBm6k_c47dDj9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T22:24:50Z", "updated_at": "2021-12-19T22:24:50Z", "author_association": "OWNER", "body": "... huh, it could even expose a JavaScript function that can be called to execute a SQL query.\r\n\r\n```javascript\r\ndatasette.query(\"select * from blah\").then(...)\r\n```\r\nMaybe it takes an optional second argument that specifies the database - defaulting to the one for the current page.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1083657868, "label": "Documented JavaScript variables on different templates made available for plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1565#issuecomment-997472639", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1565", "id": 997472639, "node_id": "IC_kwDOBm6k_c47dDl_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T22:25:50Z", "updated_at": "2021-12-19T22:25:50Z", "author_association": "OWNER", "body": "Or...\r\n```javascript\r\nrows = await datasette.query`select * from searchable where id > ${id}`;\r\n```\r\nAnd it knows how to turn that into a parameterized call using tagged template literals.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1083657868, "label": "Documented JavaScript variables on different templates made available for plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1565#issuecomment-997473856", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1565", "id": 997473856, "node_id": "IC_kwDOBm6k_c47dD5A", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T22:35:20Z", "updated_at": "2021-12-19T22:35:20Z", "author_association": "OWNER", "body": "Quick prototype of that tagged template `query` function:\r\n\r\n```javascript\r\nfunction query(pieces, ...parameters) {\r\n var qs = new URLSearchParams();\r\n var sql = pieces[0];\r\n parameters.forEach((param, i) => {\r\n sql += `:p${i}${pieces[i + 1]}`;\r\n qs.append(`p${i}`, param);\r\n });\r\n qs.append(\"sql\", sql);\r\n return qs.toString();\r\n}\r\n\r\nvar id = 4;\r\nconsole.log(query`select * from ids where id > ${id}`);\r\n```\r\nOutputs:\r\n```\r\np0=4&sql=select+*+from+ids+where+id+%3E+%3Ap0\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1083657868, "label": "Documented JavaScript variables on different templates made available for plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1565#issuecomment-997474022", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1565", "id": 997474022, "node_id": "IC_kwDOBm6k_c47dD7m", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T22:36:49Z", "updated_at": "2021-12-19T22:37:29Z", "author_association": "OWNER", "body": "No way with a tagged template literal to pass an extra database name argument, so instead I need a method that returns a callable that can be used for the tagged template literal for a specific database - or the default database.\r\n\r\nThis could work (bit weird looking though):\r\n```javascript\r\nvar rows = await datasette.query(\"fixtures\")`select * from foo`;\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1083657868, "label": "Documented JavaScript variables on different templates made available for plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997485361", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 997485361, "node_id": "IC_kwDOCGYnMM47dGsx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T23:45:30Z", "updated_at": "2021-12-19T23:45:30Z", "author_association": "OWNER", "body": "Really interesting example input for this: https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/iOS13.txt - see https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997486156", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 997486156, "node_id": "IC_kwDOCGYnMM47dG5M", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T23:51:02Z", "updated_at": "2021-12-19T23:51:02Z", "author_association": "OWNER", "body": "This is going to need a `--import` multi option too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null}