{"html_url": "https://github.com/simonw/datasette/issues/2058#issuecomment-1504295345", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2058", "id": 1504295345, "node_id": "IC_kwDOBm6k_c5Zqbmx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-12T00:01:42Z", "updated_at": "2023-04-12T00:02:26Z", "author_association": "OWNER", "body": "Here's the relevant code:\r\n\r\nhttps://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/app.py#L421-L437\r\n\r\nThis function is called on almost every request (everything that subclasses `BaseView` at least - need to remember that for the refactor in #2053 etc).\r\n\r\nhttps://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/views/base.py#L101-L103\r\n\r\nIt uses `PRAGMA schema_version` as a cheap way to determine if the schema has changed, in which case it needs to refresh the internal schema tables.\r\n\r\nThis was already the cause of a subtle bug here:\r\n- #1231\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1663399821, "label": "500 \"attempt to write a readonly database\" error caused by \"PRAGMA schema_version\""}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2058#issuecomment-1504298448", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2058", "id": 1504298448, "node_id": "IC_kwDOBm6k_c5ZqcXQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-12T00:04:01Z", "updated_at": "2023-04-12T00:04:01Z", "author_association": "OWNER", "body": "Here's a potential workaround: when I store the schema versions, I could also score an MD5 hash of the full schema (`select group_concat(sql) from sqlite_master`). When I read the schema version with `PRAGMA schema_version` I could catch that exception and, if I see it, I could calculate that MD5 hash again as a fallback and use that to determine if the schema has changed instead.\r\n\r\nThe performance overhead of this needs investigating - how much more expensive is `md5(... that SQL query result)` compared to just `PRAGMA schema_version`, especially on a database with a lot of tables?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1663399821, "label": "500 \"attempt to write a readonly database\" error caused by \"PRAGMA schema_version\""}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2058#issuecomment-1504315697", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2058", "id": 1504315697, "node_id": "IC_kwDOBm6k_c5Zqgkx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-12T00:16:22Z", "updated_at": "2023-04-12T00:27:12Z", "author_association": "OWNER", "body": "I got ChatGPT (code execution alpha) to run a micro-benchmark for me. This was the conclusion:\r\n\r\n> The benchmark using `PRAGMA schema_version` is approximately 1.36 times faster than the benchmark using `hashlib.md5` for the case with 100 tables. For the case with 200 tables, the benchmark using `PRAGMA schema_version` is approximately 2.33 times faster than the benchmark using `hashlib.md5`.\r\n\r\nHere's the chart it drew me:\r\n\r\n\r\n\r\n(It's a pretty rubbish chart though, it only took measurements at 100 and 200 and drew a line between the two, I should have told it to measure every 10 and plot that)\r\n\r\nAnd the full transcript: https://gist.github.com/simonw/2fc46effbfbe49e6de0bcfdc9e31b235\r\n\r\nThe benchmark looks good enough on first glance that I don't feel the need to be more thorough with it. `PRAGMA schema_version` is faster, but not so fast that I feel like the MD5 hack is worth worrying about too much.\r\n\r\nI'm tempted to add something to the `/-/versions` page that tries to identify if this is a problem or not though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1663399821, "label": "500 \"attempt to write a readonly database\" error caused by \"PRAGMA schema_version\""}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2058#issuecomment-1504328395", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2058", "id": 1504328395, "node_id": "IC_kwDOBm6k_c5ZqjrL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-12T00:28:38Z", "updated_at": "2023-04-12T00:28:38Z", "author_association": "OWNER", "body": "Here's a much better chart, which shows that MD5 performance unsurprisingly gets worse as the number of tables increases while `schema_version` remains constant:\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1663399821, "label": "500 \"attempt to write a readonly database\" error caused by \"PRAGMA schema_version\""}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/2058#issuecomment-1504426792", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/2058", "id": 1504426792, "node_id": "IC_kwDOBm6k_c5Zq7so", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-12T02:02:42Z", "updated_at": "2023-04-12T02:02:42Z", "author_association": "OWNER", "body": "I tightened up the benchmark (it was measuring the time taken to create the tables too) and got this:\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1663399821, "label": "500 \"attempt to write a readonly database\" error caused by \"PRAGMA schema_version\""}, "performed_via_github_app": null}