{"html_url": "https://github.com/simonw/datasette/issues/1818#issuecomment-1258738740", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1818", "id": 1258738740, "node_id": "IC_kwDOBm6k_c5LBtQ0", "user": {"value": 5363, "label": "nelsonjchen"}, "created_at": "2022-09-26T22:52:45Z", "updated_at": "2022-09-26T22:55:57Z", "author_association": "NONE", "body": "thoughts on order of precedence to use:\r\n\r\n* sqlite-utils count, if present. closest thing to a standard i guess.\r\n* row(max_id) if like, the first and/or last x amount of rows ids are all contiguous. kind of a cheap/dumb/imperfect heuristic to see if the table is dump/not dump. if the check passes, still stick on `est.` after the display.\r\n* count(*) if enabled in datasette ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384549993, "label": "Setting to turn off table row counts entirely"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1258437060", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/490", "id": 1258437060, "node_id": "IC_kwDOCGYnMM5LAjnE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T18:24:44Z", "updated_at": "2022-09-26T18:24:44Z", "author_association": "OWNER", "body": "Just saw your great write-up on this: https://jeqo.github.io/notes/2022-09-24-ingest-logs-sqlite/", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1382457780, "label": "Ability to insert multi-line files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/492#issuecomment-1258446128", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/492", "id": 1258446128, "node_id": "IC_kwDOCGYnMM5LAl0w", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T18:32:14Z", "updated_at": "2022-09-26T18:33:19Z", "author_association": "OWNER", "body": "This idea would make more sense if there was a good mechanism to say \"run the conversion script held in this file\" as opposed to passing it as an option. This would also make having to remember bash escaping rules ([see tip](https://til.simonwillison.net/zsh/argument-heredoc)) much easier!\r\n\r\n`shot-scraper` has that for `--javascript`, using the `--input` option: https://shot-scraper.datasette.io/en/stable/javascript.html#shot-scraper-javascript-help\r\n\r\nMaybe `--convert-script` would work here? Or `--convert-file`? It should accept `-` for stdin too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386530156, "label": "Idea: ability to pass extra variables to `--convert` scripts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258449887", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491", "id": 1258449887, "node_id": "IC_kwDOCGYnMM5LAmvf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T18:35:50Z", "updated_at": "2022-09-26T18:35:50Z", "author_association": "OWNER", "body": "This is a really interesting idea.\r\n\r\nI'm nervous about needing to set the rules for how duplicate tables should be merged though. This feels like a complex topic - one where there isn't necessarily an obviously \"correct\" way of doing it, but where different problems that people are solving might need different merging approaches.\r\n\r\nLikewise, merging isn't just a database-to-database thing at that point - I could see a need for merging two tables using similar rules to those used for merging two databases.\r\n\r\nSo I think I'd want to have some good concrete use-cases in mind before trying to design how something like this should work. Will leave this thread open for people to drop those in!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1383646615, "label": "Ability to merge databases and tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258450447", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491", "id": 1258450447, "node_id": "IC_kwDOCGYnMM5LAm4P", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T18:36:23Z", "updated_at": "2022-09-26T18:36:23Z", "author_association": "OWNER", "body": "This is also the kind of feature that would need to express itself in both the Python library and the CLI utility.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1383646615, "label": "Ability to merge databases and tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/483#issuecomment-1258451968", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/483", "id": 1258451968, "node_id": "IC_kwDOCGYnMM5LAnQA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T18:37:54Z", "updated_at": "2022-09-26T18:40:41Z", "author_association": "OWNER", "body": "The implementation of this can be an almost exact copy of Datasette's, which was added in this commit: https://github.com/simonw/datasette/commit/01fe5b740171bfaea3752fc5754431dac53777e3\r\n\r\nCurrent code for that is here: https://github.com/simonw/datasette/blob/0.62/datasette/cli.py#L319-L340 - which is improved to use the `from runpy import run_module` function.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1363765916, "label": "`sqlite-utils install` command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1258476455", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/493", "id": 1258476455, "node_id": "IC_kwDOCGYnMM5LAtOn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T19:01:49Z", "updated_at": "2022-09-26T19:01:49Z", "author_association": "OWNER", "body": "I tried the tips in https://stackoverflow.com/questions/15258831/how-to-handle-two-dashes-in-rest (not the settings change though, because I might want smart quotes elsewhere) and they didn't work.\r\n\r\nMaybe I should disable smart quotes entirely?\r\n\r\nI feel like there should be an escaping trick that works here though. I tried `insert -\\\\-convert` but it didn't help.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386562662, "label": "Tiny typographical error in install/uninstall docs"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/483#issuecomment-1258479462", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/483", "id": 1258479462, "node_id": "IC_kwDOCGYnMM5LAt9m", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T19:04:29Z", "updated_at": "2022-09-26T19:04:43Z", "author_association": "OWNER", "body": "Documentation:\r\n\r\n- https://sqlite-utils.datasette.io/en/latest/cli.html#cli-install\r\n- https://sqlite-utils.datasette.io/en/latest/cli.html#cli-uninstall\r\n- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#install\r\n- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#uninstall\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1363765916, "label": "`sqlite-utils install` command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/494#issuecomment-1258516872", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/494", "id": 1258516872, "node_id": "IC_kwDOCGYnMM5LA3GI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T19:28:36Z", "updated_at": "2022-09-26T19:28:36Z", "author_association": "OWNER", "body": "New documentation: https://sqlite-utils.datasette.io/en/latest/contributing.html#using-just-and-pipenv", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386593843, "label": "Document how to use Just"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/494#issuecomment-1258521333", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/494", "id": 1258521333, "node_id": "IC_kwDOCGYnMM5LA4L1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T19:32:36Z", "updated_at": "2022-09-26T19:32:36Z", "author_association": "OWNER", "body": "Tweeted about it too: https://twitter.com/simonw/status/1574481628507668480", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386593843, "label": "Document how to use Just"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1821#issuecomment-1258692555", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1821", "id": 1258692555, "node_id": "IC_kwDOBm6k_c5LBh_L", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T22:06:39Z", "updated_at": "2022-09-26T22:06:39Z", "author_association": "OWNER", "body": "- https://github.com/simonw/datasette/actions/runs/3131344150\r\n- https://github.com/simonw/datasette/releases/tag/0.63a0\r\n- https://pypi.org/project/datasette/0.63a0/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386734383, "label": "Release Datasette 0.63a0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258697384", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491", "id": 1258697384, "node_id": "IC_kwDOCGYnMM5LBjKo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T22:12:45Z", "updated_at": "2022-09-26T22:12:45Z", "author_association": "OWNER", "body": "That feels like a slightly different command to me - maybe `sqlite-utils backup data.db data-backup.db`? It doesn't have any of the mechanics for merging tables together. Could be a useful feature separately though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1383646615, "label": "Ability to merge databases and tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1818#issuecomment-1258735283", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1818", "id": 1258735283, "node_id": "IC_kwDOBm6k_c5LBsaz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T22:47:19Z", "updated_at": "2022-09-26T22:47:19Z", "author_association": "OWNER", "body": "That's a really interesting idea: for a lot of databases (those made out of straight imports from CSV) `max(rowid)` would indeed reflect the size of the table, but would be a MUCH faster operation than attempting a `count(*)`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384549993, "label": "Setting to turn off table row counts entirely"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1818#issuecomment-1258735747", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1818", "id": 1258735747, "node_id": "IC_kwDOBm6k_c5LBsiD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T22:47:59Z", "updated_at": "2022-09-26T22:47:59Z", "author_association": "OWNER", "body": "Another option here is to tie into a feature I built in `sqlite-utils` with this problem in mind but never introduced on the Datasette side of things: https://sqlite-utils.datasette.io/en/stable/python-api.html#cached-table-counts-using-triggers", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384549993, "label": "Setting to turn off table row counts entirely"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1819#issuecomment-1258738435", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1819", "id": 1258738435, "node_id": "IC_kwDOBm6k_c5LBtMD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T22:52:19Z", "updated_at": "2022-09-26T22:52:19Z", "author_association": "OWNER", "body": "This is a good idea.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1385026210, "label": "Preserve query on timeout"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1819#issuecomment-1258746600", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1819", "id": 1258746600, "node_id": "IC_kwDOBm6k_c5LBvLo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T23:05:40Z", "updated_at": "2022-09-26T23:05:40Z", "author_association": "OWNER", "body": "Implementing it like this, so at least you can copy and paste the SQL query back out again:\r\n\r\n![\"image\"](\"https://user-images.githubusercontent.com/9599/192395953-48512c94-10e0-4cf8-8ae5-b9e65e3d7b0f.png\")
\r\n\r\nI'm not doing a full textarea because this error can be raised in multiple places, including on the table page itself. It's not just an error associated with the manual query page.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1385026210, "label": "Preserve query on timeout"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1819#issuecomment-1258754105", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1819", "id": 1258754105, "node_id": "IC_kwDOBm6k_c5LBxA5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T23:16:15Z", "updated_at": "2022-09-26T23:16:15Z", "author_association": "OWNER", "body": "Demo: https://latest.datasette.io/_memory?sql=with+recursive+counter(x)+as+(%0D%0A++select+0%0D%0A++++union%0D%0A++select+x+%2B+1+from+counter%0D%0A)%2C%0D%0Ablah+as+(select+*+from+counter+limit+5000000)%0D%0Aselect+count(*)+from+blah", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1385026210, "label": "Preserve query on timeout"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1817#issuecomment-1258756231", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1817", "id": 1258756231, "node_id": "IC_kwDOBm6k_c5LBxiH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T23:19:34Z", "updated_at": "2022-09-26T23:19:34Z", "author_association": "OWNER", "body": "This is a good idea - it's something I should do before Datasette 1.0.\r\n\r\nI was a tiny bit worried about compatibility (Datasette is 3.7+) but it looks like they have been in Python since 3.0!", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1384273985, "label": "Expose `sql` and `params` arguments to various plugin hooks"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1822#issuecomment-1258757544", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1822", "id": 1258757544, "node_id": "IC_kwDOBm6k_c5LBx2o", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T23:21:23Z", "updated_at": "2022-09-26T23:21:23Z", "author_association": "OWNER", "body": "Everything on https://docs.datasette.io/en/stable/internals.html that uses keyword arguments should do this I think.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386854246, "label": "Switch to keyword-only arguments for a bunch of internal methods"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1822#issuecomment-1258760299", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1822", "id": 1258760299, "node_id": "IC_kwDOBm6k_c5LByhr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T23:25:12Z", "updated_at": "2022-09-26T23:25:55Z", "author_association": "OWNER", "body": "A start:\r\n```diff\r\ndiff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py\r\nindex 8a2fa060..41ade961 100644\r\n--- a/datasette/utils/asgi.py\r\n+++ b/datasette/utils/asgi.py\r\n@@ -118,7 +118,7 @@ class Request:\r\n return dict(parse_qsl(body.decode(\"utf-8\"), keep_blank_values=True))\r\n \r\n @classmethod\r\n- def fake(cls, path_with_query_string, method=\"GET\", scheme=\"http\", url_vars=None):\r\n+ def fake(cls, path_with_query_string, *, method=\"GET\", scheme=\"http\", url_vars=None):\r\n \"\"\"Useful for constructing Request objects for tests\"\"\"\r\n path, _, query_string = path_with_query_string.partition(\"?\")\r\n scope = {\r\n@@ -204,7 +204,7 @@ class AsgiWriter:\r\n )\r\n \r\n \r\n-async def asgi_send_json(send, info, status=200, headers=None):\r\n+async def asgi_send_json(send, info, *, status=200, headers=None):\r\n headers = headers or {}\r\n await asgi_send(\r\n send,\r\n@@ -215,7 +215,7 @@ async def asgi_send_json(send, info, status=200, headers=None):\r\n )\r\n \r\n \r\n-async def asgi_send_html(send, html, status=200, headers=None):\r\n+async def asgi_send_html(send, html, *, status=200, headers=None):\r\n headers = headers or {}\r\n await asgi_send(\r\n send,\r\n@@ -226,7 +226,7 @@ async def asgi_send_html(send, html, status=200, headers=None):\r\n )\r\n \r\n \r\n-async def asgi_send_redirect(send, location, status=302):\r\n+async def asgi_send_redirect(send, location, *, status=302):\r\n await asgi_send(\r\n send,\r\n \"\",\r\n@@ -236,12 +236,12 @@ async def asgi_send_redirect(send, location, status=302):\r\n )\r\n \r\n \r\n-async def asgi_send(send, content, status, headers=None, content_type=\"text/plain\"):\r\n+async def asgi_send(send, content, status, *, headers=None, content_type=\"text/plain\"):\r\n await asgi_start(send, status, headers, content_type)\r\n await send({\"type\": \"http.response.body\", \"body\": content.encode(\"utf-8\")})\r\n \r\n \r\n-async def asgi_start(send, status, headers=None, content_type=\"text/plain\"):\r\n+async def asgi_start(send, status, *, headers=None, content_type=\"text/plain\"):\r\n headers = headers or {}\r\n # Remove any existing content-type header\r\n headers = {k: v for k, v in headers.items() if k.lower() != \"content-type\"}\r\n@@ -259,7 +259,7 @@ async def asgi_start(send, status, headers=None, content_type=\"text/plain\"):\r\n \r\n \r\n async def asgi_send_file(\r\n- send, filepath, filename=None, content_type=None, chunk_size=4096, headers=None\r\n+ send, filepath, filename=None, *, content_type=None, chunk_size=4096, headers=None\r\n ):\r\n headers = headers or {}\r\n if filename:\r\n@@ -284,7 +284,7 @@ async def asgi_send_file(\r\n )\r\n \r\n \r\n-def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):\r\n+def asgi_static(root_path, *, chunk_size=4096, headers=None, content_type=None):\r\n root_path = Path(root_path)\r\n \r\n async def inner_static(request, send):\r\n@@ -313,7 +313,7 @@ def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):\r\n \r\n \r\n class Response:\r\n- def __init__(self, body=None, status=200, headers=None, content_type=\"text/plain\"):\r\n+ def __init__(self, body=None, *, status=200, headers=None, content_type=\"text/plain\"):\r\n self.body = body\r\n self.status = status\r\n self.headers = headers or {}\r\n@@ -346,6 +346,7 @@ class Response:\r\n self,\r\n key,\r\n value=\"\",\r\n+ *,\r\n max_age=None,\r\n expires=None,\r\n path=\"/\",\r\n@@ -374,7 +375,7 @@ class Response:\r\n self._set_cookie_headers.append(cookie.output(header=\"\").strip())\r\n \r\n @classmethod\r\n- def html(cls, body, status=200, headers=None):\r\n+ def html(cls, body, *, status=200, headers=None):\r\n return cls(\r\n body,\r\n status=status,\r\n@@ -383,7 +384,7 @@ class Response:\r\n )\r\n \r\n @classmethod\r\n- def text(cls, body, status=200, headers=None):\r\n+ def text(cls, body, *, status=200, headers=None):\r\n return cls(\r\n str(body),\r\n status=status,\r\n@@ -392,7 +393,7 @@ class Response:\r\n )\r\n \r\n @classmethod\r\n- def json(cls, body, status=200, headers=None, default=None):\r\n+ def json(cls, body, *, status=200, headers=None, default=None):\r\n return cls(\r\n json.dumps(body, default=default),\r\n status=status,\r\n@@ -401,7 +402,7 @@ class Response:\r\n )\r\n \r\n @classmethod\r\n- def redirect(cls, path, status=302, headers=None):\r\n+ def redirect(cls, path, *, status=302, headers=None):\r\n headers = headers or {}\r\n headers[\"Location\"] = path\r\n return cls(\"\", status=status, headers=headers)\r\n@@ -412,6 +413,7 @@ class AsgiFileDownload:\r\n self,\r\n filepath,\r\n filename=None,\r\n+ *,\r\n content_type=\"application/octet-stream\",\r\n headers=None,\r\n ):\r\n```\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 03d1dacc..4d4e5584 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -190,6 +190,7 @@ class Datasette:\r\n def __init__(\r\n self,\r\n files=None,\r\n+ *,\r\n immutables=None,\r\n cache_headers=True,\r\n cors=False,\r\n ```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1386854246, "label": "Switch to keyword-only arguments for a bunch of internal methods"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258508215", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491", "id": 1258508215, "node_id": "IC_kwDOCGYnMM5LA0-3", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-09-26T19:22:14Z", "updated_at": "2022-09-26T19:22:14Z", "author_association": "CONTRIBUTOR", "body": "This might be fairly straightforward using SQLite's backup utility: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.backup\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1383646615, "label": "Ability to merge databases and tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258712931", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491", "id": 1258712931, "node_id": "IC_kwDOCGYnMM5LBm9j", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2022-09-26T22:31:58Z", "updated_at": "2022-09-26T22:31:58Z", "author_association": "CONTRIBUTOR", "body": "Right. The backup command will copy tables completely, but in the case of conflicting table names, the destination gets overwritten silently. That might not be what you want here. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1383646615, "label": "Ability to merge databases and tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1258129113", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1258129113, "node_id": "IC_kwDOBm6k_c5K_YbZ", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-09-26T14:30:11Z", "updated_at": "2022-09-26T14:48:31Z", "author_association": "CONTRIBUTOR", "body": "from your analysis, it seems like the GIL is blocking on loading of the data from sqlite to python, (particularly in the `fetchmany` call)\r\n\r\nthis is probably a simplistic idea, but what if you had the python code in the `execute` method iterate over the cursor and yield out rows or small chunks of rows.\r\n\r\nsomething like: \r\n```python\r\n with sqlite_timelimit(conn, time_limit_ms):\r\n try:\r\n cursor = conn.cursor()\r\n cursor.execute(sql, params if params is not None else {})\r\n except:\r\n ...\r\n max_returned_rows = self.ds.max_returned_rows\r\n if max_returned_rows == page_size:\r\n max_returned_rows += 1\r\n if max_returned_rows and truncate:\r\n for i, row in enumerate(cursor):\r\n yield row\r\n if i == max_returned_rows - 1:\r\n break\r\n else:\r\n for row in cursor:\r\n yield row\r\n truncated = False \r\n```\r\n\r\nthis kind of thing works well with a postgres server side cursor, but i'm not sure if it will hold for sqlite. \r\n\r\nyou would still spend about the same amount of time in python and would be contending for the gil, but it would be could be non blocking.\r\n\r\ndepending on the data flow, this could also some benefit for memory. (data stays in more compact sqlite-land until you need it)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1655#issuecomment-1258166572", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1655", "id": 1258166572, "node_id": "IC_kwDOBm6k_c5K_hks", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-09-26T14:57:04Z", "updated_at": "2022-09-26T14:57:04Z", "author_association": "CONTRIBUTOR", "body": "I think that paginating, even in javascript, could be very helpful. Maybe render json or csv into the page and let javascript loading that into the dom?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1163369515, "label": "query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-1258167564", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 1258167564, "node_id": "IC_kwDOBm6k_c5K_h0M", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-09-26T14:57:44Z", "updated_at": "2022-09-26T15:08:36Z", "author_association": "CONTRIBUTOR", "body": "reading the database execute method i have a few questions.\r\n\r\nhttps://github.com/simonw/datasette/blob/cb1e093fd361b758120aefc1a444df02462389a3/datasette/database.py#L229-L242\r\n\r\n---\r\nunless i'm missing something (which is very likely!!), the `max_returned_rows` argument doesn't actually offer any protections against running very expensive queries. \r\n\r\nIt's not like adding a `LIMIT max_rows` argument. it make sense that it isn't because, the query could already have an `LIMIT` argument. Doing something like `select * from (query) limit {max_returned_rows}` **might** be protective but wouldn't always.\r\n\r\nInstead the code executes the full original query, and if still has time it fetches out the first `max_rows + 1` rows. \r\n\r\nthis *does* offer some protection of memory exhaustion, as you won't hydrate a huge result set into python (however, there are [data flow patterns](https://github.com/simonw/datasette/issues/1727#issuecomment-1258129113) that could avoid that too)\r\n\r\ngiven the current architecture, i don't see how creating a new connection would be use?\r\n\r\n---\r\n\r\nIf we just removed the `max_return_rows` limitation, then i think most things would be fine **except** for the QueryViews. Right now rendering, just [5000 rows takes a lot of client-side memory](https://github.com/simonw/datasette/issues/1655) so some form of pagination would be required.\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/526#issuecomment-1258337011", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/526", "id": 1258337011, "node_id": "IC_kwDOBm6k_c5LALLz", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-09-26T16:49:48Z", "updated_at": "2022-09-26T16:49:48Z", "author_association": "CONTRIBUTOR", "body": "i think the smallest change that gets close to what i want is to change the behavior so that `max_returned_rows` is not applied in the `execute` method when we are are asking for a csv of query.\r\n\r\nthere are some infelicities for that approach, but i'll make a PR to make it easier to discuss.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 459882902, "label": "Stream all results for arbitrary SQL and canned queries"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/pull/1820#issuecomment-1258601033", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1820", "id": 1258601033, "node_id": "IC_kwDOBm6k_c5LBLpJ", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-09-26T20:32:47Z", "updated_at": "2022-10-07T03:58:13Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1820?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nBase: **92.50**% // Head: **92.51**% // Increases project coverage by **`+0.01%`** :tada:\n> Coverage data is based on head [(`9bead2a`)](https://codecov.io/gh/simonw/datasette/pull/1820?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`eff1124`)](https://codecov.io/gh/simonw/datasette/commit/eff112498ecc499323c26612d707908831446d25?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> Patch coverage: 100.00% of modified lines in pull request are covered.\n\nAdditional details and impacted files
\n\n\n```diff\n@@ Coverage Diff @@\n## main #1820 +/- ##\n==========================================\n+ Coverage 92.50% 92.51% +0.01% \n==========================================\n Files 35 35 \n Lines 4400 4406 +6 \n==========================================\n+ Hits 4070 4076 +6 \n Misses 330 330 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1820?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.11% <\u00f8> (\u00f8)` | |\n| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `94.80% <100.00%> (+0.05%)` | :arrow_up: |\n| [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1820/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.29% <100.00%> (+0.06%)` | :arrow_up: |\n\nHelp us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n