{"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982123183", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982123183, "node_id": "IC_kwDOCGYnMM46igKv", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-11-29T23:20:35Z", "updated_at": "2021-12-11T01:02:19Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#347](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (71b6c38) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/213a0ff177f23a35f3b235386366ff132eb879f1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (213a0ff) will **increase** coverage by `0.00%`.\n> The diff coverage is `100.00%`.\n\n> :exclamation: Current head 71b6c38 differs from pull request most recent head 1a7ef2f. Consider uploading reports for the commit 1a7ef2f to get more accurate results\n[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/347/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #347 +/- ##\n=======================================\n Coverage 96.51% 96.52% \n=======================================\n Files 5 5 \n Lines 2270 2271 +1 \n=======================================\n+ Hits 2191 2192 +1 \n Misses 79 79 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/347/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.73% <100.00%> (\u00f8)` | |\n| [sqlite\\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/347/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `93.68% <100.00%> (+0.03%)` | :arrow_up: |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [213a0ff...1a7ef2f](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991386841", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/355", "id": 991386841, "node_id": "IC_kwDOCGYnMM47F1zZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T00:14:11Z", "updated_at": "2021-12-11T00:15:15Z", "author_association": "OWNER", "body": "Relevant code: https://github.com/simonw/sqlite-utils/blob/7a43af232e4bc00bd227307665163614e225948b/sqlite_utils/cli.py#L2128-L2135\r\n\r\nOne way to implement this would be to look to see if the code starts with `def ...` - but that's not going to work for proper module that start with a docstring or imports.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077322009, "label": "Allow users to pass a full convert() function definition"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991387044", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/355", "id": 991387044, "node_id": "IC_kwDOCGYnMM47F12k", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T00:14:45Z", "updated_at": "2021-12-11T00:14:45Z", "author_association": "OWNER", "body": "Maybe attempt to compile their code, and if it fails try again after adding `def fn(value):` to the start?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077322009, "label": "Allow users to pass a full convert() function definition"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/355", "id": 991393684, "node_id": "IC_kwDOCGYnMM47F3eU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T00:42:19Z", "updated_at": "2021-12-11T00:49:49Z", "author_association": "OWNER", "body": "Ideally I'd like to show the perfect syntax error messages to the user - but I don't know if it's possible to do this cleanly because the error might occur with their originally entered code OR it might occur after I add `def fn(value)` to it.\r\n\r\nI'm going to punt on that for the moment and tolerate slightly confusing syntax errors.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077322009, "label": "Allow users to pass a full convert() function definition"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991395494", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/355", "id": 991395494, "node_id": "IC_kwDOCGYnMM47F36m", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T00:50:22Z", "updated_at": "2021-12-11T00:51:15Z", "author_association": "OWNER", "body": "Here's an example of the new (slightly confusing) error message:\r\n```bash\r\nsqlite-utils convert fixtures.db roadside_attractions name '\r\ndef foo(value) \r\n bar\r\n baz\r\n'\r\nError: Syntax error in code:\r\n\r\n def foo(value)\r\n\r\ninvalid syntax\r\n```\r\nAnother:\r\n```\r\nsqlite-utils convert fixtures.db roadside_attractions name '$'\r\nError: Syntax error in code:\r\n\r\n return $\r\n\r\ninvalid syntax\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077322009, "label": "Allow users to pass a full convert() function definition"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991395919", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/354", "id": 991395919, "node_id": "IC_kwDOCGYnMM47F4BP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T00:52:31Z", "updated_at": "2021-12-11T00:52:31Z", "author_association": "OWNER", "body": "It turns out `rebuild` does indeed work against content tables, so I can put that in the test instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077243232, "label": "Test failure in test_rebuild_fts"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-991397907", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 991397907, "node_id": "IC_kwDOCGYnMM47F4gT", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T01:01:40Z", "updated_at": "2021-12-11T01:01:40Z", "author_association": "OWNER", "body": "The change I made to that test in #354 might help with this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991398367", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/354", "id": 991398367, "node_id": "IC_kwDOCGYnMM47F4nf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T01:03:14Z", "updated_at": "2021-12-11T01:03:14Z", "author_association": "OWNER", "body": "The new test: https://github.com/simonw/sqlite-utils/blob/ee13f98c2c7ca3b819bd0fc55da3108cb6a6434a/tests/test_fts.py#L270-L277", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077243232, "label": "Test failure in test_rebuild_fts"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991399604", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/354", "id": 991399604, "node_id": "IC_kwDOCGYnMM47F460", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T01:08:46Z", "updated_at": "2021-12-11T01:08:46Z", "author_association": "OWNER", "body": "That passed!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077243232, "label": "Test failure in test_rebuild_fts"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991399782", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/353", "id": 991399782, "node_id": "IC_kwDOCGYnMM47F49m", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T01:09:37Z", "updated_at": "2021-12-11T01:09:37Z", "author_association": "OWNER", "body": "OK, this is implemented. Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077102934, "label": "Allow passing a file of code to \"sqlite-utils convert\""}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991400016", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/353", "id": 991400016, "node_id": "IC_kwDOCGYnMM47F5BQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T01:10:52Z", "updated_at": "2021-12-11T01:11:02Z", "author_association": "OWNER", "body": "This won't be in a release for a little while, but you can install it to try it out using:\r\n\r\n pip install https://github.com/simonw/sqlite-utils/archive/ee13f98c2c.zip", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077102934, "label": "Allow passing a file of code to \"sqlite-utils convert\""}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991405755", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/353", "id": 991405755, "node_id": "IC_kwDOCGYnMM47F6a7", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-12-11T01:38:29Z", "updated_at": "2021-12-11T01:38:29Z", "author_association": "CONTRIBUTOR", "body": "wow! that's awesome! thanks so much, @simonw!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077102934, "label": "Allow passing a file of code to \"sqlite-utils convert\""}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-991517209", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 991517209, "node_id": "IC_kwDOCGYnMM47GVoZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T07:46:41Z", "updated_at": "2021-12-11T07:46:41Z", "author_association": "OWNER", "body": "By default this will accept single lines, but maybe there could be a `--all` option which instead grabs all of stdin into a single string against which the conversion function runs - like `git-history file`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1549#issuecomment-991752486", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1549", "id": 991752486, "node_id": "IC_kwDOBm6k_c47HPEm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T19:09:15Z", "updated_at": "2021-12-11T19:09:15Z", "author_association": "OWNER", "body": "That's what this option does:\r\n\r\n![EAB1B9E8-38E9-4C6D-8854-BD1935F163D9](https://user-images.githubusercontent.com/9599/145688531-668bafa1-e287-4bbd-84d6-157241fb1f68.jpeg)\r\n\r\nThe usability of this is pretty terrible though (including \"stream all rows\" - how are people meant to understand what that does?) so it can definitely do with some rethinking.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077620955, "label": "Redesign CSV export to improve usability"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1549#issuecomment-991754237", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1549", "id": 991754237, "node_id": "IC_kwDOBm6k_c47HPf9", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2021-12-11T19:14:39Z", "updated_at": "2021-12-11T19:14:39Z", "author_association": "CONTRIBUTOR", "body": "that option is not available on [custom queries](https://labordata.bunkum.us/odpr-962a140?sql=with+local_union_filings+as+%28%0D%0A++select+*+from+lm_data+%0D%0A++where%0D%0A++++yr_covered+%3E+cast%28strftime%28%27%25Y%27%2C+%27now%27%2C+%27-5+years%27%29+as+int%29%0D%0A++++and+desig_name+%3D+%27LU%27%0D%0A++order+by+yr_covered+desc%0D%0A%29%2C%0D%0Amost_recent_filing+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from+local_union_filings%0D%0A++group+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++most_recent_filing%0D%0Awhere%0D%0A++next_election+%3E%3D+strftime%28%27%25Y-%25m%27%2C+%27now%27%29%0D%0A++and+next_election+%3C+strftime%28%27%25Y-%25m%27%2C+%27now%27%2C+%27%2B1+year%27%29%0D%0Aorder+by%0D%0A++members+desc%3B).\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077620955, "label": "Redesign CSV export to improve usability"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1549#issuecomment-991754794", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1549", "id": 991754794, "node_id": "IC_kwDOBm6k_c47HPoq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T19:16:33Z", "updated_at": "2021-12-11T19:16:33Z", "author_association": "OWNER", "body": "Good call! I'm doing a refactor #1518 right now which will hopefully bring the functionality of those two much closer - I'll make a note to consider this there too.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077620955, "label": "Redesign CSV export to improve usability"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/617#issuecomment-991755013", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/617", "id": 991755013, "node_id": "IC_kwDOBm6k_c47HPsF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T19:17:11Z", "updated_at": "2021-12-11T19:17:11Z", "author_association": "OWNER", "body": "This work is now happening in #1518 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 519613116, "label": "Refactor TableView.data() method"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1549#issuecomment-991755245", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1549", "id": 991755245, "node_id": "IC_kwDOBm6k_c47HPvt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T19:17:54Z", "updated_at": "2021-12-11T19:17:54Z", "author_association": "OWNER", "body": "Also relevant:\r\n- #1062 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077620955, "label": "Redesign CSV export to improve usability"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1550#issuecomment-991761635", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1550", "id": 991761635, "node_id": "IC_kwDOBm6k_c47HRTj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T19:39:01Z", "updated_at": "2021-12-11T19:39:01Z", "author_association": "OWNER", "body": "I wonder if this could work for public instances too with some kind of queuing mechanism?\r\n\r\nI really need to use benchmarking to figure out what the right number of maximum SQLite connections is. I'm just guessing at the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077628073, "label": "Research option for returning all rows from arbitrary query"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1550#issuecomment-991805516", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1550", "id": 991805516, "node_id": "IC_kwDOBm6k_c47HcBM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T23:43:24Z", "updated_at": "2021-12-11T23:43:24Z", "author_association": "OWNER", "body": "I built a tiny Starlette app to experiment with this a bit:\r\n```python\r\nimport asyncio\r\nimport janus\r\nfrom starlette.applications import Starlette\r\nfrom starlette.responses import JSONResponse, HTMLResponse, StreamingResponse\r\nfrom starlette.routing import Route\r\nimport sqlite3\r\nfrom concurrent import futures\r\n\r\nexecutor = futures.ThreadPoolExecutor(max_workers=10)\r\n\r\n\r\nasync def homepage(request):\r\n return HTMLResponse(\r\n \"\"\"\r\n \r\n SQL CSV Server\r\n \r\n \r\n

SQL CSV Server

\r\n
\r\n \r\n \r\n
\r\n \r\n \"\"\"\r\n )\r\n\r\n\r\ndef run_query_in_thread(sql, sync_q):\r\n db = sqlite3.connect(\"../datasette/covid.db\")\r\n cursor = db.cursor()\r\n cursor.arraysize = 100 # Default is 1 apparently?\r\n cursor.execute(sql)\r\n columns = [d[0] for d in cursor.description]\r\n sync_q.put([columns])\r\n # Now start putting batches of rows\r\n while True:\r\n rows = cursor.fetchmany()\r\n if rows:\r\n sync_q.put(rows)\r\n else:\r\n break\r\n # Let queue know we are finished\\\r\n sync_q.put(None)\r\n\r\n\r\nasync def csv_query(request):\r\n sql = request.query_params[\"sql\"]\r\n\r\n queue = janus.Queue()\r\n loop = asyncio.get_running_loop()\r\n\r\n async def csv_generator():\r\n loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)\r\n while True:\r\n rows = await queue.async_q.get()\r\n if rows is not None:\r\n for row in rows:\r\n yield \",\".join(map(str, row)) + \"\\n \"\r\n queue.async_q.task_done()\r\n else:\r\n # Cleanup\r\n queue.close()\r\n await queue.wait_closed()\r\n break\r\n\r\n return StreamingResponse(csv_generator(), media_type='text/plain')\r\n\r\n\r\napp = Starlette(\r\n debug=True,\r\n routes=[\r\n Route(\"/\", homepage),\r\n Route(\"/csv\", csv_query),\r\n ],\r\n)\r\n```\r\nBut.. if I run this in a terminal window:\r\n```\r\n/tmp % wget 'http://127.0.0.1:8000/csv?sql=select+*+from+ny_times_us_counties'\r\n```\r\nit takes about 20 seconds to run and returns a 50MB file - but while it is running no other requests can be served by that server - not even the homepage! So something is blocking the event loop.\r\n\r\nMaybe I should be using `fut = loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)` and then awaiting `fut` somewhere, like in the Janus documentation? Don't think that's needed though. Needs more work to figure out why this is blocking.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077628073, "label": "Research option for returning all rows from arbitrary query"}, "performed_via_github_app": null}