html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982123183,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982123183,IC_kwDOCGYnMM46igKv,22429695,codecov[bot],2021-11-29T23:20:35Z,2021-12-11T01:02:19Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#347](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (71b6c38) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/213a0ff177f23a35f3b235386366ff132eb879f1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (213a0ff) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
> :exclamation: Current head 71b6c38 differs from pull request most recent head 1a7ef2f. Consider uploading reports for the commit 1a7ef2f to get more accurate results
[](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
```diff
@@ Coverage Diff @@
## main #347 +/- ##
=======================================
Coverage 96.51% 96.52%
=======================================
Files 5 5
Lines 2270 2271 +1
=======================================
+ Hits 2191 2192 +1
Misses 79 79
```
| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/347/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.73% <100.00%> (ø)` | |
| [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/347/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `93.68% <100.00%> (+0.03%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [213a0ff...1a7ef2f](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37,
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991386841,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991386841,IC_kwDOCGYnMM47F1zZ,9599,simonw,2021-12-11T00:14:11Z,2021-12-11T00:15:15Z,OWNER,"Relevant code: https://github.com/simonw/sqlite-utils/blob/7a43af232e4bc00bd227307665163614e225948b/sqlite_utils/cli.py#L2128-L2135
One way to implement this would be to look to see if the code starts with `def ...` - but that's not going to work for proper module that start with a docstring or imports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition,
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991387044,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991387044,IC_kwDOCGYnMM47F12k,9599,simonw,2021-12-11T00:14:45Z,2021-12-11T00:14:45Z,OWNER,"Maybe attempt to compile their code, and if it fails try again after adding `def fn(value):` to the start?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition,
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991393684,IC_kwDOCGYnMM47F3eU,9599,simonw,2021-12-11T00:42:19Z,2021-12-11T00:49:49Z,OWNER,"Ideally I'd like to show the perfect syntax error messages to the user - but I don't know if it's possible to do this cleanly because the error might occur with their originally entered code OR it might occur after I add `def fn(value)` to it.
I'm going to punt on that for the moment and tolerate slightly confusing syntax errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition,
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991395494,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991395494,IC_kwDOCGYnMM47F36m,9599,simonw,2021-12-11T00:50:22Z,2021-12-11T00:51:15Z,OWNER,"Here's an example of the new (slightly confusing) error message:
```bash
sqlite-utils convert fixtures.db roadside_attractions name '
def foo(value)
bar
baz
'
Error: Syntax error in code:
def foo(value)
invalid syntax
```
Another:
```
sqlite-utils convert fixtures.db roadside_attractions name '$'
Error: Syntax error in code:
return $
invalid syntax
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition,
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991395919,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991395919,IC_kwDOCGYnMM47F4BP,9599,simonw,2021-12-11T00:52:31Z,2021-12-11T00:52:31Z,OWNER,"It turns out `rebuild` does indeed work against content tables, so I can put that in the test instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts,
https://github.com/simonw/sqlite-utils/pull/347#issuecomment-991397907,https://api.github.com/repos/simonw/sqlite-utils/issues/347,991397907,IC_kwDOCGYnMM47F4gT,9599,simonw,2021-12-11T01:01:40Z,2021-12-11T01:01:40Z,OWNER,The change I made to that test in #354 might help with this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37,
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991398367,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991398367,IC_kwDOCGYnMM47F4nf,9599,simonw,2021-12-11T01:03:14Z,2021-12-11T01:03:14Z,OWNER,The new test: https://github.com/simonw/sqlite-utils/blob/ee13f98c2c7ca3b819bd0fc55da3108cb6a6434a/tests/test_fts.py#L270-L277,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts,
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991399604,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991399604,IC_kwDOCGYnMM47F460,9599,simonw,2021-12-11T01:08:46Z,2021-12-11T01:08:46Z,OWNER,That passed!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts,
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991399782,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991399782,IC_kwDOCGYnMM47F49m,9599,simonw,2021-12-11T01:09:37Z,2021-12-11T01:09:37Z,OWNER,"OK, this is implemented. Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""",
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991400016,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991400016,IC_kwDOCGYnMM47F5BQ,9599,simonw,2021-12-11T01:10:52Z,2021-12-11T01:11:02Z,OWNER,"This won't be in a release for a little while, but you can install it to try it out using:
pip install https://github.com/simonw/sqlite-utils/archive/ee13f98c2c.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""",
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991405755,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991405755,IC_kwDOCGYnMM47F6a7,536941,fgregg,2021-12-11T01:38:29Z,2021-12-11T01:38:29Z,CONTRIBUTOR,"wow! that's awesome! thanks so much, @simonw!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""",
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-991517209,https://api.github.com/repos/simonw/sqlite-utils/issues/356,991517209,IC_kwDOCGYnMM47GVoZ,9599,simonw,2021-12-11T07:46:41Z,2021-12-11T07:46:41Z,OWNER,"By default this will accept single lines, but maybe there could be a `--all` option which instead grabs all of stdin into a single string against which the conversion function runs - like `git-history file`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option,
https://github.com/simonw/datasette/issues/1549#issuecomment-991752486,https://api.github.com/repos/simonw/datasette/issues/1549,991752486,IC_kwDOBm6k_c47HPEm,9599,simonw,2021-12-11T19:09:15Z,2021-12-11T19:09:15Z,OWNER,"That's what this option does:
The usability of this is pretty terrible though (including ""stream all rows"" - how are people meant to understand what that does?) so it can definitely do with some rethinking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability,
https://github.com/simonw/datasette/issues/1549#issuecomment-991754237,https://api.github.com/repos/simonw/datasette/issues/1549,991754237,IC_kwDOBm6k_c47HPf9,536941,fgregg,2021-12-11T19:14:39Z,2021-12-11T19:14:39Z,CONTRIBUTOR,"that option is not available on [custom queries](https://labordata.bunkum.us/odpr-962a140?sql=with+local_union_filings+as+%28%0D%0A++select+*+from+lm_data+%0D%0A++where%0D%0A++++yr_covered+%3E+cast%28strftime%28%27%25Y%27%2C+%27now%27%2C+%27-5+years%27%29+as+int%29%0D%0A++++and+desig_name+%3D+%27LU%27%0D%0A++order+by+yr_covered+desc%0D%0A%29%2C%0D%0Amost_recent_filing+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from+local_union_filings%0D%0A++group+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++most_recent_filing%0D%0Awhere%0D%0A++next_election+%3E%3D+strftime%28%27%25Y-%25m%27%2C+%27now%27%29%0D%0A++and+next_election+%3C+strftime%28%27%25Y-%25m%27%2C+%27now%27%2C+%27%2B1+year%27%29%0D%0Aorder+by%0D%0A++members+desc%3B).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability,
https://github.com/simonw/datasette/issues/1549#issuecomment-991754794,https://api.github.com/repos/simonw/datasette/issues/1549,991754794,IC_kwDOBm6k_c47HPoq,9599,simonw,2021-12-11T19:16:33Z,2021-12-11T19:16:33Z,OWNER,Good call! I'm doing a refactor #1518 right now which will hopefully bring the functionality of those two much closer - I'll make a note to consider this there too.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability,
https://github.com/simonw/datasette/issues/617#issuecomment-991755013,https://api.github.com/repos/simonw/datasette/issues/617,991755013,IC_kwDOBm6k_c47HPsF,9599,simonw,2021-12-11T19:17:11Z,2021-12-11T19:17:11Z,OWNER,This work is now happening in #1518 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519613116,Refactor TableView.data() method,
https://github.com/simonw/datasette/issues/1549#issuecomment-991755245,https://api.github.com/repos/simonw/datasette/issues/1549,991755245,IC_kwDOBm6k_c47HPvt,9599,simonw,2021-12-11T19:17:54Z,2021-12-11T19:17:54Z,OWNER,"Also relevant:
- #1062 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability,
https://github.com/simonw/datasette/issues/1550#issuecomment-991761635,https://api.github.com/repos/simonw/datasette/issues/1550,991761635,IC_kwDOBm6k_c47HRTj,9599,simonw,2021-12-11T19:39:01Z,2021-12-11T19:39:01Z,OWNER,"I wonder if this could work for public instances too with some kind of queuing mechanism?
I really need to use benchmarking to figure out what the right number of maximum SQLite connections is. I'm just guessing at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077628073,Research option for returning all rows from arbitrary query,
https://github.com/simonw/datasette/issues/1550#issuecomment-991805516,https://api.github.com/repos/simonw/datasette/issues/1550,991805516,IC_kwDOBm6k_c47HcBM,9599,simonw,2021-12-11T23:43:24Z,2021-12-11T23:43:24Z,OWNER,"I built a tiny Starlette app to experiment with this a bit:
```python
import asyncio
import janus
from starlette.applications import Starlette
from starlette.responses import JSONResponse, HTMLResponse, StreamingResponse
from starlette.routing import Route
import sqlite3
from concurrent import futures
executor = futures.ThreadPoolExecutor(max_workers=10)
async def homepage(request):
return HTMLResponse(
""""""
SQL CSV Server
SQL CSV Server
""""""
)
def run_query_in_thread(sql, sync_q):
db = sqlite3.connect(""../datasette/covid.db"")
cursor = db.cursor()
cursor.arraysize = 100 # Default is 1 apparently?
cursor.execute(sql)
columns = [d[0] for d in cursor.description]
sync_q.put([columns])
# Now start putting batches of rows
while True:
rows = cursor.fetchmany()
if rows:
sync_q.put(rows)
else:
break
# Let queue know we are finished\
sync_q.put(None)
async def csv_query(request):
sql = request.query_params[""sql""]
queue = janus.Queue()
loop = asyncio.get_running_loop()
async def csv_generator():
loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)
while True:
rows = await queue.async_q.get()
if rows is not None:
for row in rows:
yield "","".join(map(str, row)) + ""\n ""
queue.async_q.task_done()
else:
# Cleanup
queue.close()
await queue.wait_closed()
break
return StreamingResponse(csv_generator(), media_type='text/plain')
app = Starlette(
debug=True,
routes=[
Route(""/"", homepage),
Route(""/csv"", csv_query),
],
)
```
But.. if I run this in a terminal window:
```
/tmp % wget 'http://127.0.0.1:8000/csv?sql=select+*+from+ny_times_us_counties'
```
it takes about 20 seconds to run and returns a 50MB file - but while it is running no other requests can be served by that server - not even the homepage! So something is blocking the event loop.
Maybe I should be using `fut = loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)` and then awaiting `fut` somewhere, like in the Janus documentation? Don't think that's needed though. Needs more work to figure out why this is blocking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077628073,Research option for returning all rows from arbitrary query,