html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1247149969,https://api.github.com/repos/simonw/sqlite-utils/issues/297,1247149969,IC_kwDOCGYnMM5KVf-R,9599,simonw,2022-09-14T18:28:53Z,2022-09-14T18:29:34Z,OWNER,"As an aside, https://avi.im/blag/2021/fast-sqlite-inserts/ inspired my to try pypy since that article claimed to get a 2.5x speedup using pypy compared to regular Python for a CSV import script.

Setup:
```
brew install pypy3
cd /tmp
pypy3 -m venv venv
source venv/bin/activate
pip install sqlite-utils
```
I grabbed the first 760M of that `https://static.openfoodfacts.org/data/en.openfoodfacts.org.products.csv` file (didn't wait for the whole thing to download).

Then:
```
time sqlite-utils insert pypy.db t en.openfoodfacts.org.products.csv --csv
  [------------------------------------]    0%
  [###################################-]   99% 
11.76s user 2.26s system 93% cpu 14.981 total
```
Compared to regular Python `sqlite-utils` doing the same thing:
```
time sqlite-utils insert py.db t en.openfoodfacts.org.products.csv --csv
  [------------------------------------]    0%
  [###################################-]   99%  
11.36s user 2.06s system 93% cpu 14.341 total
```
So no perceivable performance difference.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1246971764,https://api.github.com/repos/simonw/sqlite-utils/issues/488,1246971764,IC_kwDOCGYnMM5KU0d0,9599,simonw,2022-09-14T15:52:14Z,2022-09-14T15:52:14Z,OWNER,"Frustratingly I think this counts as a backwards-incompatible change.

Could have it be opt-in with a new option / method parameter, and then change it to the default if I release a `sqlite-utils 4`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373224657,`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float,
https://github.com/simonw/datasette/issues/1809#issuecomment-1247317941,https://api.github.com/repos/simonw/datasette/issues/1809,1247317941,IC_kwDOBm6k_c5KWI-1,9599,simonw,2022-09-14T21:24:43Z,2022-09-14T21:24:43Z,OWNER,"It looks like Datasette Lite does NOT invoke that method, which is likely a bug: https://github.com/simonw/datasette-lite/blob/e7ccaf621b3cdf613ebaf544304d387f2af32edf/webworker.js#L103-L110","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373595927,`prepare_jinja2_environment()` hook should take `datasette` argument,
https://github.com/simonw/datasette/issues/1809#issuecomment-1247316097,https://api.github.com/repos/simonw/datasette/issues/1809,1247316097,IC_kwDOBm6k_c5KWIiB,9599,simonw,2022-09-14T21:22:24Z,2022-09-14T21:22:24Z,OWNER,"It looks like this is the only place that calls `invoke_startup()`: https://github.com/simonw/datasette/blob/1d64c9a8dac45b9a3452acf8e76dfadea2b0bc49/datasette/cli.py#L590-L591

`datasette-publish-vercel` is the one deployment mechanism that skips running Uvicorn, and it calls that method separately here: https://github.com/simonw/datasette-publish-vercel/blob/1559d979b4e3b1f2f83c51c3c0c10192ff9a6d0c/datasette_publish_vercel/__init__.py#L42-L52

```python
ds = Datasette(
    [],
    {database_files},
    static_mounts=static_mounts,
    metadata=metadata{extras},
    secret=secret,
    cors=True,
    settings={settings}{crossdb}
)
asyncio.run(ds.invoke_startup())
app = ds.app()
```
So preparing the Jinja environment inside that function would work fine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373595927,`prepare_jinja2_environment()` hook should take `datasette` argument,
https://github.com/simonw/datasette/issues/1809#issuecomment-1247316715,https://api.github.com/repos/simonw/datasette/issues/1809,1247316715,IC_kwDOBm6k_c5KWIrr,9599,simonw,2022-09-14T21:23:10Z,2022-09-14T21:23:10Z,OWNER,It might be good to have Datasette LOUDLY fail if you attempt to use it without calling `await ds.invoke_startup()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373595927,`prepare_jinja2_environment()` hook should take `datasette` argument,
https://github.com/simonw/datasette/issues/1809#issuecomment-1247311275,https://api.github.com/repos/simonw/datasette/issues/1809,1247311275,IC_kwDOBm6k_c5KWHWr,9599,simonw,2022-09-14T21:16:32Z,2022-09-14T21:16:32Z,OWNER,It should also implement the `await_me_maybe` pattern so you can return an `async` function from it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373595927,`prepare_jinja2_environment()` hook should take `datasette` argument,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1246978641,https://api.github.com/repos/simonw/sqlite-utils/issues/297,1246978641,IC_kwDOCGYnMM5KU2JR,9599,simonw,2022-09-14T15:57:41Z,2022-09-14T15:57:41Z,OWNER,"One solution suggested on Discord:
```
wget https://static.openfoodfacts.org/data/en.openfoodfacts.org.products.csv
CREATE=`curl -s -L https://gist.githubusercontent.com/CharlesNepote/80fb813a416ad445fdd6e4738b4c8156/raw/032af70de631ff1c4dd09d55360f242949dcc24f/create.sql`
INDEX=`curl -s -L https://gist.githubusercontent.com/CharlesNepote/80fb813a416ad445fdd6e4738b4c8156/raw/032af70de631ff1c4dd09d55360f242949dcc24f/index.sql`
time sqlite3 products_new.db <<EOS
/* Optimisations. See: https://avi.im/blag/2021/fast-sqlite-inserts/ */;
PRAGMA page_size = 32768;
$CREATE
.mode ascii
.separator ""\t"" ""\n""
.import --skip 1 en.openfoodfacts.org.products.csv all
$INDEX
EOS
# Converting empty to NULL for columns which are either FLOAT or INTEGER
time sqlite3 products.db "".schema all"" | sed -nr 's/.*\[(.*)\] (INTEGER|FLOAT).*/\1/gp' | xargs -I % sqlite3 products.db -cmd ""PRAGMA journal_mode=OFF;"" ""UPDATE [all] SET [%] = NULL WHERE [%] = '';""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1246977989,https://api.github.com/repos/simonw/sqlite-utils/issues/297,1246977989,IC_kwDOCGYnMM5KU1_F,9599,simonw,2022-09-14T15:57:09Z,2022-09-14T15:57:09Z,OWNER,"Should consider how this could best handle creating columns that are integer and float as opposed to just text.

https://discord.com/channels/823971286308356157/823971286941302908/1019630014544748584 is a relevant discussion on Discord. Even if you create the schema in advance with the correct column types, this import mechanism can put empty strings in blank float/integer columns when ideally you would want to have nulls.

Related feature idea for `sqlite-utils transform`:
- #488

Not sure how best to handle this for `sqlite3 .import` imports.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/datasette/issues/1809#issuecomment-1247314352,https://api.github.com/repos/simonw/datasette/issues/1809,1247314352,IC_kwDOBm6k_c5KWIGw,9599,simonw,2022-09-14T21:20:12Z,2022-09-14T21:20:12Z,OWNER,"The reason to support `await_me_maybe` is in case a hook wants to execute a SQL query as part of configuring the Jinja template loader.

That's exactly what `datasette-edit-templates` needs to do, though it's currently achieving that in a `startup()` hook instead: https://github.com/simonw/datasette-edit-templates/blob/087f6a6cabc20020f2b0524f11aa3a7836320848/datasette_edit_templates/__init__.py#L32-L48

```python
@hookimpl
def startup(datasette):
    datasette._edit_templates = {}

    async def inner():
        db = get_database(datasette)
        # Does the table exist?
        if not await db.table_exists(TABLE):
            for sql in CREATE_TABLE:
                await db.execute_write(sql, block=True)
        else:
            # Load all templates from that table
            rows = await db.execute(""select template, body FROM {}"".format(TABLE))
            for name, content in rows:
                datasette._edit_templates[name] = content

    return inner
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373595927,`prepare_jinja2_environment()` hook should take `datasette` argument,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1247161510,https://api.github.com/repos/simonw/sqlite-utils/issues/297,1247161510,IC_kwDOCGYnMM5KViym,9599,simonw,2022-09-14T18:39:50Z,2022-09-14T18:39:50Z,OWNER,Wrote that up as a TIL: https://til.simonwillison.net/python/pypy-macos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/datasette/issues/1809#issuecomment-1247313134,https://api.github.com/repos/simonw/datasette/issues/1809,1247313134,IC_kwDOBm6k_c5KWHzu,9599,simonw,2022-09-14T21:18:46Z,2022-09-14T21:18:46Z,OWNER,"`await_me_maybe` might be tricky though, because the only place that hook is executed is here:

https://github.com/simonw/datasette/blob/8430c3bc7dd22b173c1a8c6cd7180e3b31240cd1/datasette/app.py#L348

Which is inside the `Datasette.__init__` method.

To implement an `await` on that would need to move it somewhere else - probably here: https://github.com/simonw/datasette/blob/8430c3bc7dd22b173c1a8c6cd7180e3b31240cd1/datasette/app.py#L391-L393

But I'm not 100% confident that is always executed at the right time to ensure the Jinja environment is properly configured? I think it is, but would need to make sure.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373595927,`prepare_jinja2_environment()` hook should take `datasette` argument,