{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1247149969", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1247149969, "node_id": "IC_kwDOCGYnMM5KVf-R", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T18:28:53Z", "updated_at": "2022-09-14T18:29:34Z", "author_association": "OWNER", "body": "As an aside, https://avi.im/blag/2021/fast-sqlite-inserts/ inspired my to try pypy since that article claimed to get a 2.5x speedup using pypy compared to regular Python for a CSV import script.\r\n\r\nSetup:\r\n```\r\nbrew install pypy3\r\ncd /tmp\r\npypy3 -m venv venv\r\nsource venv/bin/activate\r\npip install sqlite-utils\r\n```\r\nI grabbed the first 760M of that `https://static.openfoodfacts.org/data/en.openfoodfacts.org.products.csv` file (didn't wait for the whole thing to download).\r\n\r\nThen:\r\n```\r\ntime sqlite-utils insert pypy.db t en.openfoodfacts.org.products.csv --csv\r\n  [------------------------------------]    0%\r\n  [###################################-]   99% \r\n11.76s user 2.26s system 93% cpu 14.981 total\r\n```\r\nCompared to regular Python `sqlite-utils` doing the same thing:\r\n```\r\ntime sqlite-utils insert py.db t en.openfoodfacts.org.products.csv --csv\r\n  [------------------------------------]    0%\r\n  [###################################-]   99%  \r\n11.36s user 2.06s system 93% cpu 14.341 total\r\n```\r\nSo no perceivable performance difference.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1246971764", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/488", "id": 1246971764, "node_id": "IC_kwDOCGYnMM5KU0d0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T15:52:14Z", "updated_at": "2022-09-14T15:52:14Z", "author_association": "OWNER", "body": "Frustratingly I think this counts as a backwards-incompatible change.\r\n\r\nCould have it be opt-in with a new option / method parameter, and then change it to the default if I release a `sqlite-utils 4`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373224657, "label": "`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1247317941", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1247317941, "node_id": "IC_kwDOBm6k_c5KWI-1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T21:24:43Z", "updated_at": "2022-09-14T21:24:43Z", "author_association": "OWNER", "body": "It looks like Datasette Lite does NOT invoke that method, which is likely a bug: https://github.com/simonw/datasette-lite/blob/e7ccaf621b3cdf613ebaf544304d387f2af32edf/webworker.js#L103-L110", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1247316097", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1247316097, "node_id": "IC_kwDOBm6k_c5KWIiB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T21:22:24Z", "updated_at": "2022-09-14T21:22:24Z", "author_association": "OWNER", "body": "It looks like this is the only place that calls `invoke_startup()`: https://github.com/simonw/datasette/blob/1d64c9a8dac45b9a3452acf8e76dfadea2b0bc49/datasette/cli.py#L590-L591\r\n\r\n`datasette-publish-vercel` is the one deployment mechanism that skips running Uvicorn, and it calls that method separately here: https://github.com/simonw/datasette-publish-vercel/blob/1559d979b4e3b1f2f83c51c3c0c10192ff9a6d0c/datasette_publish_vercel/__init__.py#L42-L52\r\n\r\n```python\r\nds = Datasette(\r\n    [],\r\n    {database_files},\r\n    static_mounts=static_mounts,\r\n    metadata=metadata{extras},\r\n    secret=secret,\r\n    cors=True,\r\n    settings={settings}{crossdb}\r\n)\r\nasyncio.run(ds.invoke_startup())\r\napp = ds.app()\r\n```\r\nSo preparing the Jinja environment inside that function would work fine.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1247316715", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1247316715, "node_id": "IC_kwDOBm6k_c5KWIrr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T21:23:10Z", "updated_at": "2022-09-14T21:23:10Z", "author_association": "OWNER", "body": "It might be good to have Datasette LOUDLY fail if you attempt to use it without calling `await ds.invoke_startup()`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1247311275", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1247311275, "node_id": "IC_kwDOBm6k_c5KWHWr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T21:16:32Z", "updated_at": "2022-09-14T21:16:32Z", "author_association": "OWNER", "body": "It should also implement the `await_me_maybe` pattern so you can return an `async` function from it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1246978641", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1246978641, "node_id": "IC_kwDOCGYnMM5KU2JR", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T15:57:41Z", "updated_at": "2022-09-14T15:57:41Z", "author_association": "OWNER", "body": "One solution suggested on Discord:\r\n```\r\nwget https://static.openfoodfacts.org/data/en.openfoodfacts.org.products.csv\r\nCREATE=`curl -s -L https://gist.githubusercontent.com/CharlesNepote/80fb813a416ad445fdd6e4738b4c8156/raw/032af70de631ff1c4dd09d55360f242949dcc24f/create.sql`\r\nINDEX=`curl -s -L https://gist.githubusercontent.com/CharlesNepote/80fb813a416ad445fdd6e4738b4c8156/raw/032af70de631ff1c4dd09d55360f242949dcc24f/index.sql`\r\ntime sqlite3 products_new.db <<EOS\r\n/* Optimisations. See: https://avi.im/blag/2021/fast-sqlite-inserts/ */;\r\nPRAGMA page_size = 32768;\r\n$CREATE\r\n.mode ascii\r\n.separator \"\\t\" \"\\n\"\r\n.import --skip 1 en.openfoodfacts.org.products.csv all\r\n$INDEX\r\nEOS\r\n# Converting empty to NULL for columns which are either FLOAT or INTEGER\r\ntime sqlite3 products.db \".schema all\" | sed -nr 's/.*\\[(.*)\\] (INTEGER|FLOAT).*/\\1/gp' | xargs -I % sqlite3 products.db -cmd \"PRAGMA journal_mode=OFF;\" \"UPDATE [all] SET [%] = NULL WHERE [%] = '';\"\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1246977989", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1246977989, "node_id": "IC_kwDOCGYnMM5KU1_F", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T15:57:09Z", "updated_at": "2022-09-14T15:57:09Z", "author_association": "OWNER", "body": "Should consider how this could best handle creating columns that are integer and float as opposed to just text.\r\n\r\nhttps://discord.com/channels/823971286308356157/823971286941302908/1019630014544748584 is a relevant discussion on Discord. Even if you create the schema in advance with the correct column types, this import mechanism can put empty strings in blank float/integer columns when ideally you would want to have nulls.\r\n\r\nRelated feature idea for `sqlite-utils transform`:\r\n- #488\r\n\r\nNot sure how best to handle this for `sqlite3 .import` imports.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1247314352", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1247314352, "node_id": "IC_kwDOBm6k_c5KWIGw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T21:20:12Z", "updated_at": "2022-09-14T21:20:12Z", "author_association": "OWNER", "body": "The reason to support `await_me_maybe` is in case a hook wants to execute a SQL query as part of configuring the Jinja template loader.\r\n\r\nThat's exactly what `datasette-edit-templates` needs to do, though it's currently achieving that in a `startup()` hook instead: https://github.com/simonw/datasette-edit-templates/blob/087f6a6cabc20020f2b0524f11aa3a7836320848/datasette_edit_templates/__init__.py#L32-L48\r\n\r\n```python\r\n@hookimpl\r\ndef startup(datasette):\r\n    datasette._edit_templates = {}\r\n\r\n    async def inner():\r\n        db = get_database(datasette)\r\n        # Does the table exist?\r\n        if not await db.table_exists(TABLE):\r\n            for sql in CREATE_TABLE:\r\n                await db.execute_write(sql, block=True)\r\n        else:\r\n            # Load all templates from that table\r\n            rows = await db.execute(\"select template, body FROM {}\".format(TABLE))\r\n            for name, content in rows:\r\n                datasette._edit_templates[name] = content\r\n\r\n    return inner\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1247161510", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297", "id": 1247161510, "node_id": "IC_kwDOCGYnMM5KViym", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T18:39:50Z", "updated_at": "2022-09-14T18:39:50Z", "author_association": "OWNER", "body": "Wrote that up as a TIL: https://til.simonwillison.net/python/pypy-macos", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 944846776, "label": "Option for importing CSV data using the SQLite .import mechanism"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1809#issuecomment-1247313134", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1809", "id": 1247313134, "node_id": "IC_kwDOBm6k_c5KWHzu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-14T21:18:46Z", "updated_at": "2022-09-14T21:18:46Z", "author_association": "OWNER", "body": "`await_me_maybe` might be tricky though, because the only place that hook is executed is here:\r\n\r\nhttps://github.com/simonw/datasette/blob/8430c3bc7dd22b173c1a8c6cd7180e3b31240cd1/datasette/app.py#L348\r\n\r\nWhich is inside the `Datasette.__init__` method.\r\n\r\nTo implement an `await` on that would need to move it somewhere else - probably here: https://github.com/simonw/datasette/blob/8430c3bc7dd22b173c1a8c6cd7180e3b31240cd1/datasette/app.py#L391-L393\r\n\r\nBut I'm not 100% confident that is always executed at the right time to ensure the Jinja environment is properly configured? I think it is, but would need to make sure.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1373595927, "label": "`prepare_jinja2_environment()` hook should take `datasette` argument"}, "performed_via_github_app": null}