{"html_url": "https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787118691", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242", "id": 787118691, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzExODY5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-27T18:53:23Z", "updated_at": "2021-02-27T18:53:23Z", "author_association": "OWNER", "body": "Datasette has its own implementation of a write queue for exactly this purpose - and there's no reason at all that should stay in Datasette rather than being extracted out and moved over here to `sqlite-utils`.\r\n\r\nOne small concern I have is around the API design. I'd want to keep supporting the existing synchronous API while also providing a similar API with await-based methods.\r\n\r\nWhat are some good examples of libraries that do this? I like how https://www.python-httpx.org/ handles it, maybe that's a good example to imitate?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 817989436, "label": "Async support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787120136", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242", "id": 787120136, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzEyMDEzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-27T19:04:47Z", "updated_at": "2021-02-27T19:04:47Z", "author_association": "OWNER", "body": "Another option here would be to add https://github.com/omnilib/aiosqlite/blob/main/aiosqlite/core.py as a dependency - it's four years old now and actively marinated, and the code is pretty small so it looks like a solid, stable, reliable dependency.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 817989436, "label": "Async support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787121933", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242", "id": 787121933, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzEyMTkzMw==", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2021-02-27T19:18:57Z", "updated_at": "2021-02-27T19:18:57Z", "author_association": "CONTRIBUTOR", "body": "I think HTTPX gets it exactly right, with a clear separation between sync and async clients, each with a basically identical API. (I'm about to switch [feed-to-sqlite](https://github.com/eyeseast/feed-to-sqlite) over to it, from Requests, to eventually make way for async support.)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 817989436, "label": "Async support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787142066", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242", "id": 787142066, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzE0MjA2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-27T21:17:10Z", "updated_at": "2021-02-27T21:17:10Z", "author_association": "OWNER", "body": "I have a hunch this is actually going to be quite difficult, due to the internal complexity of some of the `sqlite-utils` API methods.\r\n\r\nConsider `db[table].extract(...)` for example. It does a whole bunch of extra queries inside the method - each of those would need to be turned into an `await` call for the async version. Here's the method body today:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1060-L1152\r\n\r\nWriting this method twice - looking similar but with `await ...` tucked in before every internal method it calls that needs to execute SQL - is going to be pretty messy.\r\n\r\nOne thing that would help a LOT is figuring out how to share the majority of the test code. If the exact same tests could run against both the sync and async versions with a bit of test trickery, maintaining parallel implementations would at least be a bit more feasible.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 817989436, "label": "Async support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787144523", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242", "id": 787144523, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzE0NDUyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-27T21:18:46Z", "updated_at": "2021-02-27T21:18:46Z", "author_association": "OWNER", "body": "Here's a really wild idea: I wonder if it would be possible to run a source transformation against either the sync or the async versions of the code to produce the equivalent for the other paradigm?\r\n\r\nCould that even be as simple as a set of regular expressions against the `await ...` version that strips out or replaces the `await` and `async def` and `async for` statements?\r\n\r\nIf so... I could maintain just the async version, generate the sync version with a script and rely on robust unit testing to guarantee that this actually works.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 817989436, "label": "Async support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787150276", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242", "id": 787150276, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzE1MDI3Ng==", "user": {"value": 37962604, "label": "polyrand"}, "created_at": "2021-02-27T21:27:26Z", "updated_at": "2021-02-27T21:27:26Z", "author_association": "NONE", "body": "I had this resource by Seth Michael Larson saved https://github.com/sethmlarson/pycon-async-sync-poster I haven't had a look at it, but it may contain useful info.\r\n\r\nOn twitter, I mentioned passing an aiosqlite connection during the `Database` creation. I'm not 100% familiar with the `sqlite-utils` codebase, so I may be wrong here, but maybe decorating internal functions could be an option? Then they are awaited or not inside the decorator depending on how they are called.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 817989436, "label": "Async support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787175126", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242", "id": 787175126, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzE3NTEyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-27T21:55:05Z", "updated_at": "2021-02-27T21:55:05Z", "author_association": "OWNER", "body": "\"how to use some new tools to more easily maintain a codebase that supports both async and synchronous I/O and multiple async libraries\" - yeah that's exactly what I need, thank you!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 817989436, "label": "Async support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787186826", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242", "id": 787186826, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzE4NjgyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-27T22:01:54Z", "updated_at": "2021-02-27T22:01:54Z", "author_association": "OWNER", "body": "`unasync` is an implementation of the exact pattern I was talking about above - it uses the `tokenize` module from the Python standard library to apply some clever rules to transform an async codebase into a sync one. https://unasync.readthedocs.io/en/latest/ - implementation here: https://github.com/python-trio/unasync/blob/v0.5.0/src/unasync/__init__.py", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 817989436, "label": "Async support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787190562", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242", "id": 787190562, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzE5MDU2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-27T22:04:00Z", "updated_at": "2021-02-27T22:04:00Z", "author_association": "OWNER", "body": "From the poster here: https://github.com/sethmlarson/pycon-async-sync-poster/blob/master/poster.pdf\r\n\r\n<img width=\"624\" alt=\"pycon-async-sync-poster_poster_pdf_at_master_\u00b7_sethmlarson_pycon-async-sync-poster\" src=\"https://user-images.githubusercontent.com/9599/109401634-9f0a1400-7904-11eb-8b3a-37df0678b8dc.png\">\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 817989436, "label": "Async support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787195536", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242", "id": 787195536, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzE5NTUzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-27T22:13:24Z", "updated_at": "2021-02-27T22:13:24Z", "author_association": "OWNER", "body": "Some other interesting background reading: https://docs.sqlalchemy.org/en/14/orm/extensions/asyncio.html - in particular see how SQLALchemy has a `await conn.run_sync(meta.drop_all)` mechanism for running methods that haven't themselves been provided in an async version", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 817989436, "label": "Async support"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787198202", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242", "id": 787198202, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzE5ODIwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-02-27T22:33:58Z", "updated_at": "2021-02-27T22:33:58Z", "author_association": "OWNER", "body": "Hah or use this trick, which genuinely rewrites the code at runtime using a class decorator! https://github.com/python-happybase/aiohappybase/blob/0990ef45cfdb720dc987afdb4957a0fac591cb99/aiohappybase/sync/_util.py#L19-L32", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 817989436, "label": "Async support"}, "performed_via_github_app": null}