html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/pull/683#issuecomment-590518182,https://api.github.com/repos/simonw/datasette/issues/683,590518182,MDEyOklzc3VlQ29tbWVudDU5MDUxODE4Mg==,9599,2020-02-24T19:53:12Z,2020-02-24T19:53:12Z,OWNER,"Next steps are from comment https://github.com/simonw/datasette/issues/682#issuecomment-590517338 > I'm going to move ahead without needing that ability though. I figure SQLite writes are _fast_, and plugins can be trusted to implement just fast writes. So I'm going to support either fire-and-forget writes (they get added to the queue and a task ID is returned) or have the option to block awaiting the completion of the write (using Janus) but let callers decide which version they want. I may add optional timeouts some time in the future. > > I am going to make both `execute_write()` and `execute_write_fn()` awaitable functions though, for consistency with `.execute()` and to give me flexibility to change how they work in the future. > > I'll also add a `block=True` option to both of them which causes the function to wait for the write to be successfully executed - defaults to `False` (fire-and-forget mode).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428, https://github.com/simonw/datasette/pull/683#issuecomment-590592581,https://api.github.com/repos/simonw/datasette/issues/683,590592581,MDEyOklzc3VlQ29tbWVudDU5MDU5MjU4MQ==,9599,2020-02-24T23:00:44Z,2020-02-24T23:01:09Z,OWNER,"I've been testing this out by running one-off demo plugins. I saved the following in a file called `write-plugins/log_asgi.py` (it's a hacked around copy of [asgi-log-to-sqlite](https://github.com/simonw/asgi-log-to-sqlite)) and then running `datasette data.db --plugins-dir=write-plugins/`: ```python from datasette import hookimpl import sqlite_utils import time class AsgiLogToSqliteViaWriteQueue: lookup_columns = ( ""path"", ""user_agent"", ""referer"", ""accept_language"", ""content_type"", ""query_string"", ) def __init__(self, app, db): self.app = app self.db = db self._tables_ensured = False async def ensure_tables(self): def _ensure_tables(conn): db = sqlite_utils.Database(conn) for column in self.lookup_columns: table = ""{}s"".format(column) if not db[table].exists(): db[table].create({""id"": int, ""name"": str}, pk=""id"") if ""requests"" not in db.table_names(): db[""requests""].create( { ""start"": float, ""method"": str, ""path"": int, ""query_string"": int, ""user_agent"": int, ""referer"": int, ""accept_language"": int, ""http_status"": int, ""content_type"": int, ""client_ip"": str, ""duration"": float, ""body_size"": int, }, foreign_keys=self.lookup_columns, ) await self.db.execute_write_fn(_ensure_tables) async def __call__(self, scope, receive, send): if not self._tables_ensured: self._tables_ensured = True await self.ensure_tables() response_headers = [] body_size = 0 http_status = None async def wrapped_send(message): nonlocal body_size, response_headers, http_status if message[""type""] == ""http.response.start"": response_headers = message[""headers""] http_status = message[""status""] if message[""type""] == ""http.response.body"": body_size += len(message[""body""]) await send(message) start = time.time() await self.app(scope, receive, wrapped_send) end = time.time() path = str(scope[""path""]) query_string = None if scope.get(""query_string""): query_string = ""?{}"".format(scope[""query_string""].decode(""utf8"")) request_headers = dict(scope.get(""headers"") or []) referer = header(request_headers, ""referer"") user_agent = header(request_headers, ""user-agent"") accept_language = header(request_headers, ""accept-language"") content_type = header(dict(response_headers), ""content-type"") def _log_to_database(conn): db = sqlite_utils.Database(conn) db[""requests""].insert( { ""start"": start, ""method"": scope[""method""], ""path"": lookup(db, ""paths"", path), ""query_string"": lookup(db, ""query_strings"", query_string), ""user_agent"": lookup(db, ""user_agents"", user_agent), ""referer"": lookup(db, ""referers"", referer), ""accept_language"": lookup(db, ""accept_languages"", accept_language), ""http_status"": http_status, ""content_type"": lookup(db, ""content_types"", content_type), ""client_ip"": scope.get(""client"", (None, None))[0], ""duration"": end - start, ""body_size"": body_size, }, alter=True, foreign_keys=self.lookup_columns, ) await self.db.execute_write_fn(_log_to_database) def header(d, name): return d.get(name.encode(""utf8""), b"""").decode(""utf8"") or None def lookup(db, table, value): return db[table].lookup({""name"": value}) if value else None @hookimpl def asgi_wrapper(datasette): def wrap_with_class(app): return AsgiLogToSqliteViaWriteQueue( app, next(iter(datasette.databases.values())) ) return wrap_with_class ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428, https://github.com/simonw/datasette/pull/683#issuecomment-590593120,https://api.github.com/repos/simonw/datasette/issues/683,590593120,MDEyOklzc3VlQ29tbWVudDU5MDU5MzEyMA==,9599,2020-02-24T23:02:30Z,2020-02-24T23:02:30Z,OWNER,"I'm going to muck around with a couple more demo plugins - in particular one derived from [datasette-upload-csvs](https://github.com/simonw/datasette-upload-csvs) - to make sure I'm comfortable with this API - then add a couple of tests and merge it with documentation that warns ""this is still an experimental feature and may change"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428, https://github.com/simonw/datasette/pull/683#issuecomment-590598248,https://api.github.com/repos/simonw/datasette/issues/683,590598248,MDEyOklzc3VlQ29tbWVudDU5MDU5ODI0OA==,9599,2020-02-24T23:18:50Z,2020-02-24T23:18:50Z,OWNER,"I'm not convinced by the return value of the `.execute_write_fn()` method: https://github.com/simonw/datasette/blob/ab2348280206bde1390b931ae89d372c2f74b87e/datasette/database.py#L79-L83 Do I really need that `WriteResponse` class or can I do something nicer?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428, https://github.com/simonw/datasette/pull/683#issuecomment-590598689,https://api.github.com/repos/simonw/datasette/issues/683,590598689,MDEyOklzc3VlQ29tbWVudDU5MDU5ODY4OQ==,9599,2020-02-24T23:20:11Z,2020-02-24T23:20:11Z,OWNER,"I think `if block` it makes sense to return the return value of the function that was executed. Without it all I really need to do is return the `uuid` so something could theoretically poll for completion later on. But is it weird having a function that returns different types depending on if you passed `block=True` or not? Should they be differently named functions? I'm OK with the `block=True` pattern changing the return value I think.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428, https://github.com/simonw/datasette/pull/683#issuecomment-590599257,https://api.github.com/repos/simonw/datasette/issues/683,590599257,MDEyOklzc3VlQ29tbWVudDU5MDU5OTI1Nw==,9599,2020-02-24T23:21:56Z,2020-02-24T23:22:35Z,OWNER,"Also: are UUIDs really necessary here or could I use a simpler form of task identifier? Like an in-memory counter variable that starts at 0 and increments every time this instance of Datasette issues a new task ID? The neat thing about UUIDs is that I don't have to worry if there are multiple Datasette instances accepting writes behind a load balancer. That seems pretty unlikely (especially considering SQLite databases encourage only one process to be writing at a time)... but I am experimenting with PostgreSQL support in #670 so it's probably worth ensuring these task IDs really are globally unique. I'm going to stick with UUIDs. They're short-lived enough that their size doesn't really matter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428, https://github.com/simonw/datasette/pull/683#issuecomment-590606825,https://api.github.com/repos/simonw/datasette/issues/683,590606825,MDEyOklzc3VlQ29tbWVudDU5MDYwNjgyNQ==,9599,2020-02-24T23:47:38Z,2020-02-24T23:47:38Z,OWNER,"Another demo plugin: `delete_table.py` ```python from datasette import hookimpl from datasette.utils import escape_sqlite from starlette.responses import HTMLResponse from starlette.endpoints import HTTPEndpoint class DeleteTableApp(HTTPEndpoint): def __init__(self, scope, receive, send, datasette): self.datasette = datasette super().__init__(scope, receive, send) async def post(self, request): formdata = await request.form() database = formdata[""database""] db = self.datasette.databases[database] await db.execute_write(""drop table {}"".format(escape_sqlite(formdata[""table""]))) return HTMLResponse(""Table has been deleted."") @hookimpl def asgi_wrapper(datasette): def wrap_with_asgi_auth(app): async def wrapped_app(scope, recieve, send): if scope[""path""] == ""/-/delete-table"": await DeleteTableApp(scope, recieve, send, datasette) else: await app(scope, recieve, send) return wrapped_app return wrap_with_asgi_auth ``` Then I saved this as `table.html` in the `write-templates/` directory: ```html+django {% extends ""default:table.html"" %} {% block content %}
{{ super() }} {% endblock %} ``` (Needs CSRF protection added) I ran Datasette like this: $ datasette --plugins-dir=write-plugins/ data.db --template-dir=write-templates/ Result: I can delete tables! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428, https://github.com/simonw/datasette/pull/683#issuecomment-590607385,https://api.github.com/repos/simonw/datasette/issues/683,590607385,MDEyOklzc3VlQ29tbWVudDU5MDYwNzM4NQ==,9599,2020-02-24T23:49:37Z,2020-02-24T23:49:37Z,OWNER,"Here's the `upload_csv.py` plugin file I've been playing with: ```python from datasette import hookimpl from starlette.responses import PlainTextResponse, HTMLResponse from starlette.endpoints import HTTPEndpoint import csv as csv_std import codecs import sqlite_utils class UploadApp(HTTPEndpoint): def __init__(self, scope, receive, send, datasette): self.datasette = datasette super().__init__(scope, receive, send) def get_database(self): # For the moment just use the first one that's not immutable mutable = [db for db in self.datasette.databases.values() if db.is_mutable] return mutable[0] async def get(self, request): return HTMLResponse( await self.datasette.render_template( ""upload_csv.html"", {""database_name"": self.get_database().name} ) ) async def post(self, request): formdata = await request.form() csv = formdata[""csv""] # csv.file is a SpooledTemporaryFile, I can read it directly filename = csv.filename # TODO: Support other encodings: reader = csv_std.reader(codecs.iterdecode(csv.file, ""utf-8"")) headers = next(reader) docs = (dict(zip(headers, row)) for row in reader) if filename.endswith("".csv""): filename = filename[:-4] # Import data into a table of that name using sqlite-utils db = self.get_database() def fn(conn): writable_conn = sqlite_utils.Database(db.path) writable_conn[filename].insert_all(docs, alter=True) return writable_conn[filename].count # Without block=True we may attempt 'select count(*) from ...' # before the table has been created by the write thread count = await db.execute_write_fn(fn, block=True) return HTMLResponse( await self.datasette.render_template( ""upload_csv_done.html"", { ""database"": self.get_database().name, ""table"": filename, ""num_docs"": count, }, ) ) @hookimpl def asgi_wrapper(datasette): def wrap_with_asgi_auth(app): async def wrapped_app(scope, recieve, send): if scope[""path""] == ""/-/upload-csv"": await UploadApp(scope, recieve, send, datasette) else: await app(scope, recieve, send) return wrapped_app return wrap_with_asgi_auth ``` I also dropped copies of the two template files from https://github.com/simonw/datasette-upload-csvs/tree/699e6ca591f36264bfc8e590d877e6852f274beb/datasette_upload_csvs/templates into my `write-templates/` directory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428, https://github.com/simonw/datasette/pull/683#issuecomment-590608228,https://api.github.com/repos/simonw/datasette/issues/683,590608228,MDEyOklzc3VlQ29tbWVudDU5MDYwODIyOA==,9599,2020-02-24T23:52:35Z,2020-02-24T23:52:35Z,OWNER,I'm going to punt on the ability to introspect the write queue and poll for completion using a UUID for the moment. Can add those later.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428, https://github.com/simonw/datasette/pull/683#issuecomment-590610180,https://api.github.com/repos/simonw/datasette/issues/683,590610180,MDEyOklzc3VlQ29tbWVudDU5MDYxMDE4MA==,9599,2020-02-25T00:00:07Z,2020-02-25T00:00:07Z,OWNER,"Basic stuff to cover in unit tests: - Exercise `.execute_write(sql)` - both with block=True and block=False - Exercise `.execute_write_fn(fn)` in the same way - Throw 10 updates in the queue, block on just the last one, check it worked correctly I'm going to write these tests directly against a `Database()` object rather than booting up an entire Datasette instance.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428, https://github.com/simonw/datasette/pull/683#issuecomment-590614896,https://api.github.com/repos/simonw/datasette/issues/683,590614896,MDEyOklzc3VlQ29tbWVudDU5MDYxNDg5Ng==,9599,2020-02-25T00:16:51Z,2020-02-25T00:16:51Z,OWNER,"The other problem with the poll-for-UUID-completion idea: how long does this mean Datasette needs to keep holding onto the `WriteTask` objects? Maybe we say you only get to ask ""is this UUID still in the queue"" and if the answer is ""no"" then you assume the task has been completed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",570101428, https://github.com/simonw/datasette/pull/683#issuecomment-590617822,https://api.github.com/repos/simonw/datasette/issues/683,590617822,MDEyOklzc3VlQ29tbWVudDU5MDYxNzgyMg==,9599,2020-02-25T00:26:48Z,2020-02-25T00:26:48Z,OWNER,"This failing test is a nasty one - the whole thing just hangs (so I imagine Travis will run for a while before hopefully giving up). Here's what happens if I add `--full-trace` and then hit Ctrl+C to cancel a test run: ``` $ pytest -k test_execute_write_fn_block_true --full-trace =================================================================== test session starts =================================================================== platform darwin -- Python 3.7.5, pytest-5.2.4, py-1.8.1, pluggy-0.13.1 rootdir: /Users/simonw/Dropbox/Development/datasette, inifile: pytest.ini plugins: asyncio-0.10.0 collected 410 items / 409 deselected / 1 selected tests/test_database.py ^C^C ================================================================= 409 deselected in 4.45s ================================================================= Traceback (most recent call last): File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py"", line 193, in wrap_session session.exitstatus = doit(config, session) or 0 File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/_pytest/main.py"", line 237, in _main config.hook.pytest_runtestloop(session=session) File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/hooks.py"", line 286, in __call__ return self._hookexec(self, self.get_hookimpls(), kwargs) File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 93, in _hookexec return self._inner_hookexec(hook, methods, kwargs) File ""/Users/simonw/.local/share/virtualenvs/datasette-oJRYYJuA/lib/python3.7/site-packages/pluggy/manager.py"", line 87, in