issue_comments
7,896 rows sorted by id descending
This data as json, CSV (advanced)
Suggested facets: created_at (date), updated_at (date)
| id ▲ | html_url | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1112734577 | https://github.com/simonw/datasette/issues/1729#issuecomment-1112734577 | https://api.github.com/repos/simonw/datasette/issues/1729 | IC_kwDOBm6k_c5CUvtx | simonw 9599 | 2022-04-28T23:08:42Z | 2022-04-28T23:08:42Z | OWNER | That prototype is a very small amount of code so far: ```diff diff --git a/datasette/renderer.py b/datasette/renderer.py index 4508949..b600e1b 100644 --- a/datasette/renderer.py +++ b/datasette/renderer.py @@ -28,6 +28,10 @@ def convert_specific_columns_to_json(rows, columns, json_cols): def json_renderer(args, data, view_name): """Render a response as JSON""" + from pprint import pprint + + pprint(data) + status_code = 200 # Handle the _json= parameter which may modify data["rows"] @@ -43,6 +47,41 @@ def json_renderer(args, data, view_name): if "rows" in data and not value_as_boolean(args.get("_json_infinity", "0")): data["rows"] = [remove_infinites(row) for row in data["rows"]] + # Start building the default JSON here + columns = data["columns"] + next_url = data.get("next_url") + output = { + "rows": [dict(zip(columns, row)) for row in data["rows"]], + "next": data["next"], + "next_url": next_url, + } + + extras = set(args.getlist("_extra")) + + extras_map = { + # _extra= : data[field] + "count": "filtered_table_rows_count", + "facet_results": "facet_results", + "suggested_facets": "suggested_facets", + "columns": "columns", + "primary_keys": "primary_keys", + "query_ms": "query_ms", + "query": "query", + } + for extra_key, data_key in extras_map.items(): + if extra_key in extras: + output[extra_key] = data[data_key] + + body = json.dumps(output, cls=CustomJSONEncoder) + content_type = "application/json; charset=utf-8" + headers = {} + if next_url: + headers["link"] = f'<{next_url}>; rel="next"' + return Response( + body, status=status_code, headers=headers, content_type=content_type + ) + + # Deal with the _shape option shape = args.get("_shape", "arrays") # if there's an error, ignore the shape entirely ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Implement ?_extra and new API design for TableView 1219385669 | |
| 1112732563 | https://github.com/simonw/datasette/issues/1729#issuecomment-1112732563 | https://api.github.com/repos/simonw/datasette/issues/1729 | IC_kwDOBm6k_c5CUvOT | simonw 9599 | 2022-04-28T23:05:03Z | 2022-04-28T23:05:03Z | OWNER | OK, the prototype of this is looking really good - it's very pleasant to use. `http://127.0.0.1:8001/github_memory/issue_comments.json?_search=simon&_sort=id&_size=5&_extra=query_ms&_extra=count&_col=body` returns this: ```json { "rows": [ { "id": 338854988, "body": " /database-name/table-name?name__contains=simon&sort=id+desc\r\n\r\nNote that if there's a column called \"sort\" you can still do sort__exact=blah\r\n\r\n" }, { "id": 346427794, "body": "Thanks. There is a way to use pip to grab apsw, which also let's you configure it (flags to build extensions, use an internal sqlite, etc). Don't know how that works as a dependency for another package, though.\n\nOn November 22, 2017 11:38:06 AM EST, Simon Willison <notifications@github.com> wrote:\n>I have a solution for FTS already, but I'm interested in apsw as a\n>mechanism for allowing custom virtual tables to be written in Python\n>(pysqlite only lets you write custom functions)\n>\n>Not having PyPI support is pretty tough though. I'm planning a\n>plugin/extension system which would be ideal for things like an\n>optional apsw mode, but that's a lot harder if apsw isn't in PyPI.\n>\n>-- \n>You are receiving this because you authored the thread.\n>Reply to this email directly or view it on GitHub:\n>https://github.com/simonw/datasette/issues/144#issuecomment-346405660\n" }, { "id": 348252037, "body": "WOW!\n\n--\nPaul Ford // (646) 369-7128 // @ftrain\n\nOn Thu, Nov 30, 2017 at 11:47 AM, Simon Willison <notifications@github.com>\nwrote:\n\n> Remaining work on this now lives in a milestone:\n> https://github.com/simonw/datasette/milestone/6\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/simonw/datasette/issues/153#issuecomment-348248406>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/AABPKHzaVPKwTOoHouK2aMUnM-mPnPk6ks5s7twzgaJpZM4Qq2zW>\n> .\n>\n" }, { … | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Implement ?_extra and new API design for TableView 1219385669 | |
| 1112730416 | https://github.com/simonw/datasette/issues/1729#issuecomment-1112730416 | https://api.github.com/repos/simonw/datasette/issues/1729 | IC_kwDOBm6k_c5CUusw | simonw 9599 | 2022-04-28T23:01:21Z | 2022-04-28T23:01:21Z | OWNER | I'm not sure what to do about the `"truncated": true/false` key. It's not really relevant to table results, since they are paginated whether or not you ask for them to be. It plays a role in query results, where you might run `select * from table` and get back 1000 results because Datasette truncates at that point rather than returning everything. Adding it to every table result and always setting it to `"truncated": false` feels confusing. I think I'm going to keep it exclusively in the default representation for the `/db?sql=...` query endpoint, and not return it at all for tables. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Implement ?_extra and new API design for TableView 1219385669 | |
| 1112721321 | https://github.com/simonw/datasette/issues/1729#issuecomment-1112721321 | https://api.github.com/repos/simonw/datasette/issues/1729 | IC_kwDOBm6k_c5CUsep | simonw 9599 | 2022-04-28T22:44:05Z | 2022-04-28T22:44:14Z | OWNER | I may be able to implement this mostly in the `json_renderer()` function: https://github.com/simonw/datasette/blob/94a3171b01fde5c52697aeeff052e3ad4bab5391/datasette/renderer.py#L29-L34 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Implement ?_extra and new API design for TableView 1219385669 | |
| 1112717745 | https://github.com/simonw/datasette/issues/1729#issuecomment-1112717745 | https://api.github.com/repos/simonw/datasette/issues/1729 | IC_kwDOBm6k_c5CUrmx | simonw 9599 | 2022-04-28T22:38:39Z | 2022-04-28T22:39:05Z | OWNER | (I remain keen on the idea of shipping a plugin that restores the old default API shape to people who have written pre-Datasette-1.0 code against it, but I'll tackle that much later. I really like how jQuery has a culture of doing this.) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Implement ?_extra and new API design for TableView 1219385669 | |
| 1112717210 | https://github.com/simonw/datasette/issues/1729#issuecomment-1112717210 | https://api.github.com/repos/simonw/datasette/issues/1729 | IC_kwDOBm6k_c5CUrea | simonw 9599 | 2022-04-28T22:37:37Z | 2022-04-28T22:37:37Z | OWNER | This means `filtered_table_rows_count` is going to become `count`. I had originally picked that terrible name to avoid confusion between the count of all rows in the table and the count of rows that were filtered. I'll add `?_extra=table_count` for getting back the full table count instead. I think `count` is clear enough! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Implement ?_extra and new API design for TableView 1219385669 | |
| 1112716611 | https://github.com/simonw/datasette/issues/1729#issuecomment-1112716611 | https://api.github.com/repos/simonw/datasette/issues/1729 | IC_kwDOBm6k_c5CUrVD | simonw 9599 | 2022-04-28T22:36:24Z | 2022-04-28T22:36:24Z | OWNER | Then I'm going to implement the following `?_extra=` options: - `?_extra=facet_results` - to see facet results - `?_extra=suggested_facets` - for suggested facets - `?_extra=count` - for the count of total rows - `?_extra=columns` - for a list of column names - `?_extra=primary_keys` - for a list of primary keys - `?_extra=query` - a `{"sql" "select ...", "params": {}}` object I thought about having `?_extra=facet_results` returned automatically if the user specifies at least one `?_facet` - but that doesn't work for default facets configured in `metadata.json` - how can the user opt out of those being returned? So I'm going to say you don't see facets at all if you don't include `?_extra=facet_results`. I'm tempted to add `?_extra=_all` to return everything, but I can decide if that's a good idea later. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Implement ?_extra and new API design for TableView 1219385669 | |
| 1112713581 | https://github.com/simonw/datasette/issues/1729#issuecomment-1112713581 | https://api.github.com/repos/simonw/datasette/issues/1729 | IC_kwDOBm6k_c5CUqlt | simonw 9599 | 2022-04-28T22:31:11Z | 2022-04-28T22:31:11Z | OWNER | I'm going to change the default API response to look like this: ```json { "rows": [ { "pk": 1, "created": "2019-01-14 08:00:00", "planet_int": 1, "on_earth": 1, "state": "CA", "_city_id": 1, "_neighborhood": "Mission", "tags": "[\"tag1\", \"tag2\"]", "complex_array": "[{\"foo\": \"bar\"}]", "distinct_some_null": "one", "n": "n1" }, { "pk": 2, "created": "2019-01-14 08:00:00", "planet_int": 1, "on_earth": 1, "state": "CA", "_city_id": 1, "_neighborhood": "Dogpatch", "tags": "[\"tag1\", \"tag3\"]", "complex_array": "[]", "distinct_some_null": "two", "n": "n2" } ], "next": null, "next_url": null } ``` Basically https://latest.datasette.io/fixtures/facetable.json?_shape=objects but with just the `rows`, `next` and `next_url` fields returned by default. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Implement ?_extra and new API design for TableView 1219385669 | |
| 1112711115 | https://github.com/simonw/datasette/issues/1715#issuecomment-1112711115 | https://api.github.com/repos/simonw/datasette/issues/1715 | IC_kwDOBm6k_c5CUp_L | simonw 9599 | 2022-04-28T22:26:56Z | 2022-04-28T22:26:56Z | OWNER | I'm not going to use `asyncinject` in this refactor - at least not until I really need it. My research in these issues has put me off the idea ( in favour of `asyncio.gather()` or even not trying for parallel execution at all): - #1727 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor TableView to use asyncinject 1212823665 | |
| 1112668411 | https://github.com/simonw/datasette/issues/1727#issuecomment-1112668411 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CUfj7 | simonw 9599 | 2022-04-28T21:25:34Z | 2022-04-28T21:25:44Z | OWNER | The two most promising theories at the moment, from here and Twitter and the SQLite forum, are: - SQLite is I/O bound - it generally only goes as fast as it can load data from disk. Multiple connections all competing for the same file on disk are going to end up blocked at the file system layer. But maybe this means in-memory databases will perform better? - It's the GIL. The sqlite3 C code may release the GIL, but the bits that do things like assembling `Row` objects to return still happen in Python, and that Python can only run on a single core. A couple of ways to research the in-memory theory: - Use a RAM disk on macOS (or Linux). https://stackoverflow.com/a/2033417/6083 has instructions - short version: hdiutil attach -nomount ram://$((2 * 1024 * 100)) diskutil eraseVolume HFS+ RAMDisk name-returned-by-previous-command (was `/dev/disk2` when I tried it) cd /Volumes/RAMDisk cp ~/fixtures.db . - Copy Datasette databases into an in-memory database on startup. I built a new plugin to do that here: https://github.com/simonw/datasette-copy-to-memory I need to do some more, better benchmarks using these different approaches. https://twitter.com/laurencerowe/status/1519780174560169987 also suggests: > Maybe try: > 1. Copy the sqlite file to /dev/shm and rerun (all in ram.) > 2. Create a CTE which calculates Fibonacci or similar so you can test something completely cpu bound (only return max value or something to avoid crossing between sqlite/Python.) I like that second idea a lot - I could use the mandelbrot example from https://www.sqlite.org/lang_with.html#outlandish_recursive_query_examples | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111955628 | https://github.com/simonw/datasette/issues/1633#issuecomment-1111955628 | https://api.github.com/repos/simonw/datasette/issues/1633 | IC_kwDOBm6k_c5CRxis | henrikek 6613091 | 2022-04-28T09:12:56Z | 2022-04-28T09:12:56Z | NONE | I have verified that the problem with base_url still exists in the latest version 0.61.1. I would need some guidance if my code change suggestion is correct or if base_url should be included in some other code? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | base_url or prefix does not work with _exact match 1129052172 | |
| 1111752676 | https://github.com/simonw/datasette/issues/1728#issuecomment-1111752676 | https://api.github.com/repos/simonw/datasette/issues/1728 | IC_kwDOBm6k_c5CQ__k | wragge 127565 | 2022-04-28T05:11:54Z | 2022-04-28T05:11:54Z | CONTRIBUTOR | And in terms of the bug, yep I agree that option 2 would be the most useful and least frustrating. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Writable canned queries fail with useless non-error against immutable databases 1218133366 | |
| 1111751734 | https://github.com/simonw/datasette/issues/1728#issuecomment-1111751734 | https://api.github.com/repos/simonw/datasette/issues/1728 | IC_kwDOBm6k_c5CQ_w2 | wragge 127565 | 2022-04-28T05:09:59Z | 2022-04-28T05:09:59Z | CONTRIBUTOR | Thanks, I'll give it a try! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Writable canned queries fail with useless non-error against immutable databases 1218133366 | |
| 1111726586 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111726586 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQ5n6 | simonw 9599 | 2022-04-28T04:17:16Z | 2022-04-28T04:19:31Z | OWNER | I could experiment with the `await asyncio.run_in_executor(processpool_executor, fn)` mechanism described in https://stackoverflow.com/a/29147750 Code examples: https://cs.github.com/?scopeName=All+repos&scope=&q=run_in_executor+ProcessPoolExecutor | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111725638 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111725638 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQ5ZG | simonw 9599 | 2022-04-28T04:15:15Z | 2022-04-28T04:15:15Z | OWNER | Useful theory from Keith Medcalf https://sqlite.org/forum/forumpost/e363c69d3441172e > This is true, but the concurrency is limited to the execution which occurs with the GIL released (that is, in the native C sqlite3 library itself). Each row (for example) can be retrieved in parallel but "constructing the python return objects for each row" will be serialized (by the GIL). > > That is to say that if your have two python threads each with their own connection, and each one is performing a select that returns 1,000,000 rows (lets say that is 25% of the candidates for each select) then the difference in execution time between executing two python threads in parallel vs a single serial thead will not be much different (if even detectable at all). In fact it is possible that the multiple-threaded version takes longer to run both queries to completion because of the increased contention over a shared resource (the GIL). So maybe this is a GIL thing. I should test with some expensive SQL queries (maybe big aggregations against large tables) and see if I can spot an improvement there. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111714665 | https://github.com/simonw/datasette/issues/1728#issuecomment-1111714665 | https://api.github.com/repos/simonw/datasette/issues/1728 | IC_kwDOBm6k_c5CQ2tp | simonw 9599 | 2022-04-28T03:52:47Z | 2022-04-28T03:52:58Z | OWNER | Nice custom template/theme! Yeah, for that I'd recommend hosting elsewhere - on a regular VPS (I use `systemd` like this: https://docs.datasette.io/en/stable/deploying.html#running-datasette-using-systemd ) or using Fly if you want to tub containers without managing a full server. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Writable canned queries fail with useless non-error against immutable databases 1218133366 | |
| 1111712953 | https://github.com/simonw/datasette/issues/1728#issuecomment-1111712953 | https://api.github.com/repos/simonw/datasette/issues/1728 | IC_kwDOBm6k_c5CQ2S5 | wragge 127565 | 2022-04-28T03:48:36Z | 2022-04-28T03:48:36Z | CONTRIBUTOR | I don't think that'd work for this project. The db is very big, and my aim was to have an environment where researchers could be making use of the data, but be easily able to add corrections to the HTR/OCR extracted data when they came across problems. It's in its immutable (!) form here: https://sydney-stock-exchange-xqtkxtd5za-ts.a.run.app/stock_exchange/stocks | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Writable canned queries fail with useless non-error against immutable databases 1218133366 | |
| 1111708206 | https://github.com/simonw/datasette/issues/1728#issuecomment-1111708206 | https://api.github.com/repos/simonw/datasette/issues/1728 | IC_kwDOBm6k_c5CQ1Iu | simonw 9599 | 2022-04-28T03:38:56Z | 2022-04-28T03:38:56Z | OWNER | In terms of this bug, there are a few potential fixes: 1. Detect the write to a immutable database and show the user a proper, meaningful error message in the red error box at the top of the page 2. Don't allow the user to even submit the form - show a message saying that this canned query is unavailable because the database cannot be written to 3. Don't even allow Datasette to start running at all - if there's a canned query configured in `metadata.yml` and the database it refers to is in `-i` immutable mode throw an error on startup I'm not keen on that last one because it would be frustrating if you couldn't launch Datasette just because you had an old canned query lying around in your metadata file. So I'm leaning towards option 2. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Writable canned queries fail with useless non-error against immutable databases 1218133366 | |
| 1111707384 | https://github.com/simonw/datasette/issues/1728#issuecomment-1111707384 | https://api.github.com/repos/simonw/datasette/issues/1728 | IC_kwDOBm6k_c5CQ074 | simonw 9599 | 2022-04-28T03:36:46Z | 2022-04-28T03:36:56Z | OWNER | A more realistic solution (which I've been using on several of my own projects) is to keep the data itself in GitHub and encourage users to edit it there - using the GitHub web interface to edit YAML files or similar. Needs your users to be comfortable hand-editing YAML though! You can at least guard against critical errors by having CI run tests against their YAML before deploying. I have a dream of building a more friendly web forms interface which edits the YAML back on GitHub for the user, but that's just a concept at the moment. Even more fun would be if a user-friendly form could submit PRs for review without the user having to know what a PR is! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Writable canned queries fail with useless non-error against immutable databases 1218133366 | |
| 1111706519 | https://github.com/simonw/datasette/issues/1728#issuecomment-1111706519 | https://api.github.com/repos/simonw/datasette/issues/1728 | IC_kwDOBm6k_c5CQ0uX | simonw 9599 | 2022-04-28T03:34:49Z | 2022-04-28T03:34:49Z | OWNER | I've wanted to do stuff like that on Cloud Run too. So far I've assumed that it's not feasible, but recently I've been wondering how hard it would be to have a small (like less than 100KB or so) Datasette instance which persists data to a backing GitHub repository such that when it starts up it can pull the latest copy and any time someone edits it can push their changes. I'm still not sure it would work well on Cloud Run due to the uncertainty at what would happen if Cloud Run decided to boot up a second instance - but it's still an interesting thought exercise. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Writable canned queries fail with useless non-error against immutable databases 1218133366 | |
| 1111705323 | https://github.com/simonw/datasette/issues/1728#issuecomment-1111705323 | https://api.github.com/repos/simonw/datasette/issues/1728 | IC_kwDOBm6k_c5CQ0br | wragge 127565 | 2022-04-28T03:32:06Z | 2022-04-28T03:32:06Z | CONTRIBUTOR | Ah, that would be it! I have a core set of data which doesn't change to which I want authorised users to be able to submit corrections. I was going to deal with the persistence issue by just grabbing the user corrections at regular intervals and saving to GitHub. I might need to rethink. Thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Writable canned queries fail with useless non-error against immutable databases 1218133366 | |
| 1111705069 | https://github.com/simonw/datasette/issues/1728#issuecomment-1111705069 | https://api.github.com/repos/simonw/datasette/issues/1728 | IC_kwDOBm6k_c5CQ0Xt | simonw 9599 | 2022-04-28T03:31:33Z | 2022-04-28T03:31:33Z | OWNER | Confirmed - this is a bug where immutable databases fail to show a useful error if you write to them with a canned query. Steps to reproduce: ``` echo ' databases: writable: queries: add_name: sql: insert into names(name) values (:name) write: true ' > write-metadata.yml echo '{"name": "Simon"}' | sqlite-utils insert writable.db names - datasette writable.db -m write-metadata.yml ``` Then visit http://127.0.0.1:8001/writable/add_name - adding names works. Now do this instead: ``` datasette -i writable.db -m write-metadata.yml ``` And I'm getting a broken error:  | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Writable canned queries fail with useless non-error against immutable databases 1218133366 | |
| 1111699175 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111699175 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQy7n | simonw 9599 | 2022-04-28T03:19:48Z | 2022-04-28T03:20:08Z | OWNER | I ran `py-spy` and then hammered refresh a bunch of times on the `http://127.0.0.1:8856/github/commits?_facet=repo&_facet=committer&_trace=1&_noparallel=` page - it generated this SVG profile for me. The area on the right is the threads running the DB queries:  Interactive version here: https://static.simonwillison.net/static/2022/datasette-parallel-profile.svg | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111698307 | https://github.com/simonw/datasette/issues/1728#issuecomment-1111698307 | https://api.github.com/repos/simonw/datasette/issues/1728 | IC_kwDOBm6k_c5CQyuD | simonw 9599 | 2022-04-28T03:18:02Z | 2022-04-28T03:18:02Z | OWNER | If the behaviour you are seeing is because the database is running in immutable mode then that's a bug - you should get a useful error message instead! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Writable canned queries fail with useless non-error against immutable databases 1218133366 | |
| 1111697985 | https://github.com/simonw/datasette/issues/1728#issuecomment-1111697985 | https://api.github.com/repos/simonw/datasette/issues/1728 | IC_kwDOBm6k_c5CQypB | simonw 9599 | 2022-04-28T03:17:20Z | 2022-04-28T03:17:20Z | OWNER | How did you deploy to Cloud Run? `datasette publish cloudrun` defaults to running databases there in `-i` immutable mode, because if you managed to change a file on disk on Cloud Run those changes would be lost the next time your container restarted there. That's why I upgraded `datasette-publish-fly` to provide a way of working with their volumes support - they're the best option I know of right now for running Datasette in a container with a persistent volume that can accept writes: https://simonwillison.net/2022/Feb/15/fly-volumes/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Writable canned queries fail with useless non-error against immutable databases 1218133366 | |
| 1111683539 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111683539 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQvHT | simonw 9599 | 2022-04-28T02:47:57Z | 2022-04-28T02:47:57Z | OWNER | Maybe this is the Python GIL after all? I've been hoping that the GIL won't be an issue because the `sqlite3` module releases the GIL for the duration of the execution of a SQL query - see https://github.com/python/cpython/blob/f348154c8f8a9c254503306c59d6779d4d09b3a9/Modules/_sqlite/cursor.c#L749-L759 So I've been hoping this means that SQLite code itself can run concurrently on multiple cores even when Python threads cannot. But maybe I'm misunderstanding how that works? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111681513 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111681513 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQunp | simonw 9599 | 2022-04-28T02:44:26Z | 2022-04-28T02:44:26Z | OWNER | I could try `py-spy top`, which I previously used here: - https://github.com/simonw/datasette/issues/1673 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111661331 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111661331 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQpsT | simonw 9599 | 2022-04-28T02:07:31Z | 2022-04-28T02:07:31Z | OWNER | Asked on the SQLite forum about this here: https://sqlite.org/forum/forumpost/ffbfa9f38e | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111602802 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111602802 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQbZy | simonw 9599 | 2022-04-28T00:21:35Z | 2022-04-28T00:21:35Z | OWNER | Tried this but I'm getting back an empty JSON array of traces at the bottom of the page most of the time (intermittently it works correctly): ```diff diff --git a/datasette/database.py b/datasette/database.py index ba594a8..d7f9172 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -7,7 +7,7 @@ import sys import threading import uuid -from .tracer import trace +from .tracer import trace, trace_child_tasks from .utils import ( detect_fts, detect_primary_keys, @@ -207,30 +207,31 @@ class Database: time_limit_ms = custom_time_limit with sqlite_timelimit(conn, time_limit_ms): - try: - cursor = conn.cursor() - cursor.execute(sql, params if params is not None else {}) - max_returned_rows = self.ds.max_returned_rows - if max_returned_rows == page_size: - max_returned_rows += 1 - if max_returned_rows and truncate: - rows = cursor.fetchmany(max_returned_rows + 1) - truncated = len(rows) > max_returned_rows - rows = rows[:max_returned_rows] - else: - rows = cursor.fetchall() - truncated = False - except (sqlite3.OperationalError, sqlite3.DatabaseError) as e: - if e.args == ("interrupted",): - raise QueryInterrupted(e, sql, params) - if log_sql_errors: - sys.stderr.write( - "ERROR: conn={}, sql = {}, params = {}: {}\n".format( - conn, repr(sql), params, e + with trace("sql", database=self.name, sql=sql.strip(), params=params): + try: + cursor = conn.cursor() + cursor.execute(sql, params if params is not None else {}) + … | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111597176 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111597176 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQaB4 | simonw 9599 | 2022-04-28T00:11:44Z | 2022-04-28T00:11:44Z | OWNER | Though it would be interesting to also have the trace reveal how much time is spent in the functions that wrap that core SQL - the stuff that is being measured at the moment. I have a hunch that this could help solve the over-arching performance mystery. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111595319 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111595319 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQZk3 | simonw 9599 | 2022-04-28T00:09:45Z | 2022-04-28T00:11:01Z | OWNER | Here's where read queries are instrumented: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L241-L242 So the instrumentation is actually capturing quite a bit of Python activity before it gets to SQLite: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L179-L190 And then: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L204-L233 Ideally I'd like that `trace()` block to wrap just the `cursor.execute()` and `cursor.fetchmany(...)` or `cursor.fetchall()` calls. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111558204 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111558204 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQQg8 | simonw 9599 | 2022-04-27T22:58:39Z | 2022-04-27T22:58:39Z | OWNER | I should check my timing mechanism. Am I capturing the time taken just in SQLite or does it include time spent in Python crossing between async and threaded world and waiting for a thread pool worker to become available? That could explain the longer query times. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111553029 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111553029 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQPQF | simonw 9599 | 2022-04-27T22:48:21Z | 2022-04-27T22:48:21Z | OWNER | I wonder if it would be worth exploring multiprocessing here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111551076 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111551076 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQOxk | simonw 9599 | 2022-04-27T22:44:51Z | 2022-04-27T22:45:04Z | OWNER | Really wild idea: what if I created three copies of the SQLite database file - as three separate file names - and then balanced the parallel queries across all these? Any chance that could avoid any mysterious locking issues? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111535818 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111535818 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CQLDK | simonw 9599 | 2022-04-27T22:18:45Z | 2022-04-27T22:18:45Z | OWNER | Another avenue: https://twitter.com/weargoggles/status/1519426289920270337 > SQLite has its own mutexes to provide thread safety, which as another poster noted are out of play in multi process setups. Perhaps downgrading from the “serializable” to “multi-threaded” safety would be okay for Datasette? https://sqlite.org/c3ref/c_config_covering_index_scan.html#sqliteconfigmultithread Doesn't look like there's an obvious way to access that from Python via the `sqlite3` module though. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111506339 | https://github.com/simonw/sqlite-utils/issues/159#issuecomment-1111506339 | https://api.github.com/repos/simonw/sqlite-utils/issues/159 | IC_kwDOCGYnMM5CQD2j | dracos 154364 | 2022-04-27T21:35:13Z | 2022-04-27T21:35:13Z | NONE | Just stumbled across this, wondering why none of my deletes were working. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .delete_where() does not auto-commit (unlike .insert() or .upsert()) 702386948 | |
| 1111485722 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111485722 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CP-0a | simonw 9599 | 2022-04-27T21:08:20Z | 2022-04-27T21:08:20Z | OWNER | Tried that and it didn't seem to make a difference either. I really need a much deeper view of what's going on here. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111462442 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111462442 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CP5Iq | simonw 9599 | 2022-04-27T20:40:59Z | 2022-04-27T20:42:49Z | OWNER | This looks VERY relevant: [SQLite Shared-Cache Mode](https://www.sqlite.org/sharedcache.html): > SQLite includes a special "shared-cache" mode (disabled by default) intended for use in embedded servers. If shared-cache mode is enabled and a thread establishes multiple connections to the same database, the connections share a single data and schema cache. This can significantly reduce the quantity of memory and IO required by the system. Enabled as part of the URI filename: ATTACH 'file:aux.db?cache=shared' AS aux; Turns out I'm already using this for in-memory databases that have `.memory_name` set, but not (yet) for regular file-backed databases: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L73-L75 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111460068 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111460068 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CP4jk | simonw 9599 | 2022-04-27T20:38:32Z | 2022-04-27T20:38:32Z | OWNER | WAL mode didn't seem to make a difference. I thought there was a chance it might help multiple read connections operate at the same time but it looks like it really does only matter for when writes are going on. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111456500 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111456500 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CP3r0 | simonw 9599 | 2022-04-27T20:36:01Z | 2022-04-27T20:36:01Z | OWNER | Yeah all of this is pretty much assuming read-only connections. Datasette has a separate mechanism for ensuring that writes are executed one at a time against a dedicated connection from an in-memory queue: - https://github.com/simonw/datasette/issues/682 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111451790 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111451790 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CP2iO | glyph 716529 | 2022-04-27T20:30:33Z | 2022-04-27T20:30:33Z | NONE | > I should try seeing what happens with WAL mode enabled. I've only skimmed above but it looks like you're doing mainly read-only queries? WAL mode is about better interactions between writers & readers, primarily. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111448928 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111448928 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CP11g | glyph 716529 | 2022-04-27T20:27:05Z | 2022-04-27T20:27:05Z | NONE | You don't want to re-use an SQLite connection from multiple threads anyway: https://www.sqlite.org/threadsafe.html Multiple connections can operate on the file in parallel, but a single connection can't: > Multi-thread. In this mode, SQLite can be safely used by multiple threads **provided that no single database connection is used simultaneously in two or more threads**. (emphasis mine) | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111442012 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111442012 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CP0Jc | simonw 9599 | 2022-04-27T20:19:00Z | 2022-04-27T20:19:00Z | OWNER | Something worth digging into: are these parallel queries running against the same SQLite connection or are they each rubbing against a separate SQLite connection? Just realized I know the answer: they're running against separate SQLite connections, because that's how the time limit mechanism works: it installs a progress handler for each connection which terminates it after a set time. This means that if SQLite benefits from multiple threads using the same connection (due to shared caches or similar) then Datasette will not be seeing those benefits. It also means that if there's some mechanism within SQLite that penalizes you for having multiple parallel connections to a single file (just guessing here, maybe there's some kind of locking going on?) then Datasette will suffer those penalties. I should try seeing what happens with WAL mode enabled. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111432375 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111432375 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CPxy3 | simonw 9599 | 2022-04-27T20:07:57Z | 2022-04-27T20:07:57Z | OWNER | Also useful: https://avi.im/blag/2021/fast-sqlite-inserts/ - from a tip on Twitter: https://twitter.com/ricardoanderegg/status/1519402047556235264 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111431785 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111431785 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CPxpp | simonw 9599 | 2022-04-27T20:07:16Z | 2022-04-27T20:07:16Z | OWNER | I think I need some much more in-depth tracing tricks for this. https://www.maartenbreddels.com/perf/jupyter/python/tracing/gil/2021/01/14/Tracing-the-Python-GIL.html looks relevant - uses the `perf` tool on Linux. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111408273 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111408273 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CPr6R | simonw 9599 | 2022-04-27T19:40:51Z | 2022-04-27T19:42:17Z | OWNER | Relevant: here's the code that sets up a Datasette SQLite connection: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L73-L96 It's using `check_same_thread=False` - here's [the Python docs on that](https://docs.python.org/3/library/sqlite3.html#sqlite3.connect): > By default, *check_same_thread* is [`True`](https://docs.python.org/3/library/constants.html#True "True") and only the creating thread may use the connection. If set [`False`](https://docs.python.org/3/library/constants.html#False "False"), the returned connection may be shared across multiple threads. When using multiple threads with the same connection writing operations should be serialized by the user to avoid data corruption. This is why Datasette reserves a single connection for write queries and queues them up in memory, [as described here](https://simonwillison.net/2020/Feb/26/weeknotes-datasette-writes/). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111390433 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111390433 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CPnjh | simonw 9599 | 2022-04-27T19:21:02Z | 2022-04-27T19:21:02Z | OWNER | One weird thing: I noticed that in the parallel trace above the SQL query bars are wider. Mousover shows duration in ms, and I got 13ms for this query: select message as value, count(*) as n from ( But in the `?_noparallel=1` version that some query took 2.97ms. Given those numbers though I would expect the overall page time to be MUCH worse for the parallel version - but the page load times are instead very close to each other, with parallel often winning. This is super-weird. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111385875 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111385875 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CPmcT | simonw 9599 | 2022-04-27T19:16:57Z | 2022-04-27T19:16:57Z | OWNER | I just remembered the `--setting num_sql_threads` option... which defaults to 3! https://github.com/simonw/datasette/blob/942411ef946e9a34a2094944d3423cddad27efd3/datasette/app.py#L109-L113 Would explain why the first trace never seems to show more than three SQL queries executing at once. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1111380282 | https://github.com/simonw/datasette/issues/1727#issuecomment-1111380282 | https://api.github.com/repos/simonw/datasette/issues/1727 | IC_kwDOBm6k_c5CPlE6 | simonw 9599 | 2022-04-27T19:10:27Z | 2022-04-27T19:10:27Z | OWNER | Wrote more about that here: https://simonwillison.net/2022/Apr/27/parallel-queries/ Compare https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_facet=committer&_trace=1  With the same thing but with parallel execution disabled: https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_facet=committer&_trace=1&_noparallel=1  Those total page load time numbers are very similar. Is this parallel optimization worthwhile? Maybe it's only worth it on larger databases? Or maybe larger databases perform worse with this? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research: demonstrate if parallel SQL queries are worthwhile 1217759117 | |
| 1110585475 | https://github.com/simonw/datasette/issues/1724#issuecomment-1110585475 | https://api.github.com/repos/simonw/datasette/issues/1724 | IC_kwDOBm6k_c5CMjCD | simonw 9599 | 2022-04-27T06:15:14Z | 2022-04-27T06:15:14Z | OWNER | Yeah, that page is 438K (but only 20K gzipped). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ?_trace=1 doesn't work on Global Power Plants demo 1216619276 | |
| 1110370095 | https://github.com/simonw/datasette/issues/1724#issuecomment-1110370095 | https://api.github.com/repos/simonw/datasette/issues/1724 | IC_kwDOBm6k_c5CLucv | simonw 9599 | 2022-04-27T00:18:30Z | 2022-04-27T00:18:30Z | OWNER | So this isn't a bug here, it's working as intended. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ?_trace=1 doesn't work on Global Power Plants demo 1216619276 | |
| 1110369004 | https://github.com/simonw/datasette/issues/1724#issuecomment-1110369004 | https://api.github.com/repos/simonw/datasette/issues/1724 | IC_kwDOBm6k_c5CLuLs | simonw 9599 | 2022-04-27T00:16:35Z | 2022-04-27T00:17:04Z | OWNER | I bet this is because it's exceeding the size limit: https://github.com/simonw/datasette/blob/da53e0360da4771ffb56a8e3eb3f7476f3168299/datasette/tracer.py#L80-L88 https://github.com/simonw/datasette/blob/da53e0360da4771ffb56a8e3eb3f7476f3168299/datasette/tracer.py#L102-L113 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | ?_trace=1 doesn't work on Global Power Plants demo 1216619276 | |
| 1110330554 | https://github.com/simonw/datasette/issues/1723#issuecomment-1110330554 | https://api.github.com/repos/simonw/datasette/issues/1723 | IC_kwDOBm6k_c5CLky6 | simonw 9599 | 2022-04-26T23:06:20Z | 2022-04-26T23:06:20Z | OWNER | Deployed here: https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_trace=1&_facet=committer | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research running SQL in table view in parallel using `asyncio.gather()` 1216508080 | |
| 1110305790 | https://github.com/simonw/datasette/issues/1723#issuecomment-1110305790 | https://api.github.com/repos/simonw/datasette/issues/1723 | IC_kwDOBm6k_c5CLev- | simonw 9599 | 2022-04-26T22:19:04Z | 2022-04-26T22:19:04Z | OWNER | I realized that seeing the total time in queries wasn't enough to understand this, because if the queries were executed in serial or parallel it should still sum up to the same amount of SQL time (roughly). Instead I need to know how long the page took to render. But that's hard to display on the page since you can't measure it until rendering has finished! So I built an ASGI plugin to handle that measurement: https://github.com/simonw/datasette-total-page-time And with that plugin installed, `http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel2&_facet=other_fuel1&_parallel=1` (the parallel version) takes 377ms: <img width="543" alt="CleanShot 2022-04-26 at 15 17 38@2x" src="https://user-images.githubusercontent.com/9599/165401856-d592ed7a-0240-4514-b9d8-fb9e7d8c9629.png"> While `http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel2&_facet=other_fuel1` (the serial version) takes 762ms: <img width="543" alt="image" src="https://user-images.githubusercontent.com/9599/165401933-6d647014-4cab-4fbd-b9aa-958fc24ff435.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research running SQL in table view in parallel using `asyncio.gather()` 1216508080 | |
| 1110279869 | https://github.com/simonw/datasette/issues/1723#issuecomment-1110279869 | https://api.github.com/repos/simonw/datasette/issues/1723 | IC_kwDOBm6k_c5CLYa9 | simonw 9599 | 2022-04-26T21:45:39Z | 2022-04-26T21:45:39Z | OWNER | Getting some nice traces out of this: <img width="1384" alt="CleanShot 2022-04-26 at 14 45 21@2x" src="https://user-images.githubusercontent.com/9599/165397745-e8bfbe0a-306f-45bd-81f1-f5f6fc6422b9.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research running SQL in table view in parallel using `asyncio.gather()` 1216508080 | |
| 1110278577 | https://github.com/simonw/datasette/issues/1723#issuecomment-1110278577 | https://api.github.com/repos/simonw/datasette/issues/1723 | IC_kwDOBm6k_c5CLYGx | simonw 9599 | 2022-04-26T21:44:04Z | 2022-04-26T21:44:04Z | OWNER | And some simple benchmarks with `ab` - using the `?_parallel=1` hack to try it with and without a parallel `asyncio.gather()`: ``` ~ % ab -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2' This is ApacheBench, Version 2.3 <$Revision: 1879490 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient).....done Server Software: uvicorn Server Hostname: 127.0.0.1 Server Port: 8001 Document Path: /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2 Document Length: 314187 bytes Concurrency Level: 1 Time taken for tests: 68.279 seconds Complete requests: 100 Failed requests: 13 (Connect: 0, Receive: 0, Length: 13, Exceptions: 0) Total transferred: 31454937 bytes HTML transferred: 31418437 bytes Requests per second: 1.46 [#/sec] (mean) Time per request: 682.787 [ms] (mean) Time per request: 682.787 [ms] (mean, across all concurrent requests) Transfer rate: 449.89 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 621 683 68.0 658 993 Waiting: 620 682 68.0 657 992 Total: 621 683 68.0 658 993 Percentage of the requests served within a certain time (ms) 50% 658 66% 678 75% 687 80% 711 90% 763 95% 879 98% 926 99% 993 100% 993 (longest request) ---- In parallel: ~ % ab -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1' This is ApacheBench, Version 2.3 <$Revision: 1879490 $> Copyright 1… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research running SQL in table view in parallel using `asyncio.gather()` 1216508080 | |
| 1110278182 | https://github.com/simonw/datasette/issues/1723#issuecomment-1110278182 | https://api.github.com/repos/simonw/datasette/issues/1723 | IC_kwDOBm6k_c5CLYAm | simonw 9599 | 2022-04-26T21:43:34Z | 2022-04-26T21:43:34Z | OWNER | Here's the diff I'm using: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index d66adb8..f15ef1e 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import itertools import json @@ -5,6 +6,7 @@ import markupsafe from datasette.plugins import pm from datasette.database import QueryInterrupted +from datasette import tracer from datasette.utils import ( await_me_maybe, CustomRow, @@ -150,6 +152,16 @@ class TableView(DataView): default_labels=False, _next=None, _size=None, + ): + with tracer.trace_child_tasks(): + return await self._data_traced(request, default_labels, _next, _size) + + async def _data_traced( + self, + request, + default_labels=False, + _next=None, + _size=None, ): database_route = tilde_decode(request.url_vars["database"]) table_name = tilde_decode(request.url_vars["table"]) @@ -159,6 +171,20 @@ class TableView(DataView): raise NotFound("Database not found: {}".format(database_route)) database_name = db.name + # For performance profiling purposes, ?_parallel=1 turns on asyncio.gather + async def _gather_parallel(*args): + return await asyncio.gather(*args) + + async def _gather_sequential(*args): + results = [] + for fn in args: + results.append(await fn) + return results + + gather = ( + _gather_parallel if request.args.get("_parallel") else _gather_sequential + ) + # If this is a canned query, not a table, then dispatch to QueryView instead canned_query = await self.ds.get_canned_query( database_name, table_name, request.actor @@ -174,8 +200,12 @@ class TableView(DataView): write=bool(canned_query.get("write")), ) - is_view = bool(await db.ge… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research running SQL in table view in parallel using `asyncio.gather()` 1216508080 | |
| 1110265087 | https://github.com/simonw/datasette/issues/1715#issuecomment-1110265087 | https://api.github.com/repos/simonw/datasette/issues/1715 | IC_kwDOBm6k_c5CLUz_ | simonw 9599 | 2022-04-26T21:26:17Z | 2022-04-26T21:26:17Z | OWNER | Running facets and facet suggestions in parallel using `asyncio.gather()` turns out to be a lot less hassle than I had thought - maybe I don't need `asyncinject` for this at all? ```diff if not nofacet: - for facet in facet_instances: - ( - instance_facet_results, - instance_facets_timed_out, - ) = await facet.facet_results() + # Run them in parallel + facet_awaitables = [facet.facet_results() for facet in facet_instances] + facet_awaitable_results = await asyncio.gather(*facet_awaitables) + for ( + instance_facet_results, + instance_facets_timed_out, + ) in facet_awaitable_results: for facet_info in instance_facet_results: base_key = facet_info["name"] key = base_key @@ -522,8 +540,10 @@ class TableView(DataView): and not nofacet and not nosuggest ): - for facet in facet_instances: - suggested_facets.extend(await facet.suggest()) + # Run them in parallel + facet_suggest_awaitables = [facet.suggest() for facet in facet_instances] + for suggest_result in await asyncio.gather(*facet_suggest_awaitables): + suggested_facets.extend(suggest_result) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor TableView to use asyncinject 1212823665 | |
| 1110246593 | https://github.com/simonw/datasette/issues/1715#issuecomment-1110246593 | https://api.github.com/repos/simonw/datasette/issues/1715 | IC_kwDOBm6k_c5CLQTB | simonw 9599 | 2022-04-26T21:03:56Z | 2022-04-26T21:03:56Z | OWNER | Well this is fun... I applied this change: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index d66adb8..85f9e44 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import itertools import json @@ -5,6 +6,7 @@ import markupsafe from datasette.plugins import pm from datasette.database import QueryInterrupted +from datasette import tracer from datasette.utils import ( await_me_maybe, CustomRow, @@ -174,8 +176,11 @@ class TableView(DataView): write=bool(canned_query.get("write")), ) - is_view = bool(await db.get_view_definition(table_name)) - table_exists = bool(await db.table_exists(table_name)) + with tracer.trace_child_tasks(): + is_view, table_exists = map(bool, await asyncio.gather( + db.get_view_definition(table_name), + db.table_exists(table_name) + )) # If table or view not found, return 404 if not is_view and not table_exists: ``` And now using https://datasette.io/plugins/datasette-pretty-traces I get this:  | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor TableView to use asyncinject 1212823665 | |
| 1110239536 | https://github.com/simonw/datasette/issues/1715#issuecomment-1110239536 | https://api.github.com/repos/simonw/datasette/issues/1715 | IC_kwDOBm6k_c5CLOkw | simonw 9599 | 2022-04-26T20:54:53Z | 2022-04-26T20:54:53Z | OWNER | `pytest tests/test_table_*` runs the tests quickly. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor TableView to use asyncinject 1212823665 | |
| 1110238896 | https://github.com/simonw/datasette/issues/1715#issuecomment-1110238896 | https://api.github.com/repos/simonw/datasette/issues/1715 | IC_kwDOBm6k_c5CLOaw | simonw 9599 | 2022-04-26T20:53:59Z | 2022-04-26T20:53:59Z | OWNER | I'm going to rename `database` to `database_name` and `table` to `table_name` to avoid confusion with the `Database` object as opposed to the string name for the database. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor TableView to use asyncinject 1212823665 | |
| 1110229319 | https://github.com/simonw/datasette/issues/1715#issuecomment-1110229319 | https://api.github.com/repos/simonw/datasette/issues/1715 | IC_kwDOBm6k_c5CLMFH | simonw 9599 | 2022-04-26T20:41:32Z | 2022-04-26T20:44:38Z | OWNER | This time I'm not going to bother with the `filter_args` thing - I'm going to just try to use `asyncinject` to execute some big high level things in parallel - facets, suggested facets, counts, the query - and then combine it with the `extras` mechanism I'm trying to introduce too. Most importantly: I want that `extra_template()` function that adds more template context for the HTML to be executed as part of an `asyncinject` flow! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor TableView to use asyncinject 1212823665 | |
| 1110219185 | https://github.com/simonw/datasette/issues/1715#issuecomment-1110219185 | https://api.github.com/repos/simonw/datasette/issues/1715 | IC_kwDOBm6k_c5CLJmx | simonw 9599 | 2022-04-26T20:28:40Z | 2022-04-26T20:56:48Z | OWNER | The refactor I did in #1719 pretty much clashes with all of the changes in https://github.com/simonw/datasette/commit/5053f1ea83194ecb0a5693ad5dada5b25bf0f7e6 so I'll probably need to start my `api-extras` branch again from scratch. Using a new `tableview-asyncinject` branch. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor TableView to use asyncinject 1212823665 | |
| 1110212021 | https://github.com/simonw/datasette/issues/1720#issuecomment-1110212021 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CLH21 | simonw 9599 | 2022-04-26T20:20:27Z | 2022-04-26T20:20:27Z | OWNER | Closing this because I have a good enough idea of the design for now - the details of the parameters can be figured out when I implement this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109309683 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109309683 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHrjz | simonw 9599 | 2022-04-26T04:12:39Z | 2022-04-26T04:12:39Z | OWNER | I think the rough shape of the three plugin hooks is right. The detailed decisions that are needed concern what the parameters should be, which I think will mainly happen as part of: - #1715 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109306070 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109306070 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHqrW | simonw 9599 | 2022-04-26T04:05:20Z | 2022-04-26T04:05:20Z | OWNER | The proposed plugin for annotations - allowing users to attach comments to database tables, columns and rows - would be a great application for all three of those `?_extra=` plugin hooks. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109305184 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109305184 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHqdg | simonw 9599 | 2022-04-26T04:03:35Z | 2022-04-26T04:03:35Z | OWNER | I bet there's all kinds of interesting potential extras that could be calculated by loading the results of the query into a Pandas DataFrame. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109200774 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109200774 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHQ-G | simonw 9599 | 2022-04-26T01:25:43Z | 2022-04-26T01:26:15Z | OWNER | Had a thought: if a custom HTML template is going to make use of stuff generated using these extras, it will need a way to tell Datasette to execute those extras even in the absence of the `?_extra=...` URL parameters. Is that necessary? Or should those kinds of plugins use the existing `extra_template_vars` hook instead? Or maybe the `extra_template_vars` hook gets redesigned so it can depend on other `extras` in some way? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109200335 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109200335 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHQ3P | simonw 9599 | 2022-04-26T01:24:47Z | 2022-04-26T01:24:47Z | OWNER | Sketching out a `?_extra=statistics` table plugin: ```python from datasette import hookimpl @hookimpl def register_table_extras(datasette): return [statistics] async def statistics(datasette, query, columns, sql): # ... need to figure out which columns are integer/floats # then build and execute a SQL query that calculates sum/avg/etc for each column ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109190401 | https://github.com/simonw/sqlite-utils/issues/428#issuecomment-1109190401 | https://api.github.com/repos/simonw/sqlite-utils/issues/428 | IC_kwDOCGYnMM5CHOcB | simonw 9599 | 2022-04-26T01:05:29Z | 2022-04-26T01:05:29Z | OWNER | Django makes extensive use of savepoints for nested transactions: https://docs.djangoproject.com/en/4.0/topics/db/transactions/#savepoints | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research adding support for savepoints 1215216249 | |
| 1109174715 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109174715 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHKm7 | simonw 9599 | 2022-04-26T00:40:13Z | 2022-04-26T00:43:33Z | OWNER | Some of the things I'd like to use `?_extra=` for, that may or not make sense as plugins: - Performance breakdown information, maybe including explain output for a query/table - Information about the tables that were consulted in a query - imagine pulling in additional table metadata - Statistical aggregates against the full set of results. This may well be a Datasette core feature at some point in the future, but being able to provide it early as a plugin would be really cool. - For tables, what are the other tables they can join against? - Suggested facets - Facet results themselves - New custom facets I haven't thought of - though the `register_facet_classes` hook covers that already - Table schema - Table metadata - Analytics - how many times has this table been queried? Would be a plugin thing - For geospatial data, how about a GeoJSON polygon that represents the bounding box for all returned results? Effectively this is an extra aggregation. Looking at https://github-to-sqlite.dogsheep.net/github/commits.json?_labels=on&_shape=objects for inspiration. I think there's a separate potential mechanism in the future that lets you add custom columns to a table. This would affect `.csv` and the HTML presentation too, which makes it a different concept from the `?_extra=` hook that affects the JSON export (and the context that is fed to the HTML templates). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109171871 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109171871 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHJ6f | simonw 9599 | 2022-04-26T00:34:48Z | 2022-04-26T00:34:48Z | OWNER | Let's try sketching out a `register_table_extras` plugin for something new. The first idea I came up with suggests adding new fields to the individual row records that come back - my mental model for extras so far has been that they add new keys to the root object. So if a table result looked like this: ```json { "rows": [ {"id": 1, "name": "Cleo"}, {"id": 2, "name": "Suna"} ], "next_url": null } ``` I was initially thinking that `?_extra=facets` would add a `"facets": {...}` key to that root object. Here's a plugin idea I came up with that would probably justify adding to the individual row objects instead: - `?_extra=check404s` - does an async `HEAD` request against every column value that looks like a URL and checks if it returns a 404 This could also work by adding a `"check404s": {"url-here": 200}` key to the root object though. I think I need some better plugin concepts before committing to this new hook. There's overlap between this and how I want the enrichments mechanism ([see here](https://simonwillison.net/2021/Jan/17/weeknotes-still-pretty-distracted/)) to work. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109165411 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109165411 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHIVj | simonw 9599 | 2022-04-26T00:22:42Z | 2022-04-26T00:22:42Z | OWNER | Passing `pk_values` to the plugin hook feels odd. I think I'd pass a `row` object instead and let the code look up the primary key values on that row (by introspecting the primary keys for the table). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109164803 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109164803 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHIMD | simonw 9599 | 2022-04-26T00:21:40Z | 2022-04-26T00:21:40Z | OWNER | What would the existing https://latest.datasette.io/fixtures/simple_primary_key/1.json?_extras=foreign_key_tables feature look like if it was re-imagined as a `register_row_extras()` plugin? Rough sketch, copying most of the code from https://github.com/simonw/datasette/blob/579f59dcec43a91dd7d404e00b87a00afd8515f2/datasette/views/row.py#L98 ```python from datasette import hookimpl @hookimpl def register_row_extras(datasette): return [foreign_key_tables] async def foreign_key_tables(datasette, database, table, pk_values): if len(pk_values) != 1: return [] db = datasette.get_database(database) all_foreign_keys = await db.get_all_foreign_keys() foreign_keys = all_foreign_keys[table]["incoming"] if len(foreign_keys) == 0: return [] sql = "select " + ", ".join( [ "(select count(*) from {table} where {column}=:id)".format( table=escape_sqlite(fk["other_table"]), column=escape_sqlite(fk["other_column"]), ) for fk in foreign_keys ] ) try: rows = list(await db.execute(sql, {"id": pk_values[0]})) except QueryInterrupted: # Almost certainly hit the timeout return [] foreign_table_counts = dict( zip( [(fk["other_table"], fk["other_column"]) for fk in foreign_keys], list(rows[0]), ) ) foreign_key_tables = [] for fk in foreign_keys: count = ( foreign_table_counts.get((fk["other_table"], fk["other_column"])) or 0 ) key = fk["other_column"] if key.startswith("_"): key += "__exact" link = "{}?{}={}".format( self.ds.urls.table(database, fk["other_table"]), key, ",".join(pk_values), ) foreign_key_tables.append({**fk, **{"count": count, "link": link}}) return foreign_key_tables ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109162123 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109162123 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHHiL | simonw 9599 | 2022-04-26T00:16:42Z | 2022-04-26T00:16:51Z | OWNER | Actually I'm going to imitate the existing `register_*` hooks: - `def register_output_renderer(datasette)` - `def register_facet_classes()` - `def register_routes(datasette)` - `def register_commands(cli)` - `def register_magic_parameters(datasette)` So I'm going to call the new hooks: - `register_table_extras(datasette)` - `register_row_extras(datasette)` - `register_query_extras(datasette)` They'll return a list of `async def` functions. The names of those functions will become the names of the extras. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109160226 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109160226 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHHEi | simonw 9599 | 2022-04-26T00:14:11Z | 2022-04-26T00:14:11Z | OWNER | There are four existing plugin hooks that include the word "extra" but use it to mean something else - to mean additional CSS/JS/variables to be injected into the page: - `def extra_css_urls(...)` - `def extra_js_urls(...)` - `def extra_body_script(...)` - `def extra_template_vars(...)` I think `extra_*` and `*_extras` are different enough that they won't be confused with each other. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109159307 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109159307 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHG2L | simonw 9599 | 2022-04-26T00:12:28Z | 2022-04-26T00:12:28Z | OWNER | I'm going to keep table and row separate. So I think I need to add three new plugin hooks: - `table_extras()` - `row_extras()` - `query_extras()` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1109158903 | https://github.com/simonw/datasette/issues/1720#issuecomment-1109158903 | https://api.github.com/repos/simonw/datasette/issues/1720 | IC_kwDOBm6k_c5CHGv3 | simonw 9599 | 2022-04-26T00:11:42Z | 2022-04-26T00:11:42Z | OWNER | Places this plugin hook (or hooks?) should be able to affect: - JSON for a table/view - JSON for a row - JSON for a canned query - JSON for a custom arbitrary query I'm going to combine those last two, which means there are three places. But maybe I can combine the table one and the row one as well? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Design plugin hook for extras 1215174094 | |
| 1108907238 | https://github.com/simonw/datasette/issues/1719#issuecomment-1108907238 | https://api.github.com/repos/simonw/datasette/issues/1719 | IC_kwDOBm6k_c5CGJTm | simonw 9599 | 2022-04-25T18:34:21Z | 2022-04-25T18:34:21Z | OWNER | Well this refactor turned out to be pretty quick and really does greatly simplify both the `RowView` and `TableView` classes. Very happy with this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor `RowView` and remove `RowTableShared` 1214859703 | |
| 1108890170 | https://github.com/simonw/datasette/issues/262#issuecomment-1108890170 | https://api.github.com/repos/simonw/datasette/issues/262 | IC_kwDOBm6k_c5CGFI6 | simonw 9599 | 2022-04-25T18:17:09Z | 2022-04-25T18:18:39Z | OWNER | I spotted in https://github.com/simonw/datasette/issues/1719#issuecomment-1108888494 that there's actually already an undocumented implementation of `?_extras=foreign_key_tables` - https://latest.datasette.io/fixtures/simple_primary_key/1.json?_extras=foreign_key_tables I added that feature all the way back in November 2017! https://github.com/simonw/datasette/commit/a30c5b220c15360d575e94b0e67f3255e120b916 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add ?_extra= mechanism for requesting extra properties in JSON 323658641 | |
| 1108888494 | https://github.com/simonw/datasette/issues/1719#issuecomment-1108888494 | https://api.github.com/repos/simonw/datasette/issues/1719 | IC_kwDOBm6k_c5CGEuu | simonw 9599 | 2022-04-25T18:15:42Z | 2022-04-25T18:15:42Z | OWNER | Here's an undocumented feature I forgot existed: https://latest.datasette.io/fixtures/simple_primary_key/1.json?_extras=foreign_key_tables `?_extras=foreign_key_tables` https://github.com/simonw/datasette/blob/0bc5186b7bb4fc82392df08f99a9132f84dcb331/datasette/views/table.py#L1021-L1024 It's even covered by the tests: https://github.com/simonw/datasette/blob/b9c2b1cfc8692b9700416db98721fa3ec982f6be/tests/test_api.py#L691-L703 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor `RowView` and remove `RowTableShared` 1214859703 | |
| 1108884171 | https://github.com/simonw/datasette/issues/1719#issuecomment-1108884171 | https://api.github.com/repos/simonw/datasette/issues/1719 | IC_kwDOBm6k_c5CGDrL | simonw 9599 | 2022-04-25T18:10:46Z | 2022-04-25T18:12:45Z | OWNER | It looks like the only class method from that shared class needed by `RowView` is `self.display_columns_and_rows()`. Which I've been wanting to refactor to provide to `QueryView` too: - #715 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor `RowView` and remove `RowTableShared` 1214859703 | |
| 1108877454 | https://github.com/simonw/datasette/issues/1715#issuecomment-1108877454 | https://api.github.com/repos/simonw/datasette/issues/1715 | IC_kwDOBm6k_c5CGCCO | simonw 9599 | 2022-04-25T18:04:27Z | 2022-04-25T18:04:27Z | OWNER | Pushed my WIP on this to the `api-extras` branch: 5053f1ea83194ecb0a5693ad5dada5b25bf0f7e6 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor TableView to use asyncinject 1212823665 | |
| 1108875068 | https://github.com/simonw/datasette/issues/1715#issuecomment-1108875068 | https://api.github.com/repos/simonw/datasette/issues/1715 | IC_kwDOBm6k_c5CGBc8 | simonw 9599 | 2022-04-25T18:03:13Z | 2022-04-25T18:06:33Z | OWNER | The `RowTableShared` class is making this a whole lot more complicated. I'm going to split the `RowView` view out into an entirely separate `views/row.py` module. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor TableView to use asyncinject 1212823665 | |
| 1107873311 | https://github.com/simonw/datasette/issues/1718#issuecomment-1107873311 | https://api.github.com/repos/simonw/datasette/issues/1718 | IC_kwDOBm6k_c5CCM4f | simonw 9599 | 2022-04-24T16:24:14Z | 2022-04-24T16:24:14Z | OWNER | Wrote up what I learned in a TIL: https://til.simonwillison.net/sphinx/blacken-docs | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Code examples in the documentation should be formatted with Black 1213683988 | |
| 1107873271 | https://github.com/simonw/datasette/issues/1718#issuecomment-1107873271 | https://api.github.com/repos/simonw/datasette/issues/1718 | IC_kwDOBm6k_c5CCM33 | simonw 9599 | 2022-04-24T16:23:57Z | 2022-04-24T16:23:57Z | OWNER | Turns out I didn't need that `git diff-index` trick after all - the `blacken-docs` command returns a non-zero exit code if it changes any files. Submitted a documentation PR to that project instead: - https://github.com/asottile/blacken-docs/pull/162 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Code examples in the documentation should be formatted with Black 1213683988 | |
| 1107870788 | https://github.com/simonw/datasette/issues/1718#issuecomment-1107870788 | https://api.github.com/repos/simonw/datasette/issues/1718 | IC_kwDOBm6k_c5CCMRE | simonw 9599 | 2022-04-24T16:09:23Z | 2022-04-24T16:09:23Z | OWNER | One more attempt at testing the `git diff-index` trick. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Code examples in the documentation should be formatted with Black 1213683988 | |
| 1107869884 | https://github.com/simonw/datasette/issues/1718#issuecomment-1107869884 | https://api.github.com/repos/simonw/datasette/issues/1718 | IC_kwDOBm6k_c5CCMC8 | simonw 9599 | 2022-04-24T16:04:03Z | 2022-04-24T16:04:03Z | OWNER | OK, I'm expecting this one to fail at the `git diff-index --quiet HEAD --` check. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Code examples in the documentation should be formatted with Black 1213683988 | |
| 1107869556 | https://github.com/simonw/datasette/issues/1718#issuecomment-1107869556 | https://api.github.com/repos/simonw/datasette/issues/1718 | IC_kwDOBm6k_c5CCL90 | simonw 9599 | 2022-04-24T16:02:27Z | 2022-04-24T16:02:27Z | OWNER | Looking at that first error it appears to be a place where I had deliberately omitted the body of the function: https://github.com/simonw/datasette/blob/36573638b0948174ae237d62e6369b7d55220d7f/docs/internals.rst#L196-L211 I can use `...` as the function body here to get it to pass. Fixing those warnings actually helped me spot a couple of bugs, so I'm glad this happened. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Code examples in the documentation should be formatted with Black 1213683988 | |
| 1107868585 | https://github.com/simonw/datasette/issues/1718#issuecomment-1107868585 | https://api.github.com/repos/simonw/datasette/issues/1718 | IC_kwDOBm6k_c5CCLup | simonw 9599 | 2022-04-24T15:57:10Z | 2022-04-24T15:57:19Z | OWNER | The tests failed there because of what I thought were warnings but turn out to be treated as errors: ``` % blacken-docs -l 60 docs/*.rst docs/internals.rst:196: code block parse error Cannot parse: 14:0: <line number missing in source> docs/json_api.rst:449: code block parse error Cannot parse: 1:0: <link rel="alternate" docs/plugin_hooks.rst:250: code block parse error Cannot parse: 6:4: ] docs/plugin_hooks.rst:311: code block parse error Cannot parse: 38:0: <line number missing in source> docs/testing_plugins.rst:135: code block parse error Cannot parse: 5:0: <line number missing in source> % echo $? 1 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Code examples in the documentation should be formatted with Black 1213683988 | |
| 1107867281 | https://github.com/simonw/datasette/issues/1718#issuecomment-1107867281 | https://api.github.com/repos/simonw/datasette/issues/1718 | IC_kwDOBm6k_c5CCLaR | simonw 9599 | 2022-04-24T15:49:23Z | 2022-04-24T15:49:23Z | OWNER | I'm going to push the first commit with a deliberate missing formatting to check that the tests fail. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Code examples in the documentation should be formatted with Black 1213683988 | |
| 1107866013 | https://github.com/simonw/datasette/issues/1718#issuecomment-1107866013 | https://api.github.com/repos/simonw/datasette/issues/1718 | IC_kwDOBm6k_c5CCLGd | simonw 9599 | 2022-04-24T15:42:07Z | 2022-04-24T15:42:07Z | OWNER | In the absence of `--check` I can use this to detect if changes are applied: ```zsh % git diff-index --quiet HEAD -- % echo $? 0 % blacken-docs -l 60 docs/*.rst docs/authentication.rst: Rewriting... ... % git diff-index --quiet HEAD -- % echo $? 1 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Code examples in the documentation should be formatted with Black 1213683988 | |
| 1107865493 | https://github.com/simonw/datasette/issues/1718#issuecomment-1107865493 | https://api.github.com/repos/simonw/datasette/issues/1718 | IC_kwDOBm6k_c5CCK-V | simonw 9599 | 2022-04-24T15:39:02Z | 2022-04-24T15:39:02Z | OWNER | There's no `blacken-docs --check` option so I filed a feature request: - https://github.com/asottile/blacken-docs/issues/161 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Code examples in the documentation should be formatted with Black 1213683988 | |
| 1107863924 | https://github.com/simonw/datasette/issues/1718#issuecomment-1107863924 | https://api.github.com/repos/simonw/datasette/issues/1718 | IC_kwDOBm6k_c5CCKl0 | simonw 9599 | 2022-04-24T15:30:03Z | 2022-04-24T15:30:03Z | OWNER | On the one hand, I'm not crazy about some of the indentation decisions Black made here - in particular this one, which I had indented deliberately for readability: ```diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..8008023 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,7 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so: .. code-block:: python response = Response.redirect("/") - response.set_cookie("ds_actor", datasette.sign({ - "a": { - "id": "cleopaws" - } - }, "actor")) + response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor")) ``` But... consistency is a virtue. Maybe I'm OK with just this one disagreement? Also: I've been mentally trying to keep the line lengths a bit shorter to help them be more readable on mobile devices. I'll try a different line length using `blacken-docs -l 60 docs/*.rst` instead. I like this more - here's the result for that example: ```diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..2496073 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,10 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so: .. code-block:: python response = Response.redirect("/") - response.set_cookie("ds_actor", datasette.sign({ - "a": { - "id": "cleopaws" - } - }, "actor")) + response.set_cookie( + "ds_actor", + datasette.sign({"a": {"id": "cleopaws"}}, "actor"), + ) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Code examples in the documentation should be formatted with Black 1213683988 | |
| 1107863365 | https://github.com/simonw/datasette/issues/1718#issuecomment-1107863365 | https://api.github.com/repos/simonw/datasette/issues/1718 | IC_kwDOBm6k_c5CCKdF | simonw 9599 | 2022-04-24T15:26:41Z | 2022-04-24T15:26:41Z | OWNER | Tried this: ``` pip install blacken-docs blacken-docs docs/*.rst git diff | pbcopy ``` Got this: ```diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..8008023 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,7 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so: .. code-block:: python response = Response.redirect("/") - response.set_cookie("ds_actor", datasette.sign({ - "a": { - "id": "cleopaws" - } - }, "actor")) + response.set_cookie("ds_actor", datasette.sign({"a": {"id": "cleopaws"}}, "actor")) Note that you need to pass ``"actor"`` as the namespace to :ref:`datasette_sign`. @@ -412,12 +408,16 @@ To include an expiry, add a ``"e"`` key to the cookie value containing a `base62 expires_at = int(time.time()) + (24 * 60 * 60) response = Response.redirect("/") - response.set_cookie("ds_actor", datasette.sign({ - "a": { - "id": "cleopaws" - }, - "e": baseconv.base62.encode(expires_at), - }, "actor")) + response.set_cookie( + "ds_actor", + datasette.sign( + { + "a": {"id": "cleopaws"}, + "e": baseconv.base62.encode(expires_at), + }, + "actor", + ), + ) The resulting cookie will encode data that looks something like this: diff --git a/docs/spatialite.rst b/docs/spatialite.rst index d1b300b..556bad8 100644 --- a/docs/spatialite.rst +++ b/docs/spatialite.rst @@ -58,19 +58,22 @@ Here's a recipe for taking a table with existing latitude and longitude columns, .. code-block:: python import sqlite3 - conn = sqlite3.connect('museums.db') + + conn = sqlite3.connect("museums.db") # Lead the spatialite extension: conn.enable_load_extension(True) - conn.load_extension('/usr/local/lib/mod_spatialite.dylib') + conn.load_extension("/usr/local/lib/mod_spatial… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Code examples in the documentation should be formatted with Black 1213683988 | |
| 1107862882 | https://github.com/simonw/datasette/issues/1718#issuecomment-1107862882 | https://api.github.com/repos/simonw/datasette/issues/1718 | IC_kwDOBm6k_c5CCKVi | simonw 9599 | 2022-04-24T15:23:56Z | 2022-04-24T15:23:56Z | OWNER | Found https://github.com/asottile/blacken-docs via - https://github.com/psf/black/issues/294 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Code examples in the documentation should be formatted with Black 1213683988 | |
| 1107848097 | https://github.com/simonw/datasette/pull/1717#issuecomment-1107848097 | https://api.github.com/repos/simonw/datasette/issues/1717 | IC_kwDOBm6k_c5CCGuh | simonw 9599 | 2022-04-24T14:02:37Z | 2022-04-24T14:02:37Z | OWNER | This is a neat feature, thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add timeout option to Cloudrun build 1213281044 | |
| 1107459446 | https://github.com/simonw/datasette/pull/1717#issuecomment-1107459446 | https://api.github.com/repos/simonw/datasette/issues/1717 | IC_kwDOBm6k_c5CAn12 | codecov[bot] 22429695 | 2022-04-23T11:56:36Z | 2022-04-23T11:56:36Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1717?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1717](https://codecov.io/gh/simonw/datasette/pull/1717?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9b9a314) into [main](https://codecov.io/gh/simonw/datasette/commit/d57c347f35bcd8cff15f913da851b4b8eb030867?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (d57c347) will **increase** coverage by `0.00%`. > The diff coverage is `100.00%`. ```diff @@ Coverage Diff @@ ## main #1717 +/- ## ======================================= Coverage 91.75% 91.75% ======================================= Files 34 34 Lines 4574 4575 +1 ======================================= + Hits 4197 4198 +1 Misses 377 377 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1717?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/publish/cloudrun.py](https://codecov.io/gh/simonw/datasette/pull/1717/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3B1Ymxpc2gvY2xvdWRydW4ucHk=) | `97.05% <100.00%> (+0.04%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1717?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add timeout option to Cloudrun build 1213281044 | |
| 1106989581 | https://github.com/simonw/datasette/issues/1715#issuecomment-1106989581 | https://api.github.com/repos/simonw/datasette/issues/1715 | IC_kwDOBm6k_c5B-1IN | simonw 9599 | 2022-04-22T23:03:29Z | 2022-04-22T23:03:29Z | OWNER | I'm having second thoughts about injecting `request` - might be better to have the view function pull the relevant pieces out of the request before triggering the rest of the resolution. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor TableView to use asyncinject 1212823665 | |
| 1106947168 | https://github.com/simonw/datasette/issues/1715#issuecomment-1106947168 | https://api.github.com/repos/simonw/datasette/issues/1715 | IC_kwDOBm6k_c5B-qxg | simonw 9599 | 2022-04-22T22:25:57Z | 2022-04-22T22:26:06Z | OWNER | ```python async def database(request: Request, datasette: Datasette) -> Database: database_route = tilde_decode(request.url_vars["database"]) try: return datasette.get_database(route=database_route) except KeyError: raise NotFound("Database not found: {}".format(database_route)) async def table_name(request: Request) -> str: return tilde_decode(request.url_vars["table"]) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Refactor TableView to use asyncinject 1212823665 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] (
[html_url] TEXT,
[issue_url] TEXT,
[id] INTEGER PRIMARY KEY,
[node_id] TEXT,
[user] INTEGER REFERENCES [users]([id]),
[created_at] TEXT,
[updated_at] TEXT,
[author_association] TEXT,
[body] TEXT,
[reactions] TEXT,
[issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
ON [issue_comments] ([user]);
author_association 4 ✖