html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/datasette/issues/1423#issuecomment-894454087,https://api.github.com/repos/simonw/datasette/issues/1423,894454087,IC_kwDOBm6k_c41UElH,9599,simonw,2021-08-06T18:51:42Z,2021-08-06T18:51:42Z,OWNER,"The invisible tooltip could say ""Showing 30 items, more available"" (helping save you from counting up to 20 if you know about the secret feature). The numbers could then be fully displayed on the ""..."" page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1423#issuecomment-894454644,https://api.github.com/repos/simonw/datasette/issues/1423,894454644,IC_kwDOBm6k_c41UEt0,9599,simonw,2021-08-06T18:52:49Z,2021-08-06T18:52:49Z,OWNER,"This means that the counts would be unavailable to users who cannot see tooltips (e.g. mobile users) on pages that did not have any facets that broke the 30 limit and hence displayed that ""..."" link. I think I'm OK with that, for the moment. May revisit in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1422#issuecomment-894589140,https://api.github.com/repos/simonw/datasette/issues/1422,894589140,IC_kwDOBm6k_c41UljU,9599,simonw,2021-08-07T01:58:16Z,2021-08-07T01:58:24Z,OWNER,Also need to consider this hidden field - it should pass the `_hide_sql` or `_show_sql` parameters depending on the same logic: https://github.com/simonw/datasette/blob/acc22436622ff8476c30acf45ed60f54b4aaa5d9/datasette/templates/query.html#L47-L49,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query, https://github.com/simonw/datasette/issues/1421#issuecomment-894606843,https://api.github.com/repos/simonw/datasette/issues/1421,894606843,IC_kwDOBm6k_c41Up37,9599,simonw,2021-08-07T05:17:12Z,2021-08-07T05:17:12Z,OWNER,Marking this blocked because I don't have a way around the needing-a-SQLite-SQL-parser problem at the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894606796,https://api.github.com/repos/simonw/datasette/issues/1421,894606796,IC_kwDOBm6k_c41Up3M,9599,simonw,2021-08-07T05:16:39Z,2021-08-07T05:16:39Z,OWNER,"Urgh, yeah I've seen this one before. Fixing it pretty much requires writing a full SQLite SQL syntax parser in Python, which is frustratingly complicated for solving this issue! You can work around this for a canned query by using the optional `params:` argument documented here: https://docs.datasette.io/en/stable/sql_queries.html#canned-query-parameters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1422#issuecomment-894607989,https://api.github.com/repos/simonw/datasette/issues/1422,894607989,IC_kwDOBm6k_c41UqJ1,9599,simonw,2021-08-07T05:31:57Z,2021-08-07T05:31:57Z,OWNER,"Demo: https://latest.datasette.io/fixtures/neighborhood_search Documentation: https://docs.datasette.io/en/latest/sql_queries.html#additional-canned-query-options","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query, https://github.com/simonw/datasette/issues/1421#issuecomment-894922145,https://api.github.com/repos/simonw/datasette/issues/1421,894922145,IC_kwDOBm6k_c41V22h,9599,simonw,2021-08-09T03:07:38Z,2021-08-09T03:07:38Z,OWNER,"I hoped this would work: ```sql with foo as ( explain select * from facetable where state = :state and on_earth = :on_earth and neighborhood not like '00:04' ) select p4 from foo where opcode = 'Variable' ``` But sadly [it returns an error](https://latest.datasette.io/fixtures?sql=with+foo+as+%28%0D%0A++explain+select+*+from+facetable%0D%0A++where+state+%3D+%3Astate%0D%0A++and+on_earth+%3D+%3Aon_earth%0D%0A++and+neighborhood+not+like+%2700%3A04%27%0D%0A%29%0D%0Aselect+p4+from+foo+where+opcode+%3D+%27Variable%27&state=&on_earth=&04=): > near ""explain"": syntax error","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894922703,https://api.github.com/repos/simonw/datasette/issues/1421,894922703,IC_kwDOBm6k_c41V2_P,9599,simonw,2021-08-09T03:09:29Z,2021-08-09T03:09:29Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/ad90a72afa21b737b162e2bbdddc301a97d575cd/datasette/views/database.py#L225-L231,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894921512,https://api.github.com/repos/simonw/datasette/issues/1421,894921512,IC_kwDOBm6k_c41V2so,9599,simonw,2021-08-09T03:05:26Z,2021-08-09T03:05:26Z,OWNER,"I may have a way to work around this, using `explain`. Consider this query: ```sql select * from facetable where state = :state and on_earth = :on_earth and neighborhood not like '00:04' ``` Datasette currently gets confused and shows three form fields: https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04= But... if I run `explain` [against that](https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04=) I get this (truncated): addr | opcode | p1 | p2 | p3 | p4 | p5 | comment -- | -- | -- | -- | -- | -- | -- | -- 20 | ResultRow | 6 | 10 | 0 |   | 0 |   21 | Next | 0 | 3 | 0 |   | 1 |   22 | Halt | 0 | 0 | 0 |   | 0 |   23 | Transaction | 0 | 0 | 35 | 0 | 1 |   24 | Variable | 1 | 2 | 0 | :state | 0 |   25 | Variable | 2 | 3 | 0 | :on_earth | 0 |   26 | String8 | 0 | 4 | 0 | 00:04 | 0 |   27 | Goto | 0 | 1 | 0 |   | 0 |   Could it be as simple as pulling out those `Variable` rows to figure out the names of the variables in the query?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894925437,https://api.github.com/repos/simonw/datasette/issues/1421,894925437,IC_kwDOBm6k_c41V3p9,9599,simonw,2021-08-09T03:19:00Z,2021-08-09T03:19:00Z,OWNER,"This may not work: > `ERROR: sql = 'explain select 1 + :one + :two', params = None: You did not supply a value for binding 1.` The `explain` queries themselves want me to pass them parameters. I could try using the regex to pull out candidates and passing `None` for each of those, including incorrect ones like `:31`. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894925914,https://api.github.com/repos/simonw/datasette/issues/1421,894925914,IC_kwDOBm6k_c41V3xa,9599,simonw,2021-08-09T03:20:42Z,2021-08-09T03:20:42Z,OWNER,"I think this works! ```python _re_named_parameter = re.compile("":([a-zA-Z0-9_]+)"") async def derive_named_parameters(db, sql): explain = 'explain {}'.format(sql.strip().rstrip("";"")) possible_params = _re_named_parameter.findall(sql) try: results = await db.execute(explain, {p: None for p in possible_params}) return [row[""p4""].lstrip("":"") for row in results if row[""opcode""] == ""Variable""] except sqlite3.DatabaseError: return [] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894927185,https://api.github.com/repos/simonw/datasette/issues/1421,894927185,IC_kwDOBm6k_c41V4FR,9599,simonw,2021-08-09T03:25:01Z,2021-08-09T03:25:01Z,OWNER,"One catch with this approach: if the SQL query is invalid, the parameters will not be extracted and shown as form fields. Maybe that's completely fine? Why display a form if it's going to break when the user actually runs the query? But it does bother me. I worry that someone who is still iterating on and editing their query before actually starting to use it might find the behaviour confusing. So maybe if the query raises an exception it could fall back on the regular expression results?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894929080,https://api.github.com/repos/simonw/datasette/issues/1421,894929080,IC_kwDOBm6k_c41V4i4,9599,simonw,2021-08-09T03:33:02Z,2021-08-09T03:33:02Z,OWNER,"Fixed! Fantastic, this one has been bothering me for *years*. https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894929769,https://api.github.com/repos/simonw/datasette/issues/1421,894929769,IC_kwDOBm6k_c41V4tp,9599,simonw,2021-08-09T03:36:49Z,2021-08-09T03:36:49Z,OWNER,"SQLite carries a warning about using `EXPLAIN` like this: https://www.sqlite.org/lang_explain.html > The output from EXPLAIN and EXPLAIN QUERY PLAN is intended for interactive analysis and troubleshooting only. The details of the output format are subject to change from one release of SQLite to the next. Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented. I think that's OK here, because of the regular expression fallback. If the format changes in the future in a way that breaks the query the error should be caught and the regex-captured parameters should be returned instead. Hmmm... actually that's not entirely true: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L1084-L1091 If the format changes such that the same columns are returned but the `[row[""p4""].lstrip("":"") for row in results if row[""opcode""] == ""Variable""]` list comprehension returns an empty array it will break Datasette! I'm going to take that risk for the moment, but I'll actively watch out for problems in the future. If this does turn out to be bad I can always go back to the pure regular expression mechanism. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1421#issuecomment-894930013,https://api.github.com/repos/simonw/datasette/issues/1421,894930013,IC_kwDOBm6k_c41V4xd,9599,simonw,2021-08-09T03:38:06Z,2021-08-09T03:38:06Z,OWNER,"Amusing edge-case: if you run this against a `explain ...` query it falls back to using regular expressions, because `explain explain select ...` is invalid SQL. https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times", https://github.com/simonw/datasette/issues/1425#issuecomment-894865323,https://api.github.com/repos/simonw/datasette/issues/1425,894865323,IC_kwDOBm6k_c41Vo-r,9599,simonw,2021-08-08T22:33:19Z,2021-08-08T22:33:19Z,OWNER,"I can do this with the `await_me_maybe()` function, as seen here: https://github.com/simonw/datasette/blob/a21853c9dade240734abc6b4f750fae09a3e840a/datasette/app.py#L864-L873","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1424#issuecomment-894864682,https://api.github.com/repos/simonw/datasette/issues/1424,894864682,IC_kwDOBm6k_c41Vo0q,9599,simonw,2021-08-08T22:26:46Z,2021-08-08T22:26:46Z,OWNER,"Note that the `sqlite3` exceptions are in `sqlite3` if using the Python standard library but are in `pysqlite3` if that module is being used instead. So maybe encourage people to use them from `datasette.sqlite.sqlite3` instead, which will point to the correct package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends, https://github.com/simonw/datasette/issues/1424#issuecomment-894864744,https://api.github.com/repos/simonw/datasette/issues/1424,894864744,IC_kwDOBm6k_c41Vo1o,9599,simonw,2021-08-08T22:27:31Z,2021-08-08T22:27:31Z,OWNER,https://docs.python.org/3/library/sqlite3.html#exceptions is useful - it looks like `sqlite3.DatabaseError` is the super-class of all of the other exceptions that we might see.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends, https://github.com/simonw/datasette/issues/1424#issuecomment-894864404,https://api.github.com/repos/simonw/datasette/issues/1424,894864404,IC_kwDOBm6k_c41VowU,9599,simonw,2021-08-08T22:24:06Z,2021-08-08T22:24:06Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/de5ce2e56339ad8966f417a4758f7c210c017dec/datasette/database.py#L176-L200,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends, https://github.com/simonw/datasette/issues/1424#issuecomment-894864616,https://api.github.com/repos/simonw/datasette/issues/1424,894864616,IC_kwDOBm6k_c41Vozo,9599,simonw,2021-08-08T22:26:08Z,2021-08-08T22:26:08Z,OWNER,"- `datasette.database.QueryInterrupted` for queries that were interrupted - `sqlite3.OperationalError` - `sqlite3.DatabaseError` and more","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends, https://github.com/simonw/datasette/issues/1425#issuecomment-894869692,https://api.github.com/repos/simonw/datasette/issues/1425,894869692,IC_kwDOBm6k_c41VqC8,9599,simonw,2021-08-08T23:08:29Z,2021-08-08T23:08:29Z,OWNER,Updated documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-value-column-table-database-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894881448,https://api.github.com/repos/simonw/datasette/issues/1425,894881448,IC_kwDOBm6k_c41Vs6o,9599,simonw,2021-08-09T00:24:25Z,2021-08-09T00:24:39Z,OWNER,"My hunch is that the ""skip this `render_cell()` result if it returns `None`"" logic isn't working correctly, ever since I added the `await_me_maybe` line. Could that be because Pluggy handles the ""do the next if `None` is returned"" logic itself, but I'm no-longer returning `None`, I'm returning an awaitable which when awaited returns `None`. This would suggest that all of the `await_me_maybe()` plugin hooks have the same bug. That's definitely possible - it may well be that no-one has yet stumbled across a bug caused by a plugin returning an awaitable and hence not being skipped, because plugin hooks that return awaitable are rare enough that no-one has tried two plugins which both use that trick. Still don't see why it would pass on my laptop but fail in CI though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894881016,https://api.github.com/repos/simonw/datasette/issues/1425,894881016,IC_kwDOBm6k_c41Vsz4,9599,simonw,2021-08-09T00:21:53Z,2021-08-09T00:21:53Z,OWNER,"Still one test failure: ``` def test_hook_render_cell_link_from_json(app_client): sql = """""" select '{""href"": ""http://example.com/"", ""label"":""Example""}' """""".strip() path = ""/fixtures?"" + urllib.parse.urlencode({""sql"": sql}) response = app_client.get(path) td = Soup(response.body, ""html.parser"").find(""table"").find(""tbody"").find(""td"") a = td.find(""a"") > assert a is not None, str(a) E AssertionError: None E assert None is not None ``` The weird thing about this one is that I can't replicate it on my laptop - but it happens in CI every time, including when I shell in and try to run that single test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894882123,https://api.github.com/repos/simonw/datasette/issues/1425,894882123,IC_kwDOBm6k_c41VtFL,9599,simonw,2021-08-09T00:27:43Z,2021-08-09T00:27:43Z,OWNER,"Good news: `render_cell()` is the only hook to use `firstresult=True`: https://github.com/simonw/datasette/blob/f3c9edb376a13c09b5ecf97c7390f4e49efaadf2/datasette/hookspecs.py#L62-L64 https://pluggy.readthedocs.io/en/latest/#first-result-only","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894882642,https://api.github.com/repos/simonw/datasette/issues/1425,894882642,IC_kwDOBm6k_c41VtNS,9599,simonw,2021-08-09T00:29:57Z,2021-08-09T00:29:57Z,OWNER,"Here's the code in `pluggy` that implements this: https://github.com/pytest-dev/pluggy/blob/0a064fe275060dbdb1fe6e10c888e72bc400fb33/src/pluggy/callers.py#L31-L43 ```python if hook_impl.hookwrapper: try: gen = hook_impl.function(*args) next(gen) # first yield teardowns.append(gen) except StopIteration: _raise_wrapfail(gen, ""did not yield"") else: res = hook_impl.function(*args) if res is not None: results.append(res) if firstresult: # halt further impl calls break ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894883664,https://api.github.com/repos/simonw/datasette/issues/1425,894883664,IC_kwDOBm6k_c41VtdQ,9599,simonw,2021-08-09T00:33:56Z,2021-08-09T00:33:56Z,OWNER,"I could extract that code out and write my own function which implements the equivalent of calling `pm.hook.render_cell(...)` but runs `await_me_maybe()` before checking if `res is not None`. That's pretty nasty. Could I instead call the plugin hook normally, but then have additional logic which says ""if I await it and it returns `None` then try calling the hook again but skip this one"" - not sure if there's a way to do that either. I could remove the `firstresult=True` from the hookspec - which would cause it to call and return ALL hooks - but then in my own code use only the first one. This is slightly less efficient (since it calls all the hooks and then discards all-but-one value) but it's the least unpleasant in terms of the code I would have to write - plus I don't think it's going to be THAT common for someone to have multiple expensive `render_cell()` hooks installed at once (they are usually pretty cheap).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894884874,https://api.github.com/repos/simonw/datasette/issues/1425,894884874,IC_kwDOBm6k_c41VtwK,9599,simonw,2021-08-09T00:38:20Z,2021-08-09T00:38:20Z,OWNER,I'm trying the version where I remove `firstresult=True`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894893319,https://api.github.com/repos/simonw/datasette/issues/1425,894893319,IC_kwDOBm6k_c41Vv0H,9599,simonw,2021-08-09T01:08:56Z,2021-08-09T01:09:12Z,OWNER,Demo: https://latest.datasette.io/fixtures/simple_primary_key shows `RENDER_CELL_ASYNC_RESULT` where the CSV version shows `RENDER_CELL_ASYNC`: https://latest.datasette.io/fixtures/simple_primary_key.csv - because of this test plugin code: https://github.com/simonw/datasette/blob/a390bdf9cef01d8723d025fc3348e81345ff4856/tests/plugins/my_plugin.py#L98-L122,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-894900267,https://api.github.com/repos/simonw/datasette/issues/1425,894900267,IC_kwDOBm6k_c41Vxgr,9599,simonw,2021-08-09T01:31:22Z,2021-08-09T01:31:22Z,OWNER,"I used this to build a new plugin: https://github.com/simonw/datasette-query-links Demo here: https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++%27select+*+from+[facetable]%27+as+query%0D%0Aunion%0D%0Aselect%0D%0A++%27select+sqlite_version()%27%0D%0Aunion%0D%0Aselect%0D%0A++%27select+this+is+invalid+SQL+so+will+not+be+linked%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1425#issuecomment-895003796,https://api.github.com/repos/simonw/datasette/issues/1425,895003796,IC_kwDOBm6k_c41WKyU,3243482,abdusco,2021-08-09T07:14:35Z,2021-08-09T07:14:35Z,CONTRIBUTOR,"I believe this also provides a workaround for the problem I face in https://github.com/simonw/datasette/issues/1300. Now I should be able to get table PKs and generate a row URL. I'll test this out and report my findings. ```py from datasette.utils import path_from_row_pks pks = await db.primary_keys(table) url = self.ds.urls.row_blob( database, table, path_from_row_pks(row, pks, not pks), column, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable, https://github.com/simonw/datasette/issues/1426#issuecomment-895500565,https://api.github.com/repos/simonw/datasette/issues/1426,895500565,IC_kwDOBm6k_c41YEEV,9599,simonw,2021-08-09T20:00:04Z,2021-08-09T20:00:04Z,OWNER,"A few options for how this would work: - `datasette ... --robots allow` - `datasette ... --setting robots allow` Options could be: - `allow` - allow all crawling - `deny` - deny all crawling - `limited` - allow access to the homepage and the index pages for each database and each table, but disallow crawling any further than that The ""limited"" mode is particularly interesting. Could even make it the default, but I think that may be a bit too confusing. Idea would be to get the key pages indexed but use `nofollow` to discourage crawlers from indexing individual row pages or deep pages like `https://datasette.io/content/repos?_facet=owner&_facet=language&_facet_array=topics&topics__arraycontains=sqlite#facet-owner`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-895509536,https://api.github.com/repos/simonw/datasette/issues/1426,895509536,IC_kwDOBm6k_c41YGQg,9599,simonw,2021-08-09T20:12:57Z,2021-08-09T20:12:57Z,OWNER,I could try out the `X-Robots` HTTP header too: https://developers.google.com/search/docs/advanced/robots/robots_meta_tag#xrobotstag,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-895510773,https://api.github.com/repos/simonw/datasette/issues/1426,895510773,IC_kwDOBm6k_c41YGj1,9599,simonw,2021-08-09T20:14:50Z,2021-08-09T20:19:22Z,OWNER,"https://twitter.com/mal/status/1424825895139876870 > True pinging google should be part of the build process on a static site :) That's another aspect of this: if you DO want your site crawled, teaching the `datasette publish` command how to ping Google when a deploy has gone out could be a nice improvement. Annoyingly it looks like you need to configure an auth token of some sort in order to use their API though, which is likely too much hassle to be worth building into Datasette itself: https://developers.google.com/search/apis/indexing-api/v3/using-api ``` curl -X POST https://indexing.googleapis.com/v3/urlNotifications:publish -d '{ ""url"": ""https://careers.google.com/jobs/google/technical-writer"", ""type"": ""URL_UPDATED"" }' -H ""Content-Type: application/json"" { ""error"": { ""code"": 401, ""message"": ""Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project."", ""status"": ""UNAUTHENTICATED"" } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-895522818,https://api.github.com/repos/simonw/datasette/issues/1426,895522818,IC_kwDOBm6k_c41YJgC,9599,simonw,2021-08-09T20:34:10Z,2021-08-09T20:34:10Z,OWNER,At the very least Datasette should serve a blank `/robots.txt` by default - I'm seeing a ton of 404s for it in the logs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/859#issuecomment-905899177,https://api.github.com/repos/simonw/datasette/issues/859,905899177,IC_kwDOBm6k_c41_uyp,2670795,brandonrobertz,2021-08-25T21:48:00Z,2021-08-25T21:48:00Z,CONTRIBUTOR,"Upon first stab, there's two issues here: - DB/table/row counts (as discussed above). This isn't too bad if the DBs are actually above the MAX limit check. - Populating the internal DB. On first load of a giant set of DBs, it can take 10-20 mins to populate. By altering datasette and persisting the internal DB to disk, this problem is vastly improved, but I'm sure this will cause problems elsewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts), https://github.com/simonw/datasette/issues/859#issuecomment-905900807,https://api.github.com/repos/simonw/datasette/issues/859,905900807,IC_kwDOBm6k_c41_vMH,9599,simonw,2021-08-25T21:51:10Z,2021-08-25T21:51:10Z,OWNER,"10-20 minutes to populate `_internal`! How many databases and tables is that for? I may have to rethink the `_internal` mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed. I need to get my own stress testing rig setup for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts), https://github.com/simonw/datasette/issues/859#issuecomment-905904540,https://api.github.com/repos/simonw/datasette/issues/859,905904540,IC_kwDOBm6k_c41_wGc,2670795,brandonrobertz,2021-08-25T21:59:14Z,2021-08-25T21:59:55Z,CONTRIBUTOR,"I did two tests: one with 1000 5-30mb DBs and a second with 20 multi gig DBs. For the second, I created them like so: `for i in {1..20}; do sqlite-generate db$i.db --tables ${i}00 --rows 100,2000 --columns 5,100 --pks 0 --fks 0; done` This was for deciding whether to use lots of small DBs or to group things into a smaller number of bigger DBs. The second strategy wins. By simply persisting the `_internal` DB to disk, I was able to avoid most of the performance issues I was experiencing previously. (To do this, I changed the `datasette/internal_db.py:init_internal_db` creates to if not exists, and changed the `_internal` DB instantiation in `datasette/app.py:Datasette.__init__` to a path with `is_mutable=True`.) Super rough, but the pages now load so I can continue testing ideas.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts), https://github.com/simonw/datasette/issues/1293#issuecomment-898063815,https://api.github.com/repos/simonw/datasette/issues/1293,898063815,IC_kwDOBm6k_c41h13H,9599,simonw,2021-08-13T00:33:17Z,2021-08-13T00:33:17Z,OWNER,"Improved version of that function: ```python def columns_for_query(conn, sql): """""" Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of ``(table_name, column_name)`` pairs, one per returned column. ``(None, None)`` if no table and column could be derived. """""" rows = conn.execute('explain ' + sql).fetchall() table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'} names_by_rootpage = dict( conn.execute( 'select rootpage, name from sqlite_master where rootpage in ({})'.format( ', '.join(map(str, table_rootpage_by_register.values())) ) ) ) columns_by_column_register = {} for row in rows: if row['opcode'] in ('Rowid', 'Column'): addr, opcode, table_id, cid, column_register, p4, p5, comment = row table = names_by_rootpage[table_rootpage_by_register[table_id]] columns_by_column_register[column_register] = (table, cid) result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0] registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""])) all_column_names = {} for table in names_by_rootpage.values(): table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall() for row in table_xinfo: all_column_names[(table, row[""cid""])] = row[""name""] final_output = [] for r in registers: try: table, cid = columns_by_column_register[r] final_output.append((table, all_column_names[table, cid])) except KeyError: final_output.append((None, None)) return final_output ``` It works! ```diff diff --git a/datasette/templates/query.html b/datasette/templates/query.html index 75f7f1b..9fe1d4f 100644 --- a/datasette/templates/query.html +++ b/datasette/templates/query.html @@ -67,6 +67,8 @@

+extra_column_info: {{ extra_column_info }} + {% if display_rows %}

This data as {% for name, url in renderers.items() %}{{ name }}{{ "", "" if not loop.last }}{% endfor %}, CSV

diff --git a/datasette/views/database.py b/datasette/views/database.py index 7c36034..02f8039 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -10,6 +10,7 @@ import markupsafe from datasette.utils import ( await_me_maybe, check_visibility, + columns_for_query, derive_named_parameters, to_css_class, validate_sql_select, @@ -248,6 +249,8 @@ class QueryView(DataView): query_error = None + extra_column_info = None + # Execute query - as write or as read if write: if request.method == ""POST"": @@ -334,6 +337,10 @@ class QueryView(DataView): database, sql, params_for_query, truncate=True, **extra_args ) columns = [r[0] for r in results.description] + + # Try to figure out extra column information + db = self.ds.get_database(database) + extra_column_info = await db.execute_fn(lambda conn: columns_for_query(conn, sql)) except sqlite3.DatabaseError as e: query_error = e results = None @@ -462,6 +469,7 @@ class QueryView(DataView): ""show_hide_text"": show_hide_text, ""show_hide_hidden"": markupsafe.Markup(show_hide_hidden), ""hide_sql"": hide_sql, + ""extra_column_info"": extra_column_info, } return ( ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898065011,https://api.github.com/repos/simonw/datasette/issues/1293,898065011,IC_kwDOBm6k_c41h2Jz,9599,simonw,2021-08-13T00:36:30Z,2021-08-13T00:36:30Z,OWNER,"> https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because `paginated_view` is defined like this: > > ```sql > CREATE VIEW paginated_view AS > SELECT > content, > '- ' || content || ' -' AS content_extra > FROM no_primary_key; > ``` > > So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression. Here's what it does for that: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898065948,https://api.github.com/repos/simonw/datasette/issues/1293,898065948,IC_kwDOBm6k_c41h2Yc,9599,simonw,2021-08-13T00:38:58Z,2021-08-13T00:38:58Z,OWNER,"Trying to run `explain select * from facetable` fails with an error in my prototype, because it tries to execute `explain explain select * from facetable` - so I need to spot that error and ignore it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898066466,https://api.github.com/repos/simonw/datasette/issues/1293,898066466,IC_kwDOBm6k_c41h2gi,9599,simonw,2021-08-13T00:40:24Z,2021-08-13T00:40:24Z,OWNER,"It figures out renamed columns too: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1431#issuecomment-898072940,https://api.github.com/repos/simonw/datasette/issues/1431,898072940,IC_kwDOBm6k_c41h4Fs,9599,simonw,2021-08-13T00:58:40Z,2021-08-13T00:58:40Z,OWNER,"While I'm doing this I should rename this internal variable to avoid confusion in the future: https://github.com/simonw/datasette/blob/e837095ef35ae155b4c78cc9a8b7133a48c94f03/datasette/app.py#L203","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969840302,`--help-config` should be called `--help-settings`, https://github.com/simonw/datasette/issues/1432#issuecomment-898074849,https://api.github.com/repos/simonw/datasette/issues/1432,898074849,IC_kwDOBm6k_c41h4jh,9599,simonw,2021-08-13T01:03:40Z,2021-08-13T01:03:40Z,OWNER,"Also this method: https://github.com/simonw/datasette/blob/77f46297a88ac7e49dad2139410b01ee56d5f99c/datasette/app.py#L422-L424 And the places that use it: https://github.com/simonw/datasette/blob/fc4846850fffd54561bc125332dfe97bb41ff42e/datasette/views/base.py#L617 https://github.com/simonw/datasette/blob/fc4846850fffd54561bc125332dfe97bb41ff42e/datasette/views/database.py#L459 Which is used in this template: https://github.com/simonw/datasette/blob/77f46297a88ac7e49dad2139410b01ee56d5f99c/datasette/templates/table.html#L204 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/issues/1432#issuecomment-898079507,https://api.github.com/repos/simonw/datasette/issues/1432,898079507,IC_kwDOBm6k_c41h5sT,9599,simonw,2021-08-13T01:08:42Z,2021-08-13T01:09:41Z,OWNER,"This is going to break some plugins: https://ripgrep.datasette.io/-/ripgrep?pattern=config%3D&literal=on&glob=%21datasette%2F** > ### datasette-cluster-map/tests/test_cluster_map.py > > @pytest.mark.asyncio > > async def test_respects_base_url(): > ds = Datasette([], memory=True, config={""base_url"": ""/foo/""}) > response = await ds.client.get(""/:memory:?sql=select+1+as+latitude,+2+as+longitude"") > assert ( > > ### datasette-export-notebook/tests/test_export_notebook.py > > @pytest.mark.asyncio > > async def test_notebook_no_csv(db_path): > datasette = Datasette([db_path], config={""allow_csv_stream"": False}) > response = await datasette.client.get(""/db/big.Notebook"") > assert "".csv"" not in response.text > > ### datasette-publish-vercel/tests/test_publish_vercel.py > metadata=metadata, > cors=True, > config={""default_page_size"": 10, ""sql_time_limit_ms"": 2000} > ).app() > """""" > > ### datasette-publish-vercel/datasette_publish_vercel/__init__.py > metadata=metadata{extras}, > cors=True, > config={settings} > > ).app() > > """""".strip() > > ### datasette-search-all/tests/test_search_all.py > > async def test_base_url(db_path, path): > sqlite_utils.Database(db_path)[""creatures""].enable_fts([""name"", ""description""]) > datasette = Datasette([db_path], config={""base_url"": ""/foo/""}) > response = await datasette.client.get(path) > assert response.status_code == 200 I should fix those as soon as this goes out in a release. I won't close this issue until then.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/issues/1432#issuecomment-898084675,https://api.github.com/repos/simonw/datasette/issues/1432,898084675,IC_kwDOBm6k_c41h69D,9599,simonw,2021-08-13T01:11:30Z,2021-08-13T01:11:30Z,OWNER,It's only `datasette-publish-vercel` that will break the actual functionality - the others will have broken tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/issues/1429#issuecomment-897960049,https://api.github.com/repos/simonw/datasette/issues/1429,897960049,IC_kwDOBm6k_c41hchx,9599,simonw,2021-08-12T20:53:04Z,2021-08-12T20:53:04Z,OWNER,"Maybe something like this: > [Next page](#) - 100 per page ([show 1,000 per page](#))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969548935,UI for setting `?_size=max` on table page, https://github.com/simonw/datasette/issues/942#issuecomment-897996296,https://api.github.com/repos/simonw/datasette/issues/942,897996296,IC_kwDOBm6k_c41hlYI,9599,simonw,2021-08-12T22:01:36Z,2021-08-12T22:01:36Z,OWNER,"I'm going with `""columns"": {""name-of-column"": ""description-of-column""}`. If I decide to make `""col""` and `""nocol""` available in metadata I'll use those as the keys in the metadata, for consistency with the existing query string parameters. I'm OK with having both `""columns"": ...` and `""col"": ...` keys in the metadata, even though they could be a tiny bit confusing without the documentation.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898021895,https://api.github.com/repos/simonw/datasette/issues/942,898021895,IC_kwDOBm6k_c41hroH,9599,simonw,2021-08-12T22:51:36Z,2021-08-12T22:51:36Z,OWNER,"Prototype: ```diff diff --git a/datasette/static/app.css b/datasette/static/app.css index c6be1e9..5ca64cb 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -784,9 +784,14 @@ svg.dropdown-menu-icon { font-size: 0.7em; color: #666; margin: 0; - padding: 0; padding: 4px 8px 4px 8px; } +.dropdown-menu .dropdown-column-description { + margin: 0; + color: #666; + padding: 4px 8px 4px 8px; + max-width: 20em; +} .dropdown-menu li { border-bottom: 1px solid #ccc; } diff --git a/datasette/static/table.js b/datasette/static/table.js index 991346d..a903112 100644 --- a/datasette/static/table.js +++ b/datasette/static/table.js @@ -9,6 +9,7 @@ var DROPDOWN_HTML = `
  • Show not-blank rows

    • +

    `; var DROPDOWN_ICON_SVG = ` @@ -166,6 +167,14 @@ var DROPDOWN_ICON_SVG = `
    {% for column in display_columns %} - + {% if not column.sortable %} {{ column.name }} {% else %} diff --git a/datasette/views/table.py b/datasette/views/table.py index 456d806..486a613 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -125,6 +125,7 @@ class RowTableShared(DataView): """"""Returns columns, rows for specified table - including fancy foreign key treatment"""""" db = self.ds.databases[database] table_metadata = self.ds.table_metadata(database, table) + column_descriptions = table_metadata.get(""columns"") or {} column_details = {col.name: col for col in await db.table_column_details(table)} sortable_columns = await self.sortable_columns_for_table(database, table, True) pks = await db.primary_keys(table) @@ -147,6 +148,7 @@ class RowTableShared(DataView): ""is_pk"": r[0] in pks_for_display, ""type"": type_, ""notnull"": notnull, + ""description"": column_descriptions.get(r[0]), } ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898022235,https://api.github.com/repos/simonw/datasette/issues/942,898022235,IC_kwDOBm6k_c41hrtb,9599,simonw,2021-08-12T22:52:23Z,2021-08-12T22:52:23Z,OWNER,I like this. Need to solve for mobile though where the cog menu isn't visible - I think I'll do that with a definition list at the top of the page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898032118,https://api.github.com/repos/simonw/datasette/issues/942,898032118,IC_kwDOBm6k_c41huH2,596279,zaneselvans,2021-08-12T23:12:00Z,2021-08-12T23:12:00Z,NONE,"This looks awesome. We'll definitely make extensive use of this feature! On Thu, Aug 12, 2021 at 5:52 PM Simon Willison ***@***.***> wrote: > I like this. Need to solve for mobile though where the cog menu isn't > visible - I think I'll do that with a definition list at the top of the > page. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > , > or unsubscribe > > . > Triage notifications on the go with GitHub Mobile for iOS > > or Android > > . > -- Zane A. Selvans, PhD Chief Data Wrangler Catalyst Cooperative https://catalyst.coop ***@***.*** Signal/WhatsApp/SMS: +1 720 443 1363 Twitter: @ZaneSelvans PGP : 0x64F7B56F3A127B04 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898037456,https://api.github.com/repos/simonw/datasette/issues/942,898037456,IC_kwDOBm6k_c41hvbQ,9599,simonw,2021-08-12T23:23:34Z,2021-08-12T23:23:34Z,OWNER,"Prototype with a `
    `: ```diff diff --git a/datasette/static/app.css b/datasette/static/app.css index c6be1e9..bf068fd 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -836,6 +841,16 @@ svg.dropdown-menu-icon { background-repeat: no-repeat; } +dl.column-descriptions dt { + font-weight: bold; +} +dl.column-descriptions dd { + padding-left: 1.5em; + white-space: pre-wrap; + line-height: 1.1em; + color: #666; +} + .anim-scale-in { animation-name: scale-in; animation-duration: 0.15s; diff --git a/datasette/templates/table.html b/datasette/templates/table.html index 211352b..466e8a4 100644 --- a/datasette/templates/table.html +++ b/datasette/templates/table.html @@ -51,6 +51,14 @@ {% block description_source_license %}{% include ""_description_source_license.html"" %}{% endblock %} +{% if metadata.columns %} +
    + {% for column_name, column_description in metadata.columns.items() %} +
    {{ column_name }}
    {{ column_description }}
    + {% endfor %} +
    +{% endif %} + {% if filtered_table_rows_count or human_description_en %}

    {% if filtered_table_rows_count or filtered_table_rows_count == 0 %}{{ ""{:,}"".format(filtered_table_rows_count) }} row{% if filtered_table_rows_count == 1 %}{% else %}s{% endif %}{% endif %} {% if human_description_en %}{{ human_description_en }}{% endif %} ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898037650,https://api.github.com/repos/simonw/datasette/issues/942,898037650,IC_kwDOBm6k_c41hveS,9599,simonw,2021-08-12T23:23:54Z,2021-08-12T23:23:54Z,OWNER,I like this enough that I'm going to ship it as an alpha and try it out on a couple of live projects.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/pull/1430#issuecomment-898043575,https://api.github.com/repos/simonw/datasette/issues/1430,898043575,IC_kwDOBm6k_c41hw63,22429695,codecov[bot],2021-08-12T23:39:36Z,2021-08-12T23:49:51Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1430](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9419947) into [main](https://codecov.io/gh/simonw/datasette/commit/b1fed48a95516ae84c0f020582303ab50ab817e2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b1fed48) will **increase** coverage by `0.00%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1430/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1430 +/- ## ======================================= Coverage 91.71% 91.71% ======================================= Files 34 34 Lines 4417 4418 +1 ======================================= + Hits 4051 4052 +1 Misses 366 366 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1430/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <100.00%> (+<0.01%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b1fed48...9419947](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969758038,Column metadata, https://github.com/simonw/datasette/issues/942#issuecomment-898051645,https://api.github.com/repos/simonw/datasette/issues/942,898051645,IC_kwDOBm6k_c41hy49,9599,simonw,2021-08-13T00:02:25Z,2021-08-13T00:02:25Z,OWNER,"And on mobile: ![5FAF8D73-7199-4BB7-A5B8-9E46DCB4A985](https://user-images.githubusercontent.com/9599/129284817-dc13cbf4-144e-4f4c-8fb7-470602e2eea0.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/942#issuecomment-898050457,https://api.github.com/repos/simonw/datasette/issues/942,898050457,IC_kwDOBm6k_c41hymZ,9599,simonw,2021-08-12T23:59:53Z,2021-08-12T23:59:53Z,OWNER,"Documentation: https://docs.datasette.io/en/latest/metadata.html#column-descriptions Live demo: https://latest.datasette.io/fixtures/roadside_attractions ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/1293#issuecomment-898056013,https://api.github.com/repos/simonw/datasette/issues/1293,898056013,IC_kwDOBm6k_c41hz9N,9599,simonw,2021-08-13T00:12:09Z,2021-08-13T00:12:09Z,OWNER,"Having added column metadata in #1430 (ref #942) I could also include a definition list at the top of the query results page exposing the column descriptions for any columns, using the same EXPLAIN mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1429#issuecomment-898185944,https://api.github.com/repos/simonw/datasette/issues/1429,898185944,IC_kwDOBm6k_c41iTrY,9599,simonw,2021-08-13T04:37:41Z,2021-08-13T04:37:41Z,OWNER,"If a count is available and the count is less than 1,000 it could say ""Show all"" instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969548935,UI for setting `?_size=max` on table page, https://github.com/simonw/datasette/pull/1433#issuecomment-898450402,https://api.github.com/repos/simonw/datasette/issues/1433,898450402,IC_kwDOBm6k_c41jUPi,22429695,codecov[bot],2021-08-13T13:15:55Z,2021-08-13T13:15:55Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1433](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ddba6cc) into [main](https://codecov.io/gh/simonw/datasette/commit/2883098770fc66e50183b2b231edbde20848d4d6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2883098) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1433/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1433 +/- ## ======================================= Coverage 91.82% 91.82% ======================================= Files 34 34 Lines 4418 4418 ======================================= Hits 4057 4057 Misses 361 361 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2883098...ddba6cc](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",970386262,"Update trustme requirement from <0.9,>=0.7 to >=0.7,<0.10", https://github.com/simonw/datasette/issues/1293#issuecomment-898506647,https://api.github.com/repos/simonw/datasette/issues/1293,898506647,IC_kwDOBm6k_c41jh-X,9599,simonw,2021-08-13T14:43:19Z,2021-08-13T14:43:19Z,OWNER,Work will continue in PR #1434.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898517872,https://api.github.com/repos/simonw/datasette/issues/1293,898517872,IC_kwDOBm6k_c41jktw,9599,simonw,2021-08-13T15:00:50Z,2021-08-13T15:00:50Z,OWNER,"The primary key column (or `rowid`) often resolves to an `index` record in the `sqlite_master` table, e.g. the second row in this: type | name | tbl_name | rootpage | sql -- | -- | -- | -- | -- table | simple_primary_key | simple_primary_key | 2 | CREATE TABLE simple_primary_key ( id varchar(30) primary key, content text ) index | sqlite_autoindex_simple_primary_key_1 | simple_primary_key | 3 |   ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898519924,https://api.github.com/repos/simonw/datasette/issues/1293,898519924,IC_kwDOBm6k_c41jlN0,9599,simonw,2021-08-13T15:03:36Z,2021-08-13T15:03:36Z,OWNER,"Weird edge-case: adding an `order by` changes the order of the columns with respect to the information I am deriving about them. Without order by this gets it right: With order by: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898524057,https://api.github.com/repos/simonw/datasette/issues/1293,898524057,IC_kwDOBm6k_c41jmOZ,9599,simonw,2021-08-13T15:06:37Z,2021-08-13T15:06:37Z,OWNER,"Comparing the `explain` for the two versions of that query - one with the order by and one without: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898527525,https://api.github.com/repos/simonw/datasette/issues/1293,898527525,IC_kwDOBm6k_c41jnEl,9599,simonw,2021-08-13T15:08:03Z,2021-08-13T15:08:03Z,OWNER,Am I going to need to look at the `ResultRow` and its columns but then wind back to that earlier `MakeRecord` and its columns?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898536181,https://api.github.com/repos/simonw/datasette/issues/1293,898536181,IC_kwDOBm6k_c41jpL1,9599,simonw,2021-08-13T15:17:20Z,2021-08-13T15:20:33Z,OWNER,"Documentation for `MakeRecord`: https://www.sqlite.org/opcode.html#MakeRecord Running `explain` inside `sqlite3` provides extra comments and indentation which make it easier to understand: ``` sqlite> explain select neighborhood, facet_cities.name, state ...> from facetable ...> join facet_cities ...> on facetable.city_id = facet_cities.id ...> where neighborhood like '%bob%'; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 15 0 00 Start at 15 1 OpenRead 0 43 0 7 00 root=43 iDb=0; facetable 2 OpenRead 1 42 0 2 00 root=42 iDb=0; facet_cities 3 Rewind 0 14 0 00 4 Column 0 6 3 00 r[3]=facetable.neighborhood 5 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 6 IfNot 1 13 1 00 7 Column 0 5 4 00 r[4]=facetable.city_id 8 SeekRowid 1 13 4 00 intkey=r[4] 9 Column 0 6 5 00 r[5]=facetable.neighborhood 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 ResultRow 5 3 0 00 output=r[5..7] 13 Next 0 4 0 01 14 Halt 0 0 0 00 15 Transaction 0 0 35 0 01 usesStmtJournal=0 16 String8 0 2 0 %bob% 00 r[2]='%bob%' 17 Goto 0 1 0 00 ``` Compared with: ``` sqlite> explain select neighborhood, facet_cities.name, state ...> from facetable ...> join facet_cities ...> on facetable.city_id = facet_cities.id ...> where neighborhood like '%bob%' order by neighborhood ...> ; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 25 0 00 Start at 25 1 SorterOpen 2 5 0 k(1,B) 00 2 OpenRead 0 43 0 7 00 root=43 iDb=0; facetable 3 OpenRead 1 42 0 2 00 root=42 iDb=0; facet_cities 4 Rewind 0 16 0 00 5 Column 0 6 3 00 r[3]=facetable.neighborhood 6 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 7 IfNot 1 15 1 00 8 Column 0 5 4 00 r[4]=facetable.city_id 9 SeekRowid 1 15 4 00 intkey=r[4] 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 Column 0 6 5 00 r[5]=facetable.neighborhood 13 MakeRecord 5 3 9 00 r[9]=mkrec(r[5..7]) 14 SorterInsert 2 9 5 3 00 key=r[9] 15 Next 0 5 0 01 16 OpenPseudo 3 10 5 00 5 columns in r[10] 17 SorterSort 2 24 0 00 18 SorterData 2 10 3 00 r[10]=data 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] 23 SorterNext 2 18 0 00 24 Halt 0 0 0 00 25 Transaction 0 0 35 0 01 usesStmtJournal=0 26 String8 0 2 0 %bob% 00 r[2]='%bob%' 27 Goto 0 1 0 00 ``` So actually it looks like the `SorterSort` may be key to understanding this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898540260,https://api.github.com/repos/simonw/datasette/issues/1293,898540260,IC_kwDOBm6k_c41jqLk,9599,simonw,2021-08-13T15:23:28Z,2021-08-13T15:23:28Z,OWNER,"SorterInsert: > Register P2 holds an SQL index key made using the MakeRecord instructions. This opcode writes that key into the sorter P1. Data for the entry is nil. SorterData: > Write into register P2 the current sorter data for sorter cursor P1. Then clear the column header cache on cursor P3. > > This opcode is normally use to move a record out of the sorter and into a register that is the source for a pseudo-table cursor created using OpenPseudo. That pseudo-table cursor is the one that is identified by parameter P3. Clearing the P3 column cache as part of this opcode saves us from having to issue a separate NullRow instruction to clear that cache. OpenPseudo: > Open a new cursor that points to a fake table that contains a single row of data. The content of that one row is the content of memory register P2. In other words, cursor P1 becomes an alias for the MEM_Blob content contained in register P2. > > A pseudo-table created by this opcode is used to hold a single row output from the sorter so that the row can be decomposed into individual columns using the Column opcode. The Column opcode is the only cursor opcode that works with a pseudo-table. > > P3 is the number of fields in the records that will be stored by the pseudo-table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898541543,https://api.github.com/repos/simonw/datasette/issues/1293,898541543,IC_kwDOBm6k_c41jqfn,9599,simonw,2021-08-13T15:25:26Z,2021-08-13T15:25:26Z,OWNER,"But the debug output here seems to be saying what we want it to say: ``` 17 SorterSort 2 24 0 00 18 SorterData 2 10 3 00 r[10]=data 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] ``` We want to get back `neighborhood`, `facet_cities.name`, `state`. Why then are we seeing `[('facet_cities', 'name'), ('facetable', 'state'), (None, None)]`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898541972,https://api.github.com/repos/simonw/datasette/issues/1293,898541972,IC_kwDOBm6k_c41jqmU,9599,simonw,2021-08-13T15:26:06Z,2021-08-13T15:29:06Z,OWNER,"ResultRow: > The registers P1 through P1+P2-1 contain a single row of results. This opcode causes the sqlite3_step() call to terminate with an SQLITE_ROW return code and it sets up the sqlite3_stmt structure to provide access to the r(P1)..r(P1+P2-1) values as the result row. Column: > Interpret the data that cursor P1 points to as a structure built using the MakeRecord instruction. (See the MakeRecord opcode for additional information about the format of the data.) Extract the P2-th column from this record. If there are less that (P2+1) values in the record, extract a NULL. > > The value extracted is stored in register P3.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898545815,https://api.github.com/repos/simonw/datasette/issues/1293,898545815,IC_kwDOBm6k_c41jriX,9599,simonw,2021-08-13T15:31:53Z,2021-08-13T15:31:53Z,OWNER,"My hunch here is that registers or columns are being reused in a way that makes my code break - my code is pretty dumb, there are places in it where maybe the first mention of a register wins instead of the last one?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898554427,https://api.github.com/repos/simonw/datasette/issues/1293,898554427,IC_kwDOBm6k_c41jto7,9599,simonw,2021-08-13T15:45:32Z,2021-08-13T15:45:32Z,OWNER,"Some useful debug output: ``` table_rootpage_by_register={0: 43, 1: 42} names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} result_registers=[6, 7, 8] columns_by_column_register={3: ('facetable', 6), 4: ('facetable', 5), 6: ('facet_cities', 1), 7: ('facetable', 4), 5: ('facetable', 6)} all_column_names={('facet_cities', 0): 'id', ('facet_cities', 1): 'name', ('facetable', 0): 'pk', ('facetable', 1): 'created', ('facetable', 2): 'planet_int', ('facetable', 3): 'on_earth', ('facetable', 4): 'state', ('facetable', 5): 'city_id', ('facetable', 6): 'neighborhood', ('facetable', 7): 'tags', ('facetable', 8): 'complex_array', ('facetable', 9): 'distinct_some_null'} ``` The `result_registers` should each correspond to the correct entry in `columns_by_column_register` but they do not. Python code: ```python def columns_for_query(conn, sql, params=None): """""" Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of ``(table_name, column_name)`` pairs corresponding to the columns that would be returned by that SQL query. Each pair indicates the source table and column for the returned column, or ``(None, None)`` if no table and column could be derived (e.g. for ""select 1"") """""" if sql.lower().strip().startswith(""explain""): return [] opcodes = conn.execute(""explain "" + sql, params).fetchall() table_rootpage_by_register = { r[""p1""]: r[""p2""] for r in opcodes if r[""opcode""] == ""OpenRead"" } print(f""{table_rootpage_by_register=}"") names_and_types_by_rootpage = dict( [(r[0], (r[1], r[2])) for r in conn.execute( ""select rootpage, name, type from sqlite_master where rootpage in ({})"".format( "", "".join(map(str, table_rootpage_by_register.values())) ) )] ) print(f""{names_and_types_by_rootpage=}"") columns_by_column_register = {} for opcode in opcodes: if opcode[""opcode""] in (""Rowid"", ""Column""): addr, opcode, table_id, cid, column_register, p4, p5, comment = opcode try: table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] columns_by_column_register[column_register] = (table, cid) except KeyError: pass result_row = [dict(r) for r in opcodes if r[""opcode""] == ""ResultRow""][0] result_registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""])) print(f""{result_registers=}"") print(f""{columns_by_column_register=}"") all_column_names = {} for (table, _) in names_and_types_by_rootpage.values(): table_xinfo = conn.execute(""pragma table_xinfo({})"".format(table)).fetchall() for column_info in table_xinfo: all_column_names[(table, column_info[""cid""])] = column_info[""name""] print(f""{all_column_names=}"") final_output = [] for register in result_registers: try: table, cid = columns_by_column_register[register] final_output.append((table, all_column_names[table, cid])) except KeyError: final_output.append((None, None)) return final_output ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898554859,https://api.github.com/repos/simonw/datasette/issues/1293,898554859,IC_kwDOBm6k_c41jtvr,9599,simonw,2021-08-13T15:46:18Z,2021-08-13T15:46:18Z,OWNER,So it looks like the bug is in the code that populates `columns_by_column_register`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898567974,https://api.github.com/repos/simonw/datasette/issues/1293,898567974,IC_kwDOBm6k_c41jw8m,9599,simonw,2021-08-13T16:07:00Z,2021-08-13T16:07:00Z,OWNER,"So this line: ``` 19 Column 3 2 8 00 r[8]=state ``` Means ""Take column 2 of table 3 (the pseudo-table) and store it in register 8""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898564705,https://api.github.com/repos/simonw/datasette/issues/1293,898564705,IC_kwDOBm6k_c41jwJh,9599,simonw,2021-08-13T16:02:12Z,2021-08-13T16:04:06Z,OWNER,"More debug output: ``` table_rootpage_by_register={0: 43, 1: 42} names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_id=0 cid=6 column_register=3 table_id=0 cid=5 column_register=4 table_id=1 cid=1 column_register=6 table_id=0 cid=4 column_register=7 table_id=0 cid=6 column_register=5 table_id=3 cid=2 column_register=8 table_id=3 cid=2 column_register=8 KeyError 3 table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3 columns_by_column_register[column_register] = (table, cid) column_register=8 = (table='facetable', cid=2) table_id=3 cid=1 column_register=7 KeyError 3 table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3 columns_by_column_register[column_register] = (table, cid) column_register=7 = (table='facetable', cid=1) table_id=3 cid=0 column_register=6 KeyError 3 table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3 columns_by_column_register[column_register] = (table, cid) column_register=6 = (table='facetable', cid=0) result_registers=[6, 7, 8] columns_by_column_register={3: ('facetable', 6), 4: ('facetable', 5), 6: ('facet_cities', 1), 7: ('facetable', 4), 5: ('facetable', 6)} all_column_names={('facet_cities', 0): 'id', ('facet_cities', 1): 'name', ('facetable', 0): 'pk', ('facetable', 1): 'created', ('facetable', 2): 'planet_int', ('facetable', 3): 'on_earth', ('facetable', 4): 'state', ('facetable', 5): 'city_id', ('facetable', 6): 'neighborhood', ('facetable', 7): 'tags', ('facetable', 8): 'complex_array', ('facetable', 9): 'distinct_some_null'} ``` Those `KeyError` are happening here because of a lookup in `table_rootpage_by_register` for `table_id=3` - but `table_rootpage_by_register` only has keys 0 and 1. It looks like that `3` actually corresponds to the `OpenPseudo` table from here: ``` 16 OpenPseudo 3 10 5 00 5 columns in r[10] 17 SorterSort 2 24 0 00 18 SorterData 2 10 3 00 r[10]=data 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] ``` Python code: ```python def columns_for_query(conn, sql, params=None): """""" Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of ``(table_name, column_name)`` pairs corresponding to the columns that would be returned by that SQL query. Each pair indicates the source table and column for the returned column, or ``(None, None)`` if no table and column could be derived (e.g. for ""select 1"") """""" if sql.lower().strip().startswith(""explain""): return [] opcodes = conn.execute(""explain "" + sql, params).fetchall() table_rootpage_by_register = { r[""p1""]: r[""p2""] for r in opcodes if r[""opcode""] == ""OpenRead"" } print(f""{table_rootpage_by_register=}"") names_and_types_by_rootpage = dict( [(r[0], (r[1], r[2])) for r in conn.execute( ""select rootpage, name, type from sqlite_master where rootpage in ({})"".format( "", "".join(map(str, table_rootpage_by_register.values())) ) )] ) print(f""{names_and_types_by_rootpage=}"") columns_by_column_register = {} for opcode_row in opcodes: if opcode_row[""opcode""] in (""Rowid"", ""Column""): addr, opcode, table_id, cid, column_register, p4, p5, comment = opcode_row print(f""{table_id=} {cid=} {column_register=}"") try: table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0] columns_by_column_register[column_register] = (table, cid) except KeyError as e: print("" KeyError"") print("" "", e) print("" table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]"") print(f"" {names_and_types_by_rootpage=} {table_rootpage_by_register=} {table_id=}"") print("" columns_by_column_register[column_register] = (table, cid)"") print(f"" {column_register=} = ({table=}, {cid=})"") pass result_row = [dict(r) for r in opcodes if r[""opcode""] == ""ResultRow""][0] result_registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""])) print(f""{result_registers=}"") print(f""{columns_by_column_register=}"") all_column_names = {} for (table, _) in names_and_types_by_rootpage.values(): table_xinfo = conn.execute(""pragma table_xinfo({})"".format(table)).fetchall() for column_info in table_xinfo: all_column_names[(table, column_info[""cid""])] = column_info[""name""] print(f""{all_column_names=}"") final_output = [] for register in result_registers: try: table, cid = columns_by_column_register[register] final_output.append((table, all_column_names[table, cid])) except KeyError: final_output.append((None, None)) return final_output ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898572065,https://api.github.com/repos/simonw/datasette/issues/1293,898572065,IC_kwDOBm6k_c41jx8h,9599,simonw,2021-08-13T16:13:16Z,2021-08-13T16:13:16Z,OWNER,"Aha! That `MakeRecord` line says `r[5..7]` - and r5 = neighborhood, r6 = facet_cities.name, r7 = facetable.state So if the `MakeRecord` defines what goes into that pseudo-table column 2 of that pseudo-table would be `state` - which is what we want. This is really convoluted. I'm no longer confident I can get this to work in a sensible way, especially since I've not started exploring what complex nested tables with CTEs and sub-selects do yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898569319,https://api.github.com/repos/simonw/datasette/issues/1293,898569319,IC_kwDOBm6k_c41jxRn,9599,simonw,2021-08-13T16:09:01Z,2021-08-13T16:10:48Z,OWNER,"Need to figure out what column 2 of that pseudo-table is. I think the answer is here: ``` 4 Rewind 0 16 0 00 5 Column 0 6 3 00 r[3]=facetable.neighborhood 6 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 7 IfNot 1 15 1 00 8 Column 0 5 4 00 r[4]=facetable.city_id 9 SeekRowid 1 15 4 00 intkey=r[4] 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 Column 0 6 5 00 r[5]=facetable.neighborhood 13 MakeRecord 5 3 9 00 r[9]=mkrec(r[5..7]) 14 SorterInsert 2 9 5 3 00 key=r[9] 15 Next 0 5 0 01 16 OpenPseudo 3 10 5 00 5 columns in r[10] ``` I think the `OpenPseduo` line puts five columns in `r[10]` - and those five columns are the five from the previous block - maybe the five leading up to the `MakeRecord` call on line 13. In which case column 2 would be `facet_cities.name` - assuming we start counting from 0. But the debug code said ""r[8]=state"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898576097,https://api.github.com/repos/simonw/datasette/issues/1293,898576097,IC_kwDOBm6k_c41jy7h,9599,simonw,2021-08-13T16:19:57Z,2021-08-13T16:19:57Z,OWNER,"I think I need to look out for `OpenPseudo` and, when that occurs, take a look at the most recent `SorterInsert` and use that to find the `MakeRecord` and then use the `MakeRecord` to figure out the columns that went into it. After all of that I'll be able to resolve that ""table 3"" reference.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898760020,https://api.github.com/repos/simonw/datasette/issues/1293,898760020,IC_kwDOBm6k_c41kf1U,9599,simonw,2021-08-13T23:00:28Z,2021-08-13T23:01:27Z,OWNER,"New theory: this is all about `SorterOpen` and `SorterInsert`. Consider the following with extra annotations at the end of the lines after the `--`: ``` addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 25 0 00 Start at 25 1 SorterOpen 2 5 0 k(1,B) 00 -- New SORTER in r2 with 5 slots 2 OpenRead 0 43 0 7 00 root=43 iDb=0; facetable 3 OpenRead 1 42 0 2 00 root=42 iDb=0; facet_cities 4 Rewind 0 16 0 00 5 Column 0 6 3 00 r[3]=facetable.neighborhood 6 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3]) 7 IfNot 1 15 1 00 8 Column 0 5 4 00 r[4]=facetable.city_id 9 SeekRowid 1 15 4 00 intkey=r[4] 10 Column 1 1 6 00 r[6]=facet_cities.name 11 Column 0 4 7 00 r[7]=facetable.state 12 Column 0 6 5 00 r[5]=facetable.neighborhood 13 MakeRecord 5 3 9 00 r[9]=mkrec(r[5..7]) 14 SorterInsert 2 9 5 3 00 key=r[9]-- WRITES record from r9 (line above) into sorter in r2 15 Next 0 5 0 01 16 OpenPseudo 3 10 5 00 5 columns in r[10] 17 SorterSort 2 24 0 00 -- runs the sort, not relevant to my goal 18 SorterData 2 10 3 00 r[10]=data -- ""Write into register P2 (r10) the current sorter data for sorter cursor P1 (sorter 2)"" 19 Column 3 2 8 00 r[8]=state 20 Column 3 1 7 00 r[7]=facet_cities.name 21 Column 3 0 6 00 r[6]=neighborhood 22 ResultRow 6 3 0 00 output=r[6..8] 23 SorterNext 2 18 0 00 24 Halt 0 0 0 00 25 Transaction 0 0 35 0 01 usesStmtJournal=0 26 String8 0 2 0 %bob% 00 r[2]='%bob%' 27 Goto 0 1 0 00 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898760808,https://api.github.com/repos/simonw/datasette/issues/1293,898760808,IC_kwDOBm6k_c41kgBo,9599,simonw,2021-08-13T23:03:01Z,2021-08-13T23:03:01Z,OWNER,Another idea: strip out any `order by` clause to try and keep this simpler. I doubt that's going to cope with complex nested queries though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898788262,https://api.github.com/repos/simonw/datasette/issues/1293,898788262,IC_kwDOBm6k_c41kmum,9599,simonw,2021-08-14T01:22:26Z,2021-08-14T01:51:08Z,OWNER,"Tried a more complicated query: ```sql explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101 ``` Here's the explain: ``` sqlite> explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101 ...> ; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 41 0 00 Start at 41 1 OpenEphemeral 2 6 0 k(1,-B) 00 nColumn=6 2 Integer 101 1 0 00 r[1]=101; LIMIT counter 3 OpenRead 0 32 0 4 00 root=32 iDb=0; searchable 4 Integer 16 3 0 00 r[3]=16; return address 5 Once 0 16 0 00 6 OpenEphemeral 3 1 0 k(1,) 00 nColumn=1; Result of SELECT 1 7 VOpen 1 0 0 vtab:7FCBCA72BE80 00 8 Function0 1 7 6 unknown(-1) 01 r[6]=func(r[7]) 9 Integer 5 4 0 00 r[4]=5 10 Integer 1 5 0 00 r[5]=1 11 VFilter 1 16 4 00 iplan=r[4] zplan='' 12 Rowid 1 8 0 00 r[8]=rowid 13 MakeRecord 8 1 9 C 00 r[9]=mkrec(r[8]) 14 IdxInsert 3 9 8 1 00 key=r[9] 15 VNext 1 12 0 00 16 Return 3 0 0 00 17 Rewind 3 33 0 00 18 Column 3 0 2 00 r[2]= 19 IsNull 2 32 0 00 if r[2]==NULL goto 32 20 SeekRowid 0 32 2 00 intkey=r[2] 21 Column 0 1 10 00 r[10]=searchable.text1 22 Sequence 2 11 0 00 r[11]=cursor[2].ctr++ 23 IfNotZero 1 27 0 00 if r[1]!=0 then r[1]--, goto 27 24 Last 2 0 0 00 25 IdxLE 2 32 10 1 00 key=r[10] 26 Delete 2 0 0 00 27 Rowid 0 12 0 00 r[12]=rowid 28 Column 0 2 13 00 r[13]=searchable.text2 29 Column 0 3 14 00 r[14]=searchable.name with . and spaces 30 MakeRecord 10 5 16 00 r[16]=mkrec(r[10..14]) 31 IdxInsert 2 16 10 5 00 key=r[16] 32 Next 3 18 0 00 33 Sort 2 40 0 00 34 Column 2 4 15 00 r[15]=[name with . and spaces] 35 Column 2 3 14 00 r[14]=text2 36 Column 2 0 13 00 r[13]=text1 37 Column 2 2 12 00 r[12]=pk 38 ResultRow 12 4 0 00 output=r[12..15] 39 Next 2 34 0 00 40 Halt 0 0 0 00 41 Transaction 0 0 35 0 01 usesStmtJournal=0 42 Variable 1 7 0 :search 00 r[7]=parameter(1,:search) 43 Goto 0 1 0 00 ``` Here the `ResultRow` is for registers `12..15` - but those all refer to `Column` records in `2` - where `2` is the first `OpenEphemeral` declared right at the start. I'm having enormous trouble figuring out how that ephemeral table gets populated by the other operations in a way that would let me derive which columns end up in the `ResultRow`. Frustratingly SQLite seems to be able to figure that out just fine, see the column of comments on the right hand side - but I only get those in the `sqlite3` CLI shell, they're not available to me with SQLite when called as a library from Python. Maybe the key to that is this section: ``` 27 Rowid 0 12 0 00 r[12]=rowid 28 Column 0 2 13 00 r[13]=searchable.text2 29 Column 0 3 14 00 r[14]=searchable.name with . and spaces 30 MakeRecord 10 5 16 00 r[16]=mkrec(r[10..14]) 31 IdxInsert 2 16 10 5 00 key=r[16] ``` MakeRecord: > Convert P2 registers beginning with P1 into the record format use as a data record in a database table or as a key in an index. The Column opcode can decode the record later. > > P4 may be a string that is P2 characters long. The N-th character of the string indicates the column affinity that should be used for the N-th field of the index key. > > The mapping from character to affinity is given by the SQLITE_AFF_ macros defined in sqliteInt.h. > > If P4 is NULL then all index fields have the affinity BLOB. > > The meaning of P5 depends on whether or not the SQLITE_ENABLE_NULL_TRIM compile-time option is enabled: > > * If SQLITE_ENABLE_NULL_TRIM is enabled, then the P5 is the index of the right-most table that can be null-trimmed. > > * If SQLITE_ENABLE_NULL_TRIM is omitted, then P5 has the value OPFLAG_NOCHNG_MAGIC if the MakeRecord opcode is allowed to accept no-change records with serial_type 10. This value is only used inside an assert() and does not affect the end result. IdxInsert: > Register P2 holds an SQL index key made using the MakeRecord instructions. This opcode writes that key into the index P1. Data for the entry is nil. > > If P4 is not zero, then it is the number of values in the unpacked key of reg(P2). In that case, P3 is the index of the first register for the unpacked key. The availability of the unpacked key can sometimes be an optimization. > > If P5 has the OPFLAG_APPEND bit set, that is a hint to the b-tree layer that this insert is likely to be an append. > > If P5 has the OPFLAG_NCHANGE bit set, then the change counter is incremented by this instruction. If the OPFLAG_NCHANGE bit is clear, then the change counter is unchanged. > > If the OPFLAG_USESEEKRESULT flag of P5 is set, the implementation might run faster by avoiding an unnecessary seek on cursor P1. However, the OPFLAG_USESEEKRESULT flag must only be set if there have been no prior seeks on the cursor or if the most recent seek used a key equivalent to P2. > > This instruction only works for indices. The equivalent instruction for tables is Insert. IdxLE: > The P4 register values beginning with P3 form an unpacked index key that omits the PRIMARY KEY or ROWID. Compare this key value against the index that P1 is currently pointing to, ignoring the PRIMARY KEY or ROWID on the P1 index. > > If the P1 index entry is less than or equal to the key value then jump to P2. Otherwise fall through to the next instruction.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898913554,https://api.github.com/repos/simonw/datasette/issues/1293,898913554,IC_kwDOBm6k_c41lFUS,9599,simonw,2021-08-14T16:13:40Z,2021-08-14T16:13:40Z,OWNER,"I think I need to care about the following: - `ResultRow` and `Column` for the final result - `OpenRead` for opening tables - `OpenEphemeral` then `MakeRecord` and `IdxInsert` for writing records into ephemeral tables `Column` may reference either a table (from `OpenRead`) or an ephemeral table (from `OpenEphemeral`). That *might* be enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898913629,https://api.github.com/repos/simonw/datasette/issues/1293,898913629,IC_kwDOBm6k_c41lFVd,9599,simonw,2021-08-14T16:14:12Z,2021-08-14T16:14:12Z,OWNER,I would feel a lot more comfortable about all of this if I had a robust mechanism for running the Datasette test suite against multiple versions of SQLite itself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898936068,https://api.github.com/repos/simonw/datasette/issues/1293,898936068,IC_kwDOBm6k_c41lK0E,9599,simonw,2021-08-14T17:44:54Z,2021-08-14T17:44:54Z,OWNER,"Another interesting query to consider: https://latest.datasette.io/fixtures?sql=explain+select+*+from++pragma_table_info%28+%27123_starts_with_digits%27%29 That one shows `VColumn` instead of `Column`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898933865,https://api.github.com/repos/simonw/datasette/issues/1293,898933865,IC_kwDOBm6k_c41lKRp,9599,simonw,2021-08-14T17:27:16Z,2021-08-14T17:28:29Z,OWNER,"Maybe I split this out into a separate Python library that gets tested against *every* SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself? Datasette could depend on that library. The library could be released independently of Datasette any time a new SQLite version comes out. I could even run a separate git scraper repo that checks for new SQLite releases and submits PRs against the library when a new release comes out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-898961535,https://api.github.com/repos/simonw/datasette/issues/1293,898961535,IC_kwDOBm6k_c41lRB_,9599,simonw,2021-08-14T21:37:24Z,2021-08-14T21:37:24Z,OWNER,Did some more research into building SQLite custom versions via `pysqlite3` - here's what I figured out for macOS (which should hopefully work for Linux too): https://til.simonwillison.net/sqlite/build-specific-sqlite-pysqlite-macos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1293#issuecomment-899915829,https://api.github.com/repos/simonw/datasette/issues/1293,899915829,IC_kwDOBm6k_c41o6A1,9599,simonw,2021-08-17T01:02:35Z,2021-08-17T01:02:35Z,OWNER,"New approach: this time I'm building a simplified executor for the bytecode operations themselves. ```python def execute_operations(operations, max_iterations = 100, trace=None): trace = trace or (lambda *args: None) registers: Dict[int, Any] = {} cursors: Dict[int, Tuple[str, Dict]] = {} instruction_pointer = 0 iterations = 0 result_row = None while True: iterations += 1 if iterations > max_iterations: break operation = operations[instruction_pointer] trace(instruction_pointer, dict(operation)) opcode = operation[""opcode""] if opcode == ""Init"": if operation[""p2""] != 0: instruction_pointer = operation[""p2""] continue else: instruction_pointer += 1 continue elif opcode == ""Goto"": instruction_pointer = operation[""p2""] continue elif opcode == ""Halt"": break elif opcode == ""OpenRead"": cursors[operation[""p1""]] = (""database_table"", { ""rootpage"": operation[""p2""], ""connection"": operation[""p3""], }) elif opcode == ""OpenEphemeral"": cursors[operation[""p1""]] = (""ephemeral"", { ""num_columns"": operation[""p2""], ""index_keys"": [], }) elif opcode == ""MakeRecord"": registers[operation[""p3""]] = (""MakeRecord"", { ""registers"": list(range(operation[""p1""] + operation[""p2""])) }) elif opcode == ""IdxInsert"": record = registers[operation[""p2""]] cursors[operation[""p1""]][1][""index_keys""].append(record) elif opcode == ""Rowid"": registers[operation[""p2""]] = (""rowid"", { ""table"": operation[""p1""] }) elif opcode == ""Sequence"": registers[operation[""p2""]] = (""sequence"", { ""next_from_cursor"": operation[""p1""] }) elif opcode == ""Column"": registers[operation[""p3""]] = (""column"", { ""cursor"": operation[""p1""], ""column_offset"": operation[""p2""] }) elif opcode == ""ResultRow"": p1 = operation[""p1""] p2 = operation[""p2""] trace(""ResultRow: "", list(range(p1, p1 + p2)), registers) result_row = [registers.get(i) for i in range(p1, p1 + p2)] elif opcode == ""Integer"": registers[operation[""p2""]] = (""Integer"", operation[""p1""]) elif opcode == ""String8"": registers[operation[""p2""]] = (""String"", operation[""p4""]) instruction_pointer += 1 return {""registers"": registers, ""cursors"": cursors, ""result_row"": result_row} ``` Results are promising! ``` execute_operations(db.execute(""explain select 'hello', 55, rowid, * from searchable"").fetchall()) {'registers': {1: ('String', 'hello'), 2: ('Integer', 55), 3: ('rowid', {'table': 0}), 4: ('rowid', {'table': 0}), 5: ('column', {'cursor': 0, 'column_offset': 1}), 6: ('column', {'cursor': 0, 'column_offset': 2}), 7: ('column', {'cursor': 0, 'column_offset': 3})}, 'cursors': {0: ('database_table', {'rootpage': 32, 'connection': 0})}, 'result_row': [('String', 'hello'), ('Integer', 55), ('rowid', {'table': 0}), ('rowid', {'table': 0}), ('column', {'cursor': 0, 'column_offset': 1}), ('column', {'cursor': 0, 'column_offset': 2}), ('column', {'cursor': 0, 'column_offset': 3})]} ``` Here's what happens with a union across three tables: ``` execute_operations(db.execute(f"""""" explain select data as content from binary_data union select pk as content from complex_foreign_keys union select name as content from facet_cities """"""}).fetchall()) {'registers': {1: ('column', {'cursor': 4, 'column_offset': 0}), 2: ('MakeRecord', {'registers': [0, 1, 2, 3]}), 3: ('column', {'cursor': 0, 'column_offset': 1}), 4: ('column', {'cursor': 3, 'column_offset': 0})}, 'cursors': {3: ('ephemeral', {'num_columns': 1, 'index_keys': [('MakeRecord', {'registers': [0, 1]}), ('MakeRecord', {'registers': [0, 1]}), ('MakeRecord', {'registers': [0, 1, 2, 3]})]}), 2: ('database_table', {'rootpage': 44, 'connection': 0}), 4: ('database_table', {'rootpage': 24, 'connection': 0}), 0: ('database_table', {'rootpage': 42, 'connection': 0})}, 'result_row': [('column', {'cursor': 3, 'column_offset': 0})]} ``` Note how the result_row refers to cursor 3, which is an ephemeral table which had three different sets of `MakeRecord` index keys assigned to it - indicating that the output column is NOT from the same underlying table source.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1423#issuecomment-899744109,https://api.github.com/repos/simonw/datasette/issues/1423,899744109,IC_kwDOBm6k_c41oQFt,9599,simonw,2021-08-16T18:58:29Z,2021-08-16T18:58:29Z,OWNER,"I didn't bother with the tooltip, just the visible display if `?_facet_size=max`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1423#issuecomment-899749881,https://api.github.com/repos/simonw/datasette/issues/1423,899749881,IC_kwDOBm6k_c41oRf5,9599,simonw,2021-08-16T19:07:02Z,2021-08-16T19:07:02Z,OWNER,"Demo: https://latest.datasette.io/fixtures/compound_three_primary_keys?_facet=content&_facet_size=max&_facet=pk1&_facet=pk2 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1438#issuecomment-900681413,https://api.github.com/repos/simonw/datasette/issues/1438,900681413,IC_kwDOBm6k_c41r07F,9599,simonw,2021-08-17T22:47:44Z,2021-08-17T22:47:44Z,OWNER,I deployed another copy of `fixtures.db` on Vercel at https://til.simonwillison.net/fixtures so I can compare it with `fixtures.db` on Cloud Run at https://latest.datasette.io/fixtures,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900690998,https://api.github.com/repos/simonw/datasette/issues/1438,900690998,IC_kwDOBm6k_c41r3Q2,9599,simonw,2021-08-17T23:11:16Z,2021-08-17T23:12:25Z,OWNER,"I have completely failed to replicate this initial bug - but it's still there on the `thesession.vercel.app` deployment (even though my own deployments to Vercel do not exhibit it). Here's a one-liner to replicate it against that deployment: `curl -s 'https://thesession.vercel.app/thesession?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22' | rg '.csv'` Whit outputs this: `

    This data as json, CSV

    ` It looks like, rather than being URL-encoded, the original query string is somehow making it through to Jinja and then being auto-escaped there. The weird thing is that the equivalent query executed against my `til.simonwillison.net` Vercel instance does this: `curl -s 'https://til.simonwillison.net/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25a%25%22' | rg '.csv'` `

    This data as json, CSV

    `","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1439#issuecomment-900699670,https://api.github.com/repos/simonw/datasette/issues/1439,900699670,IC_kwDOBm6k_c41r5YW,9599,simonw,2021-08-17T23:34:23Z,2021-08-17T23:34:23Z,OWNER,"The challenge comes down to telling the difference between the following: - `/db/table` - an HTML table page - `/db/table.csv` - the CSV version of `/db/table` - `/db/table.csv` - no this one is actually a database table called `table.csv` - `/db/table.csv.csv` - the CSV version of `/db/table.csv` - `/db/table.csv.csv.csv` and so on...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900705226,https://api.github.com/repos/simonw/datasette/issues/1439,900705226,IC_kwDOBm6k_c41r6vK,9599,simonw,2021-08-17T23:50:32Z,2021-08-17T23:50:47Z,OWNER,"An alternative solution would be to use some form of escaping for the characters that form the name of the table. The obvious way to do this would be URL-encoding - but it doesn't hold for `.` characters. The hex for that is `%2E` but watch what happens with that in a URL: ``` # Against Cloud Run: curl -s 'https://datasette.io/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path 'path': '/-/asgi-scope/foo/bar/baz.', 'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz.', 'root_path': '', # Against Vercel: curl -s 'https://til.simonwillison.net/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path 'path': '/-/asgi-scope/foo/bar%2Fbaz%2E', 'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz%2E', 'root_path': '', ``` Surprisingly in this case Vercel DOES keep it intact, but Cloud Run does not. It's still no good though: I need a solution that works on Vercel, Cloud Run and every other potential hosting provider too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900709703,https://api.github.com/repos/simonw/datasette/issues/1439,900709703,IC_kwDOBm6k_c41r71H,9599,simonw,2021-08-18T00:03:09Z,2021-08-18T00:03:09Z,OWNER,"But... what if I invent my own escaping scheme? I actually did this once before, in https://github.com/simonw/datasette/commit/9fdb47ca952b93b7b60adddb965ea6642b1ff523 - while I was working on porting Datasette to ASGI in https://github.com/simonw/datasette/issues/272#issuecomment-494192779 because ASGI didn't yet have the `raw_path` mechanism. I could bring that back - it looked like this: ``` ""table/and/slashes"" => ""tableU+002FandU+002Fslashes"" ""~table"" => ""U+007Etable"" ""+bobcats!"" => ""U+002Bbobcats!"" ""U+007Etable"" => ""UU+002B007Etable"" ``` But I didn't particularly like it - it was quite verbose.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900711967,https://api.github.com/repos/simonw/datasette/issues/1439,900711967,IC_kwDOBm6k_c41r8Yf,9599,simonw,2021-08-18T00:08:09Z,2021-08-18T00:08:09Z,OWNER,"Here's an alternative I just made up which I'm calling ""dot dash"" encoding: ```python def dot_dash_encode(s): return s.replace(""-"", ""--"").replace(""."", ""-."") def dot_dash_decode(s): return s.replace(""-."", ""."").replace(""--"", ""-"") ``` And some examples: ```python for example in ( ""hello"", ""hello.csv"", ""hello-and-so-on.csv"", ""hello-.csv"", ""hello--and--so--on-.csv"", ""hello.csv."", ""hello.csv.-"", ""hello.csv.--"", ): print(example) print(dot_dash_encode(example)) print(example == dot_dash_decode(dot_dash_encode(example))) print() ``` Outputs: ``` hello hello True hello.csv hello-.csv True hello-and-so-on.csv hello--and--so--on-.csv True hello-.csv hello---.csv True hello--and--so--on-.csv hello----and----so----on---.csv True hello.csv. hello-.csv-. True hello.csv.- hello-.csv-.-- True hello.csv.-- hello-.csv-.---- True ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900712981,https://api.github.com/repos/simonw/datasette/issues/1439,900712981,IC_kwDOBm6k_c41r8oV,9599,simonw,2021-08-18T00:09:59Z,2021-08-18T00:12:32Z,OWNER,"So given the original examples, a table called `table.csv` would have the following URLs: - `/db/table-.csv` - the HTML version - `/db/table-.csv.csv` - the CSV version - `/db/table-.csv.json` - the JSON version And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this: - `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version - `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version - `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900714630,https://api.github.com/repos/simonw/datasette/issues/1439,900714630,IC_kwDOBm6k_c41r9CG,9599,simonw,2021-08-18T00:13:33Z,2021-08-18T00:13:33Z,OWNER,"The documentation should definitely cover how table names become URLs, in case any third party code needs to be able to calculate this themselves.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-900715375,https://api.github.com/repos/simonw/datasette/issues/1439,900715375,IC_kwDOBm6k_c41r9Nv,9599,simonw,2021-08-18T00:15:28Z,2021-08-18T00:15:28Z,OWNER,"Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1438#issuecomment-900500824,https://api.github.com/repos/simonw/datasette/issues/1438,900500824,IC_kwDOBm6k_c41rI1Y,9599,simonw,2021-08-17T17:38:16Z,2021-08-17T17:38:16Z,OWNER,"Relevant template code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/templates/query.html#L71 `renderers` comes from here: https://github.com/simonw/datasette/blob/2883098770fc66e50183b2b231edbde20848d4d6/datasette/views/base.py#L593-L608","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900502364,https://api.github.com/repos/simonw/datasette/issues/1438,900502364,IC_kwDOBm6k_c41rJNc,9599,simonw,2021-08-17T17:40:41Z,2021-08-17T17:40:41Z,OWNER,Bug is likely in `path_with_format` itself: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L710-L729,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900513267,https://api.github.com/repos/simonw/datasette/issues/1438,900513267,IC_kwDOBm6k_c41rL3z,9599,simonw,2021-08-17T17:57:05Z,2021-08-17T17:57:05Z,OWNER,"I'm having trouble replicating this bug outside of Vercel. Against Cloud Run: view-source:https://latest.datasette.io/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22 The HTML here is: ```html

    This data as json, ... CSV

    ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900516826,https://api.github.com/repos/simonw/datasette/issues/1438,900516826,IC_kwDOBm6k_c41rMva,9599,simonw,2021-08-17T18:02:27Z,2021-08-17T18:02:27Z,OWNER,"The key difference I can spot between Vercel and Cloud Run is that `+` in a query string gets converted to `%20` by Vercel before it gets to my app, but does not for Cloud Run: ``` # Vercel ~ % curl -s 'https://til.simonwillison.net/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'sql=select%20*%20from%20tunes%20where%20name%20like%20%22%25' b'wise%20maid%25%22%0D%0A', 'raw_path': b'/-/asgi-scope', # Cloud Run ~ % curl -s 'https://latest-with-plugins.datasette.io/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%2' b'2%0D%0A', 'raw_path': b'/-/asgi-scope', ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1438#issuecomment-900518343,https://api.github.com/repos/simonw/datasette/issues/1438,900518343,IC_kwDOBm6k_c41rNHH,9599,simonw,2021-08-17T18:04:42Z,2021-08-17T18:04:42Z,OWNER,Here's how `request.query_string` works: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/asgi.py#L86-L88,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions, https://github.com/simonw/datasette/issues/1293#issuecomment-901475812,https://api.github.com/repos/simonw/datasette/issues/1293,901475812,IC_kwDOBm6k_c41u23k,9599,simonw,2021-08-18T22:41:19Z,2021-08-18T22:41:19Z,OWNER,"> Maybe I split this out into a separate Python library that gets tested against _every_ SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself? I'm going to do this, and call the Python library `sqlite-explain`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results, https://github.com/simonw/datasette/issues/1415#issuecomment-902251316,https://api.github.com/repos/simonw/datasette/issues/1415,902251316,IC_kwDOBm6k_c41x0M0,9599,simonw,2021-08-19T21:14:15Z,2021-08-19T21:14:15Z,OWNER,"https://github.com/ahmetb/cloud-run-faq#how-do-i-continuously-deploy-to-cloud-run suggests the following: > - `roles/run.admin` to deploy applications > - `roles/iam.serviceAccountUser` on the service account that your app will use It also links to https://cloud.google.com/run/docs/reference/iam/roles","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959137143,feature request: document minimum permissions for service account for cloudrun, https://github.com/simonw/datasette/issues/1443#issuecomment-902258509,https://api.github.com/repos/simonw/datasette/issues/1443,902258509,IC_kwDOBm6k_c41x19N,9599,simonw,2021-08-19T21:25:07Z,2021-08-19T21:25:07Z,OWNER,https://docs.datasette.io/en/latest/internals.html#databases,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974995592,datasette.databases should be a documented property, https://github.com/simonw/datasette/pull/1434#issuecomment-902254712,https://api.github.com/repos/simonw/datasette/issues/1434,902254712,IC_kwDOBm6k_c41x1B4,9599,simonw,2021-08-19T21:18:31Z,2021-08-19T21:18:57Z,OWNER,"I deployed a demo to https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app using the mechanism from #1442. e.g. demo here: https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app/fixtures?sql=select+*+from+searchable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",970463436,Enrich arbitrary query results with foreign key links and column descriptions, https://github.com/simonw/datasette/issues/1426#issuecomment-902260338,https://api.github.com/repos/simonw/datasette/issues/1426,902260338,IC_kwDOBm6k_c41x2Zy,9599,simonw,2021-08-19T21:28:25Z,2021-08-19T21:29:40Z,OWNER,"Actually it looks like you can send a `sitemap.xml` to Google using an unauthenticated GET request to: https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP According to https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-902260799,https://api.github.com/repos/simonw/datasette/issues/1426,902260799,IC_kwDOBm6k_c41x2g_,9599,simonw,2021-08-19T21:29:13Z,2021-08-19T21:29:13Z,OWNER,"Bing's equivalent is: https://www.bing.com/webmasters/help/Sitemaps-3b5cf6ed http://www.bing.com/ping?sitemap=FULL_URL_OF_SITEMAP","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1426#issuecomment-902263367,https://api.github.com/repos/simonw/datasette/issues/1426,902263367,IC_kwDOBm6k_c41x3JH,9599,simonw,2021-08-19T21:33:51Z,2021-08-19T21:36:28Z,OWNER,"I was worried about if it's possible to allow access to `/fixtures` but deny access to `/fixtures?sql=...` From various answers on Stack Overflow it looks like this should handle that: ``` User-agent: * Disallow: /fixtures? ``` I could use this for tables too - it may well be OK to access table index pages while still avoiding pagination, facets etc. I think this should block both query strings and row pages while allowing the table page itself: ``` User-agent: * Disallow: /fixtures/searchable? Disallow: /fixtures/searchable/* ``` Could even accompany that with a `sitemap.xml` that explicitly lists all of the tables - which would mean adding sitemaps to Datasette core too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1442#issuecomment-902191150,https://api.github.com/repos/simonw/datasette/issues/1442,902191150,IC_kwDOBm6k_c41xlgu,9599,simonw,2021-08-19T19:43:05Z,2021-08-19T19:43:59Z,OWNER,"Maybe as simple as teaching https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml to run on pushes to ALL branches: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L3-L6 And then quit early if the branch is not in some allow-list. If it IS in the allow-list, use the name of the branch to dynamically construct the name of the Cloud Run service here: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L60 Need to skip the documentation build and deployment stuff for other branches though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902217726,https://api.github.com/repos/simonw/datasette/issues/1442,902217726,IC_kwDOBm6k_c41xr_-,9599,simonw,2021-08-19T20:21:47Z,2021-08-19T20:21:47Z,OWNER,I think the neatest way to implement this would be for the `on -> push -> branches` list to be the list of branches that should be deployed in this way. The rest of the code can react to that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902231018,https://api.github.com/repos/simonw/datasette/issues/1442,902231018,IC_kwDOBm6k_c41xvPq,9599,simonw,2021-08-19T20:42:08Z,2021-08-19T20:42:08Z,OWNER,If I get this working I should document it on https://docs.datasette.io/en/stable/contributing.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902235714,https://api.github.com/repos/simonw/datasette/issues/1442,902235714,IC_kwDOBm6k_c41xwZC,9599,simonw,2021-08-19T20:50:38Z,2021-08-19T20:50:38Z,OWNER,"Would this allow anyone to push a PR to this repo that would result in their code being deployed against my Cloud Run account? I'm reasonably confident that it would not, since the secrets would not be visible to their PR branch.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902239215,https://api.github.com/repos/simonw/datasette/issues/1442,902239215,IC_kwDOBm6k_c41xxPv,9599,simonw,2021-08-19T20:56:46Z,2021-08-19T20:56:46Z,OWNER,"I'm going to only run the tests if it's a push to `main` - that way I can ship demo branches really quickly, even if they don't yet have passing tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1442#issuecomment-902243498,https://api.github.com/repos/simonw/datasette/issues/1442,902243498,IC_kwDOBm6k_c41xySq,9599,simonw,2021-08-19T21:04:01Z,2021-08-19T21:04:01Z,OWNER,That successfully deployed to https://datasette-latest-deploy-this-branch-j7hipcg4aq-uc.a.run.app/ even though the tests failed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos, https://github.com/simonw/datasette/issues/1415#issuecomment-902250361,https://api.github.com/repos/simonw/datasette/issues/1415,902250361,IC_kwDOBm6k_c41xz95,9599,simonw,2021-08-19T21:12:28Z,2021-08-19T21:12:28Z,OWNER,I would love to know this too! I always find figuring out minimal permissions to be really difficult.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959137143,feature request: document minimum permissions for service account for cloudrun, https://github.com/simonw/datasette/issues/894#issuecomment-902375088,https://api.github.com/repos/simonw/datasette/issues/894,902375088,IC_kwDOBm6k_c41ySaw,9599,simonw,2021-08-20T02:07:13Z,2021-08-20T02:07:26Z,OWNER,Maybe `?_sort_numeric=col` and `?_sort_numeric_desc=col` would be better here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/894#issuecomment-902375388,https://api.github.com/repos/simonw/datasette/issues/894,902375388,IC_kwDOBm6k_c41ySfc,9599,simonw,2021-08-20T02:07:53Z,2021-08-20T02:07:53Z,OWNER,I could add these sorting links to the cog menu for any `TEXT` columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/1473#issuecomment-922363640,https://api.github.com/repos/simonw/datasette/issues/1473,922363640,IC_kwDOBm6k_c42-ib4,192568,mroswell,2021-09-18T19:45:47Z,2021-09-18T19:45:47Z,CONTRIBUTOR,"An update, if I remove the `img` tag and replace it with the text, ""Safer or Toxic?"" it links to the right place. Also, if I keep things exactly as they are, and it improperly, but consistently goes to the `undefined` page, on THAT 404 page, a click on the image properly clicks through to the www.SaferOrToxic.org page. Weird stuff.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",999902754,base logo link visits `undefined` rather than href url, https://github.com/simonw/datasette/issues/1473#issuecomment-922394999,https://api.github.com/repos/simonw/datasette/issues/1473,922394999,IC_kwDOBm6k_c42-qF3,192568,mroswell,2021-09-19T00:44:39Z,2021-09-19T00:45:32Z,CONTRIBUTOR,"I replaced: ``` ``` with: ``` ``` I'd still love to know what caused this (and how to troubleshoot to figure it out), so I'll leave it open for a bit, but I do have a functional logo linking to the Hugo home page, at least locally. I'll likely push tomorrow. (Before trying this, I tried to apply a background image to the `a` tag. That didn't work.) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",999902754,base logo link visits `undefined` rather than href url, https://github.com/simonw/datasette/issues/236#issuecomment-920543967,https://api.github.com/repos/simonw/datasette/issues/236,920543967,IC_kwDOBm6k_c423mLf,164214,sethvincent,2021-09-16T03:19:08Z,2021-09-16T03:19:08Z,NONE,":wave: I just put together a small example using the lambda container image support: https://github.com/sethvincent/datasette-aws-lambda-example It uses mangum and AWS's [python runtime interface client](https://github.com/aws/aws-lambda-python-runtime-interface-client) to handle the lambda event stuff. I'd be happy to help with a publish plugin for AWS lambda as I plan to use this for upcoming projects. The example uses the [serverless](https://www.serverless.com) cli for deployment but there might be a more suitable deployment approach for the plugin. It would be cool if users didn't have to install anything additional other than the aws cli and its associated config/credentials setup.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin, https://github.com/simonw/datasette/issues/236#issuecomment-922075480,https://api.github.com/repos/simonw/datasette/issues/236,922075480,IC_kwDOBm6k_c429cFY,9599,simonw,2021-09-17T20:54:13Z,2021-09-17T20:54:13Z,OWNER,"That's so useful @sethvincent! Really interesting reading your code there, especially clever how you're using the `base_url` config. I'd be very interested to see what your demo looks like without using serverless - completely agree that the less additional dependencies there are for this the better. I'm also very interested in figuring out a way to run Datasette in Lambda but with the SQLite database on an EFS volume. Do you have a feel for how hard that would be?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin, https://github.com/simonw/datasette/pull/1448#issuecomment-906414811,https://api.github.com/repos/simonw/datasette/issues/1448,906414811,IC_kwDOBm6k_c42Bsrb,22429695,codecov[bot],2021-08-26T13:29:02Z,2021-08-26T13:29:02Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1448](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a211747) into [main](https://codecov.io/gh/simonw/datasette/commit/a1a33bb5822214be1cebd98cd858b2058d91a4aa?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a1a33bb) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1448/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1448 +/- ## ======================================= Coverage 91.82% 91.82% ======================================= Files 34 34 Lines 4418 4418 ======================================= Hits 4057 4057 Misses 361 361 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a1a33bb...a211747](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",980228553,"Update pluggy requirement from ~=0.13.0 to >=0.13,<1.1", https://github.com/simonw/datasette/issues/1449#issuecomment-907537693,https://api.github.com/repos/simonw/datasette/issues/1449,907537693,IC_kwDOBm6k_c42F-0d,9599,simonw,2021-08-28T00:31:26Z,2021-08-28T00:31:26Z,OWNER,Terminology question: is it correct to call these subcommands or should they be commands? `publish_subcommand()` adds subcommands of the format `datasette publish X` - but are we instead adding commands with this new one?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907537366,https://api.github.com/repos/simonw/datasette/issues/1449,907537366,IC_kwDOBm6k_c42F-vW,9599,simonw,2021-08-28T00:29:16Z,2021-08-28T00:29:29Z,OWNER,"The closest plugin hook to this right now is [publish_subcommand](https://docs.datasette.io/en/stable/plugin_hooks.html#publish-subcommand-publish) - which looks like this: ```python @hookimpl def publish_subcommand(publish): @publish.command() @add_common_publish_arguments_and_options @click.option( ""-k"", ""--api_key"", help=""API key for talking to my hosting provider"", ) def my_hosting_provider(...): ``` But there are also several plugin hooks with `register_` prefixes, which may be a good naming convention to stick to here: `register_output_renderer(datasette)`, `register_routes(datasette)`, `register_facet_classes()`, `register_magic_parameters(datasette)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907537610,https://api.github.com/repos/simonw/datasette/issues/1449,907537610,IC_kwDOBm6k_c42F-zK,9599,simonw,2021-08-28T00:30:51Z,2021-08-28T00:30:51Z,OWNER,"There's also the option for plugins to muck around with existing registered commands - this could get a bit untidy if multiple plugins try to do it, but being able to replace `serve` with a fresh implementation that adds an additional command-line option before calling back to the original might open up some interesting possibilities.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907542214,https://api.github.com/repos/simonw/datasette/issues/1449,907542214,IC_kwDOBm6k_c42F_7G,9599,simonw,2021-08-28T01:02:38Z,2021-08-28T01:02:38Z,OWNER,"Partial prototype of `datasette-verify`: ```python from datasette import hookimpl import click @hookimpl def register_commands(cli): from datasette.cli import sqlite_extensions @cli.command() @click.argument(""files"", type=click.Path(exists=True), nargs=-1) @sqlite_extensions def verify(files, sqlite_extensions): ""Verify that files can be opened by Datasette"" for file in files: print(file) ``` I had to move the `from datasette.cli import sqlite_extensions` inside the hook function to avoid a circular import.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907538940,https://api.github.com/repos/simonw/datasette/issues/1449,907538940,IC_kwDOBm6k_c42F_H8,9599,simonw,2021-08-28T00:39:28Z,2021-08-28T00:39:28Z,OWNER,"Here's the answer to that: ``` ~ % datasette --help Usage: datasette [OPTIONS] COMMAND [ARGS]... Datasette! Options: --version Show the version and exit. --help Show this message and exit. Commands: serve* Serve up specified SQLite database files with a web UI inspect install Install Python packages - e.g. package Package specified SQLite files into a new datasette Docker... plugins List currently available plugins publish Publish specified SQLite database files to the internet... uninstall Uninstall Python packages (e.g. ``` Since it's adding extra things that show up in `--help` under the ""Commands:"" heading, I should call them commands.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907539065,https://api.github.com/repos/simonw/datasette/issues/1449,907539065,IC_kwDOBm6k_c42F_J5,9599,simonw,2021-08-28T00:40:24Z,2021-08-28T00:40:24Z,OWNER,I'm going to call the new hook `register_commands` - since it will allow ambitious plugins to register more than one command if they want to. That's also pleasingly similar to `register_routes`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907539251,https://api.github.com/repos/simonw/datasette/issues/1449,907539251,IC_kwDOBm6k_c42F_Mz,9599,simonw,2021-08-28T00:41:37Z,2021-08-28T00:41:50Z,OWNER,The first example plugin I'm going to build for this will be `datasette verify file.db file2.db` - it will take one or more paths to SQLite files and verify if they can be opened by Datasette or not.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907539668,https://api.github.com/repos/simonw/datasette/issues/1449,907539668,IC_kwDOBm6k_c42F_TU,9599,simonw,2021-08-28T00:44:16Z,2021-08-28T00:44:16Z,OWNER,"Considering this piece of code: https://github.com/simonw/datasette/blob/a1a33bb5822214be1cebd98cd858b2058d91a4aa/datasette/cli.py#L122-L142 I think the hook itself gets called with a single argument, `cli`, which it can then use in the standard Click way to register extra stuff. I can't pass it `datasette` (like I do with `register_routes()`) because the Datasette object itself is instantiated by the `serve` command, which will not have been called.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1450#issuecomment-907540240,https://api.github.com/repos/simonw/datasette/issues/1450,907540240,IC_kwDOBm6k_c42F_cQ,9599,simonw,2021-08-28T00:48:30Z,2021-08-28T00:48:30Z,OWNER,"I'll go with this: ``` % datasette --help Usage: datasette [OPTIONS] COMMAND [ARGS]... Datasette is an open source multi-tool for exploring and publishing data About Datasette: https://datasette.io/ Full documentation: https://docs.datasette.io/ Options: --version Show the version and exit. --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981681138,"Datasette --help should show something more useful than ""Datasette!""", https://github.com/simonw/datasette/issues/1449#issuecomment-907540790,https://api.github.com/repos/simonw/datasette/issues/1449,907540790,IC_kwDOBm6k_c42F_k2,9599,simonw,2021-08-28T00:52:32Z,2021-08-28T00:52:32Z,OWNER,I don't think I can get this new hook to support the handy [--plugins-dir= mechanism](https://docs.datasette.io/en/stable/plugins.html#one-off-plugins-using-plugins-dir) for loading plugins from Python files as opposed to registering them with setuptools - that mechanism is itself implemented inside of code called by `datasette serve` so I don't have a way of taking advantage of it from outside that command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907540928,https://api.github.com/repos/simonw/datasette/issues/1449,907540928,IC_kwDOBm6k_c42F_nA,9599,simonw,2021-08-28T00:53:37Z,2021-08-28T00:53:37Z,OWNER,I'll probably have to use this mechanism for the tests then: https://til.simonwillison.net/pytest/registering-plugins-in-tests,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907543982,https://api.github.com/repos/simonw/datasette/issues/1449,907543982,IC_kwDOBm6k_c42GAWu,9599,simonw,2021-08-28T01:14:27Z,2021-08-28T01:14:27Z,OWNER,"Writing the test for this is proving difficult, because the `cli` module has already been imported when I attempt to register a new plugin - so it doesn't pick up on the additional command registrations. Trying to work around that with `importlib.reload(cli)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907547624,https://api.github.com/repos/simonw/datasette/issues/1449,907547624,IC_kwDOBm6k_c42GBPo,9599,simonw,2021-08-28T01:44:57Z,2021-08-28T01:58:35Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#register-commands-cli,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/issues/1449#issuecomment-907547736,https://api.github.com/repos/simonw/datasette/issues/1449,907547736,IC_kwDOBm6k_c42GBRY,9599,simonw,2021-08-28T01:45:36Z,2021-08-28T01:45:36Z,OWNER,I need to push this out as an alpha so I can release a demo plugin that uses it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands, https://github.com/simonw/datasette/pull/1453#issuecomment-908337614,https://api.github.com/repos/simonw/datasette/issues/1453,908337614,IC_kwDOBm6k_c42JCHO,22429695,codecov[bot],2021-08-30T13:20:26Z,2021-08-30T13:20:26Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1453](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4f492a7) into [main](https://codecov.io/gh/simonw/datasette/commit/67cbf0ae7243431bf13702e6e3ba466b619c4d6f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (67cbf0a) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1453/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1453 +/- ## ======================================= Coverage 91.83% 91.83% ======================================= Files 34 34 Lines 4421 4421 ======================================= Hits 4060 4060 Misses 361 361 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [67cbf0a...4f492a7](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",982780906,Bump black from 21.7b0 to 21.8b0, https://github.com/simonw/datasette/issues/1446#issuecomment-908832938,https://api.github.com/repos/simonw/datasette/issues/1446,908832938,IC_kwDOBm6k_c42K7Cq,9599,simonw,2021-08-31T01:54:59Z,2021-08-31T01:54:59Z,OWNER,I used the sticky footer mechanism in `datasette.app`: https://github.com/simonw/datasette.app/issues/3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978357984,Modify base.html template to support optional sticky footer, https://github.com/simonw/datasette/pull/1455#issuecomment-913001282,https://api.github.com/repos/simonw/datasette/issues/1455,913001282,IC_kwDOBm6k_c42a0tC,51016,ctb,2021-09-04T16:31:24Z,2021-09-04T16:31:24Z,CONTRIBUTOR,I love it! maybe 'researchers' instead? Or 'scientists and researchers'?,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988325628,Add scientists to target groups, https://github.com/simonw/datasette/pull/1455#issuecomment-913001298,https://api.github.com/repos/simonw/datasette/issues/1455,913001298,IC_kwDOBm6k_c42a0tS,9599,simonw,2021-09-04T16:31:32Z,2021-09-04T16:31:32Z,OWNER,Great idea!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988325628,Add scientists to target groups, https://github.com/simonw/datasette/pull/1455#issuecomment-913001416,https://api.github.com/repos/simonw/datasette/issues/1455,913001416,IC_kwDOBm6k_c42a0vI,9599,simonw,2021-09-04T16:32:21Z,2021-09-04T16:32:21Z,OWNER,I'll add researchers too.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988325628,Add scientists to target groups, https://github.com/simonw/datasette/issues/1459#issuecomment-913218494,https://api.github.com/repos/simonw/datasette/issues/1459,913218494,IC_kwDOBm6k_c42bpu-,9599,simonw,2021-09-05T19:58:51Z,2021-09-05T19:59:15Z,OWNER,"This idea makes sense to me. There's actually an existing option that takes a path, called `--get` - it returns the HTML or JSON for that oath directly to the console, eg `datasette my.db --get /mydb/mytable.json` So... one option would be to allow combining that with `-o` to open that URL in the browser: datasette my.db -o --get /mydb So some options here are: - `datasette my.db --open-url /mydb` - `datasette my.db --open-path /mydb` - `datasette my.db --open --get /mydb` I quite like that last combination option, mainly to avoid adding even more command options. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988556488,suggestion: allow `datasette --open` to take a relative URL, https://github.com/simonw/datasette/issues/1461#issuecomment-914439356,https://api.github.com/repos/simonw/datasette/issues/1461,914439356,IC_kwDOBm6k_c42gTy8,9599,simonw,2021-09-07T16:11:37Z,2021-09-07T16:11:37Z,OWNER,"``` (datasette) datasette % blacken-docs docs/*.rst docs/authentication.rst: Rewriting... docs/internals.rst:169: code block parse error Cannot parse: 14:0: docs/plugin_hooks.rst:251: code block parse error Cannot parse: 6:4: ] docs/plugin_hooks.rst:312: code block parse error Cannot parse: 38:0: docs/spatialite.rst: Rewriting... docs/testing_plugins.rst:135: code block parse error Cannot parse: 5:0: docs/writing_plugins.rst: Rewriting... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",989986586,Try blacken-docs, https://github.com/simonw/datasette/issues/1461#issuecomment-914440282,https://api.github.com/repos/simonw/datasette/issues/1461,914440282,IC_kwDOBm6k_c42gUBa,9599,simonw,2021-09-07T16:12:57Z,2021-09-07T16:12:57Z,OWNER,"Here's the diff it produced from that first run: ```diff diff --git a/docs/authentication.rst b/docs/authentication.rst index 0d98cf8..8008023 100644 --- a/docs/authentication.rst +++ b/docs/authentication.rst @@ -381,11 +381,7 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so: .. code-block:: python response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - } - }, ""actor"")) + response.set_cookie(""ds_actor"", datasette.sign({""a"": {""id"": ""cleopaws""}}, ""actor"")) Note that you need to pass ``""actor""`` as the namespace to :ref:`datasette_sign`. @@ -412,12 +408,16 @@ To include an expiry, add a ``""e""`` key to the cookie value containing a `base62 expires_at = int(time.time()) + (24 * 60 * 60) response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - }, - ""e"": baseconv.base62.encode(expires_at), - }, ""actor"")) + response.set_cookie( + ""ds_actor"", + datasette.sign( + { + ""a"": {""id"": ""cleopaws""}, + ""e"": baseconv.base62.encode(expires_at), + }, + ""actor"", + ), + ) The resulting cookie will encode data that looks something like this: diff --git a/docs/spatialite.rst b/docs/spatialite.rst index d1b300b..556bad8 100644 --- a/docs/spatialite.rst +++ b/docs/spatialite.rst @@ -58,19 +58,22 @@ Here's a recipe for taking a table with existing latitude and longitude columns, .. code-block:: python import sqlite3 - conn = sqlite3.connect('museums.db') + + conn = sqlite3.connect(""museums.db"") # Lead the spatialite extension: conn.enable_load_extension(True) - conn.load_extension('/usr/local/lib/mod_spatialite.dylib') + conn.load_extension(""/usr/local/lib/mod_spatialite.dylib"") # Initialize spatial metadata for this database: - conn.execute('select InitSpatialMetadata(1)') + conn.execute(""select InitSpatialMetadata(1)"") # Add a geometry column called point_geom to our museums table: conn.execute(""SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);"") # Now update that geometry column with the lat/lon points - conn.execute(''' + conn.execute( + """""" UPDATE museums SET point_geom = GeomFromText('POINT('||""longitude""||' '||""latitude""||')',4326); - ''') + """""" + ) # Now add a spatial index to that column conn.execute('select CreateSpatialIndex(""museums"", ""point_geom"");') # If you don't commit your changes will not be persisted: @@ -186,13 +189,14 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac .. code-block:: python import sqlite3 - conn = sqlite3.connect('places.db') + + conn = sqlite3.connect(""places.db"") # Enable SpatialLite extension conn.enable_load_extension(True) - conn.load_extension('/usr/local/lib/mod_spatialite.dylib') + conn.load_extension(""/usr/local/lib/mod_spatialite.dylib"") # Create the masic countries table - conn.execute('select InitSpatialMetadata(1)') - conn.execute('create table places (id integer primary key, name text);') + conn.execute(""select InitSpatialMetadata(1)"") + conn.execute(""create table places (id integer primary key, name text);"") # Add a MULTIPOLYGON Geometry column conn.execute(""SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);"") # Add a spatial index against the new column @@ -201,13 +205,17 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac from shapely.geometry.multipolygon import MultiPolygon from shapely.geometry import shape import requests - geojson = requests.get('https://data.whosonfirst.org/404/227/475/404227475.geojson').json() + + geojson = requests.get( + ""https://data.whosonfirst.org/404/227/475/404227475.geojson"" + ).json() # Convert to ""Well Known Text"" format - wkt = shape(geojson['geometry']).wkt + wkt = shape(geojson[""geometry""]).wkt # Insert and commit the record - conn.execute(""INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))"", ( - ""Wales"", wkt - )) + conn.execute( + ""INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))"", + (""Wales"", wkt), + ) conn.commit() Querying polygons using within() diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst index bd60a4b..5af01f6 100644 --- a/docs/writing_plugins.rst +++ b/docs/writing_plugins.rst @@ -18,9 +18,10 @@ The quickest way to start writing a plugin is to create a ``my_plugin.py`` file from datasette import hookimpl + @hookimpl def prepare_connection(conn): - conn.create_function('hello_world', 0, lambda: 'Hello world!') + conn.create_function(""hello_world"", 0, lambda: ""Hello world!"") If you save this in ``plugins/my_plugin.py`` you can then start Datasette like this:: @@ -60,22 +61,18 @@ The example consists of two files: a ``setup.py`` file that defines the plugin: from setuptools import setup - VERSION = '0.1' + VERSION = ""0.1"" setup( - name='datasette-plugin-demos', - description='Examples of plugins for Datasette', - author='Simon Willison', - url='https://github.com/simonw/datasette-plugin-demos', - license='Apache License, Version 2.0', + name=""datasette-plugin-demos"", + description=""Examples of plugins for Datasette"", + author=""Simon Willison"", + url=""https://github.com/simonw/datasette-plugin-demos"", + license=""Apache License, Version 2.0"", version=VERSION, - py_modules=['datasette_plugin_demos'], - entry_points={ - 'datasette': [ - 'plugin_demos = datasette_plugin_demos' - ] - }, - install_requires=['datasette'] + py_modules=[""datasette_plugin_demos""], + entry_points={""datasette"": [""plugin_demos = datasette_plugin_demos""]}, + install_requires=[""datasette""], ) And a Python module file, ``datasette_plugin_demos.py``, that implements the plugin: @@ -88,12 +85,12 @@ And a Python module file, ``datasette_plugin_demos.py``, that implements the plu @hookimpl def prepare_jinja2_environment(env): - env.filters['uppercase'] = lambda u: u.upper() + env.filters[""uppercase""] = lambda u: u.upper() @hookimpl def prepare_connection(conn): - conn.create_function('random_integer', 2, random.randint) + conn.create_function(""random_integer"", 2, random.randint) Having built a plugin in this way you can turn it into an installable package using the following command:: @@ -123,11 +120,13 @@ To bundle the static assets for a plugin in the package that you publish to PyPI .. code-block:: python - package_data={ - 'datasette_plugin_name': [ - 'static/plugin.js', - ], - }, + package_data = ( + { + ""datasette_plugin_name"": [ + ""static/plugin.js"", + ], + }, + ) Where ``datasette_plugin_name`` is the name of the plugin package (note that it uses underscores, not hyphens) and ``static/plugin.js`` is the path within that package to the static file. @@ -152,11 +151,13 @@ Templates should be bundled for distribution using the same ``package_data`` mec .. code-block:: python - package_data={ - 'datasette_plugin_name': [ - 'templates/my_template.html', - ], - }, + package_data = ( + { + ""datasette_plugin_name"": [ + ""templates/my_template.html"", + ], + }, + ) You can also use wildcards here such as ``templates/*.html``. See `datasette-edit-schema `__ for an example of this pattern. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",989986586,Try blacken-docs, https://github.com/simonw/datasette/issues/1461#issuecomment-914441037,https://api.github.com/repos/simonw/datasette/issues/1461,914441037,IC_kwDOBm6k_c42gUNN,9599,simonw,2021-09-07T16:13:59Z,2021-09-07T16:13:59Z,OWNER,"I don't think I'll adopt it for this project. For example, here: ```diff response = Response.redirect(""/"") - response.set_cookie(""ds_actor"", datasette.sign({ - ""a"": { - ""id"": ""cleopaws"" - } - }, ""actor"")) + response.set_cookie(""ds_actor"", datasette.sign({""a"": {""id"": ""cleopaws""}}, ""actor"")) ``` I chose to use the multi-line version to help emphasize the structure - the single-line replacement loses that. I think I'll continue to make my own editorial choices about how the code examples are laid out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",989986586,Try blacken-docs, https://github.com/simonw/datasette/issues/1462#issuecomment-914644260,https://api.github.com/repos/simonw/datasette/issues/1462,914644260,IC_kwDOBm6k_c42hF0k,9599,simonw,2021-09-07T21:34:32Z,2021-09-07T21:34:32Z,OWNER,"I think this is a setting. There are two relevant settings at the moment: ``` ""template_debug"": false, ""trace_debug"": false, ``` For consistence then this should be called `something_debug` - but do I want a single setting that exposes the `_internal` database and adds those debug options to the menu, or do I want those as two separate settings? - `internal_debug` to enable access to that `_internal` database - `menu_debug` for those menu options?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990367646,"Separate out ""debug"" options from ""root"" options", https://github.com/simonw/datasette/pull/1463#issuecomment-915229323,https://api.github.com/repos/simonw/datasette/issues/1463,915229323,IC_kwDOBm6k_c42jUqL,22429695,codecov[bot],2021-09-08T13:15:26Z,2021-10-13T21:52:23Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1463](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2dd94c5) into [main](https://codecov.io/gh/simonw/datasette/commit/0d5cc20aeffa3537cfc9296d01ec24b9c6e23dcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0d5cc20) will **decrease** coverage by `0.00%`. > The diff coverage is `n/a`. > :exclamation: Current head 2dd94c5 differs from pull request most recent head 481212c. Consider uploading reports for the commit 481212c to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1463/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1463 +/- ## ========================================== - Coverage 91.83% 91.83% -0.01% ========================================== Files 34 34 Lines 4422 4421 -1 ========================================== - Hits 4061 4060 -1 Misses 361 361 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1463/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <0.00%> (-0.01%)` | :arrow_down: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [e1012e7...481212c](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991121619,"Update beautifulsoup4 requirement from <4.10.0,>=4.8.1 to >=4.8.1,<4.11.0", https://github.com/simonw/datasette/issues/1464#issuecomment-915279711,https://api.github.com/repos/simonw/datasette/issues/1464,915279711,IC_kwDOBm6k_c42jg9f,51016,ctb,2021-09-08T14:16:49Z,2021-09-08T14:16:49Z,CONTRIBUTOR,on commit d57ab156b35ec642,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures, https://github.com/simonw/datasette/issues/1464#issuecomment-915299013,https://api.github.com/repos/simonw/datasette/issues/1464,915299013,IC_kwDOBm6k_c42jlrF,7476523,bobwhitelock,2021-09-08T14:40:28Z,2021-09-08T14:40:28Z,CONTRIBUTOR,"What are the full errors you're getting? This *may* be the same issue as described in https://github.com/simonw/datasette/pull/1223 - essentially the test suite (and corresponding Datasette features I assume) are by default implicitly dependent on your Sqlite installation having been compiled with the `SQLITE_ENABLE_FTS3_PARENTHESIS` option. If this is the same issue then I think this can be fixed either by recompiling with that option or (probably more easily) by running `pip install pysqlite3-binary`, which will be used in preference to your system Sqlite installation and has this option enabled. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures, https://github.com/simonw/datasette/issues/1464#issuecomment-915302885,https://api.github.com/repos/simonw/datasette/issues/1464,915302885,IC_kwDOBm6k_c42jmnl,51016,ctb,2021-09-08T14:44:50Z,2021-09-08T14:44:50Z,CONTRIBUTOR,"thanks for the response! full errors attached; excerpt: ``` ... def test_searchmode(table_metadata, querystring, expected_rows): with make_app_client( metadata={""databases"": {""fixtures"": {""tables"": {""searchable"": table_metadata}}}} ) as client: response = client.get(""/fixtures/searchable.json?"" + querystring) > assert expected_rows == response.json[""rows""] E AssertionError: assert [[1, 'barry c...sel', 'puma']] == [] E Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther'] E Use -v to get the full diff /Users/t/dev/datasette/tests/test_api.py:1115: AssertionError ``` [errors.txt](https://github.com/simonw/datasette/files/7129719/errors.txt) A quick scan of #1223 suggests you're right. Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures, https://github.com/simonw/datasette/issues/1464#issuecomment-915343886,https://api.github.com/repos/simonw/datasette/issues/1464,915343886,IC_kwDOBm6k_c42jwoO,7476523,bobwhitelock,2021-09-08T15:32:06Z,2021-09-08T15:32:06Z,CONTRIBUTOR,"Thanks, that does look similar! > Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later. Ah that makes sense, I guess that's why this isn't just always installed already. I wonder if a possible solution to this issue could be doing feature detection on whether this feature is supported by the current Sqlite version, and if not these tests could be disabled locally? But possibly there's a better way to handle this, will see what @simonw thinks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures, https://github.com/simonw/datasette/issues/1464#issuecomment-917642487,https://api.github.com/repos/simonw/datasette/issues/1464,917642487,IC_kwDOBm6k_c42shz3,51016,ctb,2021-09-12T14:03:09Z,2021-09-12T14:03:09Z,CONTRIBUTOR,"haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures, https://github.com/simonw/datasette/issues/1468#issuecomment-917839801,https://api.github.com/repos/simonw/datasette/issues/1468,917839801,IC_kwDOBm6k_c42tR-5,9599,simonw,2021-09-13T04:54:17Z,2021-09-13T04:54:17Z,OWNER,Here's a already open issue for this: #972,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994390593,Faceting for custom SQL queries, https://github.com/simonw/datasette/issues/1468#issuecomment-917839507,https://api.github.com/repos/simonw/datasette/issues/1468,917839507,IC_kwDOBm6k_c42tR6T,9599,simonw,2021-09-13T04:53:22Z,2021-09-13T04:53:22Z,OWNER,"At the moment this isn't possible - though there's a workaround which is to define a SQL view for the query, at which point facets will be displayed again. I did a lot of the work required to support this when I refactored how facets worked a while back - but to finally implement this I need to refactor the table view and the arbitrary query view to share much more logic than they do at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994390593,Faceting for custom SQL queries, https://github.com/simonw/datasette/issues/1469#issuecomment-917839062,https://api.github.com/repos/simonw/datasette/issues/1469,917839062,IC_kwDOBm6k_c42tRzW,9599,simonw,2021-09-13T04:52:01Z,2021-09-13T04:52:01Z,OWNER,Here's the code at fault: https://github.com/simonw/datasette/blob/b28b6cd2fe97f7e193a235877abeec2c8eb0a821/datasette/static/table.js#L137-L146,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994450961,"Column cog shows ""facet by this"" when already default faceted", https://github.com/simonw/datasette/issues/1466#issuecomment-917840012,https://api.github.com/repos/simonw/datasette/issues/1466,917840012,IC_kwDOBm6k_c42tSCM,9599,simonw,2021-09-13T04:54:59Z,2021-09-13T04:54:59Z,OWNER,Especially relevant now that 0.2.0 is out which is a much higher quality release. https://github.com/simonw/datasette-app/releases/tag/0.2.0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991467558,Add Datasette Desktop to installation documentation, https://github.com/simonw/datasette/issues/1464#issuecomment-918621705,https://api.github.com/repos/simonw/datasette/issues/1464,918621705,IC_kwDOBm6k_c42wQ4J,7476523,bobwhitelock,2021-09-13T22:17:17Z,2021-09-13T22:17:17Z,CONTRIBUTOR,"> haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there? Not certain, but I think tests in CI run on Ubuntu and don't appear to install any additional Sqlite-related dependencies, and so my guess is the version of Sqlite installed by default on Ubuntu has the `SQLITE_ENABLE_FTS3_PARENTHESIS` option enabled and so doesn't run into this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures, https://github.com/simonw/datasette/pull/1453#issuecomment-919135732,https://api.github.com/repos/simonw/datasette/issues/1453,919135732,IC_kwDOBm6k_c42yOX0,49699333,dependabot[bot],2021-09-14T13:10:38Z,2021-09-14T13:10:38Z,CONTRIBUTOR,Superseded by #1471.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",982780906,Bump black from 21.7b0 to 21.8b0, https://github.com/simonw/datasette/pull/1471#issuecomment-919141156,https://api.github.com/repos/simonw/datasette/issues/1471,919141156,IC_kwDOBm6k_c42yPsk,22429695,codecov[bot],2021-09-14T13:16:29Z,2021-10-13T21:12:25Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1471](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (01b05ed) into [main](https://codecov.io/gh/simonw/datasette/commit/0d5cc20aeffa3537cfc9296d01ec24b9c6e23dcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0d5cc20) will **decrease** coverage by `0.00%`. > The diff coverage is `n/a`. > :exclamation: Current head 01b05ed differs from pull request most recent head 847238a. Consider uploading reports for the commit 847238a to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1471/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1471 +/- ## ========================================== - Coverage 91.83% 91.83% -0.01% ========================================== Files 34 34 Lines 4422 4421 -1 ========================================== - Hits 4061 4060 -1 Misses 361 361 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1471/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <0.00%> (-0.01%)` | :arrow_down: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a673a93...847238a](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",996002181,Bump black from 21.7b0 to 21.9b0, https://github.com/simonw/datasette/pull/1481#issuecomment-939074818,https://api.github.com/repos/simonw/datasette/issues/1481,939074818,IC_kwDOBm6k_c43-SUC,9599,simonw,2021-10-08T19:40:23Z,2021-10-08T19:40:23Z,OWNER,"Then I created myself a temporary 3.10 environment using `pipenv` like so: cd /tmp mkdir py310 cd py310 pipenv shell --python /Users/simon/.pyenv/versions/3.10.0/bin/python And used that with my Datasette checkout like so: cd ~/.../datasette pip install -e '.[test]' pytest ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939075686,https://api.github.com/repos/simonw/datasette/issues/1481,939075686,IC_kwDOBm6k_c43-Shm,9599,simonw,2021-10-08T19:42:00Z,2021-10-08T19:42:00Z,OWNER,"Running `pytest -x --pdb` helped me see this error: ``` File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 122, in dispatch_request await self.ds.refresh_schemas() File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 344, in refresh_schemas await self._refresh_schemas() File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 349, in _refresh_schemas await init_internal_db(internal_db) File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/internal_db.py"", line 5, in init_internal_db await db.execute_write( File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 102, in execute_write return await self.execute_write_fn(_inner, block=block) File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 113, in execute_write_fn reply_queue = janus.Queue() File ""/Users/simon/.local/share/virtualenvs/py310-Z8fTATkJ/lib/python3.10/site-packages/janus/__init__.py"", line 39, in __init__ self._async_not_empty = asyncio.Condition(self._async_mutex) File ""/Users/simon/.pyenv/versions/3.10.0/lib/python3.10/asyncio/locks.py"", line 234, in __init__ raise ValueError(""loop argument must agree with lock"") ValueError: loop argument must agree with lock ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939076399,https://api.github.com/repos/simonw/datasette/issues/1481,939076399,IC_kwDOBm6k_c43-Ssv,9599,simonw,2021-10-08T19:43:33Z,2021-10-08T19:43:33Z,OWNER,"So maybe this is an issue with Janus? I'm using https://pypi.org/project/janus/ 0.6.1 which is the latest release, from October 2020.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939078095,https://api.github.com/repos/simonw/datasette/issues/1481,939078095,IC_kwDOBm6k_c43-THP,9599,simonw,2021-10-08T19:47:29Z,2021-10-08T19:47:29Z,OWNER,"Only mention I can find of that ""loop argument must agree with lock"" error is here - which doesn't have any tips for a workaround yet: https://giters.com/django/channels_redis/issues/278","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939078872,https://api.github.com/repos/simonw/datasette/issues/1481,939078872,IC_kwDOBm6k_c43-TTY,9599,simonw,2021-10-08T19:49:08Z,2021-10-08T19:49:08Z,OWNER,"Here's the code that raises that error: https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/asyncio/locks.py#L219-L234 ```python class Condition(_ContextManagerMixin, mixins._LoopBoundMixin): """"""Asynchronous equivalent to threading.Condition. This class implements condition variable objects. A condition variable allows one or more coroutines to wait until they are notified by another coroutine. A new Lock object is created and used as the underlying lock. """""" def __init__(self, lock=None, *, loop=mixins._marker): super().__init__(loop=loop) if lock is None: lock = Lock() elif lock._loop is not self._get_loop(): raise ValueError(""loop argument must agree with lock"") ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939079727,https://api.github.com/repos/simonw/datasette/issues/1481,939079727,IC_kwDOBm6k_c43-Tgv,9599,simonw,2021-10-08T19:50:52Z,2021-10-08T19:50:52Z,OWNER,"And here's the relevant Janus code: https://github.com/aio-libs/janus/blob/d7970f8b76bcac2e087067ca4575ac845e481874/janus/__init__.py#L24-L42 ```python class Queue(Generic[T]): def __init__(self, maxsize: int = 0) -> None: self._loop = current_loop() self._maxsize = maxsize self._init(maxsize) self._unfinished_tasks = 0 self._sync_mutex = threading.Lock() self._sync_not_empty = threading.Condition(self._sync_mutex) self._sync_not_full = threading.Condition(self._sync_mutex) self._all_tasks_done = threading.Condition(self._sync_mutex) self._async_mutex = asyncio.Lock() # ""loop argument must agree with lock"" exception is raised here: self._async_not_empty = asyncio.Condition(self._async_mutex) self._async_not_full = asyncio.Condition(self._async_mutex) self._finished = asyncio.Event() self._finished.set() ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939100803,https://api.github.com/repos/simonw/datasette/issues/1481,939100803,IC_kwDOBm6k_c43-YqD,9599,simonw,2021-10-08T20:33:42Z,2021-10-08T20:33:42Z,OWNER,"There's a tiny chance this could be a bug in Python 3.10 itself - I filed an issue here: https://bugs.python.org/issue45416 - in which I said: > In Python 3.10 it is not possible to instantiate an asyncio.Condition that wraps an asyncio.Lock without raising a ""loop argument must agree with lock"" exception. > > This code raises that exception: > > asyncio.Condition(asyncio.Lock()) > > This worked in previous Python versions. > > Note that the error only occurs if an event loop is running. Here's a simple script that replicates the problem: > > import asyncio > > # This runs without an exception: > print(asyncio.Condition(asyncio.Lock())) > > # This does not work: > async def example(): > print(asyncio.Condition(asyncio.Lock())) > > # This raises ""ValueError: loop argument must agree with lock"": > asyncio.run(example())","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939180313,https://api.github.com/repos/simonw/datasette/issues/1481,939180313,IC_kwDOBm6k_c43-sEZ,9599,simonw,2021-10-08T23:41:39Z,2021-10-08T23:41:39Z,OWNER,I submitted a PR to Janus with a workaround for this: https://github.com/aio-libs/janus/pull/359,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-939185319,https://api.github.com/repos/simonw/datasette/issues/1481,939185319,IC_kwDOBm6k_c43-tSn,9599,simonw,2021-10-09T00:04:54Z,2021-10-09T00:04:54Z,OWNER,"I applied my PR against Janus to my local copy of Datasette like so: pip uninstall janus pip install https://github.com/aio-libs/janus/archive/9e13d3fb74e2c93d7501443b370a455d1b302b1f.zip Then I ran the Datasette tests and got a much happier pass rate. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/issues/1482#issuecomment-939191311,https://api.github.com/repos/simonw/datasette/issues/1482,939191311,IC_kwDOBm6k_c43-uwP,9599,simonw,2021-10-09T00:35:04Z,2021-10-09T00:35:04Z,OWNER,I think that SQLite error message difference was caused by https://github.com/python/cpython/commit/a50e28377bcf37121b55c2de70d95a5386c478f8 or related work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1021550542,Support Python 3.10, https://github.com/simonw/datasette/issues/1480#issuecomment-938171377,https://api.github.com/repos/simonw/datasette/issues/1480,938171377,IC_kwDOBm6k_c4361vx,110420,ghing,2021-10-07T21:33:12Z,2021-10-07T21:33:12Z,CONTRIBUTOR,"Thanks for the reply @simonw. What services have you had better success with than Cloud Run for larger database? Also, what about my issue description makes you think there may be a workaround? Is there any instrumentation I could add to see at which point in the deploy the memory usage spikes? Should I be able to see this whether it's running under Docker locally, or do you suspect this is Cloud Run-specific?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database, https://github.com/simonw/datasette/issues/1470#issuecomment-938124652,https://api.github.com/repos/simonw/datasette/issues/1470,938124652,IC_kwDOBm6k_c436qVs,9599,simonw,2021-10-07T20:17:53Z,2021-10-07T20:18:55Z,OWNER,"Here's the exception: ``` -> params[f""p{len(params)}""] = components[0] (Pdb) list 603 604 # Figure out the SQL for next-based-on-primary-key first 605 next_by_pk_clauses = [] 606 if use_rowid: 607 next_by_pk_clauses.append(f""rowid > :p{len(params)}"") 608 -> params[f""p{len(params)}""] = components[0] 609 else: 610 # Apply the tie-breaker based on primary keys 611 if len(components) == len(pks): 612 param_len = len(params) 613 next_by_pk_clauses.append( ``` Debugger shows that `components` is an empty array, so `components[0]` cannot be resolved: ``` -> params[f""p{len(params)}""] = components[0] (Pdb) params {'search': 'hello'} (Pdb) components [] ``` So the bug is in this code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/views/table.py#L604-L617 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error, https://github.com/simonw/datasette/issues/1470#issuecomment-938131806,https://api.github.com/repos/simonw/datasette/issues/1470,938131806,IC_kwDOBm6k_c436sFe,9599,simonw,2021-10-07T20:28:30Z,2021-10-07T20:28:30Z,OWNER,"On further investigation this isn't related to `_search` at all - it happens when you explicitly sort by `_sort=rowid` and apply a `_next` - https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 works without an error (currently) - https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid shows that error","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error, https://github.com/simonw/datasette/issues/1480#issuecomment-938134038,https://api.github.com/repos/simonw/datasette/issues/1480,938134038,IC_kwDOBm6k_c436soW,9599,simonw,2021-10-07T20:31:46Z,2021-10-07T20:31:46Z,OWNER,"I've had this problem too - my solution was to not use Cloud Run for databases larger than about 2GB, but the way you describe it here makes me think that maybe there is a workaround here which could get it to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database, https://github.com/simonw/datasette/pull/1481#issuecomment-938141121,https://api.github.com/repos/simonw/datasette/issues/1481,938141121,IC_kwDOBm6k_c436uXB,22429695,codecov[bot],2021-10-07T20:42:37Z,2021-10-24T22:19:28Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1481](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (77542e7) into [main](https://codecov.io/gh/simonw/datasette/commit/63886178a649586b403966a27a45881709d2b868?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (6388617) will **decrease** coverage by `0.01%`. > The diff coverage is `n/a`. > :exclamation: Current head 77542e7 differs from pull request most recent head 50005bd. Consider uploading reports for the commit 50005bd to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1481/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1481 +/- ## ========================================== - Coverage 91.83% 91.82% -0.02% ========================================== Files 34 34 Lines 4421 4426 +5 ========================================== + Hits 4060 4064 +4 - Misses 361 362 +1 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.36% <0.00%> (-0.14%)` | :arrow_down: | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.41% <0.00%> (ø)` | | | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.56% <0.00%> (ø)` | | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <0.00%> (+<0.01%)` | :arrow_up: | | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.78% <0.00%> (+0.02%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [6388617...50005bd](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-938142436,https://api.github.com/repos/simonw/datasette/issues/1481,938142436,IC_kwDOBm6k_c436urk,9599,simonw,2021-10-07T20:44:43Z,2021-10-07T20:44:43Z,OWNER,"The 3.10 tests failed a lot. Trying to run this locally: ``` /tmp % pyenv install 3.10 python-build: definition not found: 3.10 The following versions contain `3.10' in the name: 3.10.0a6 3.10-dev miniconda-3.10.1 miniconda3-3.10.1 See all available versions with `pyenv install --list'. If the version you need is missing, try upgrading pyenv: brew update && brew upgrade pyenv ``` So trying: brew update && brew upgrade pyenv Then did this: ``` /tmp % brew upgrade pyenv ==> Upgrading 1 outdated package: pyenv 1.2.24.1 -> 2.1.0 ``` This decided to upgrade everything by downloaded everything on the internet. Aah, Homebrew. But it looks like I have `3.10.0` available to `pyenv` now. ``` /tmp % pyenv install 3.10.0 python-build: use openssl@1.1 from homebrew python-build: use readline from homebrew Downloading Python-3.10.0.tar.xz... -> https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tar.xz Installing Python-3.10.0... ... ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/issues/111#issuecomment-923106887,https://api.github.com/repos/simonw/datasette/issues/111,923106887,IC_kwDOBm6k_c43BX5H,9599,simonw,2021-09-20T16:58:39Z,2021-09-20T16:58:39Z,OWNER,Still a good idea today too! Would be great for https://cdc-vaccination-history.datasette.io/ for example.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/pull/1476#issuecomment-923979964,https://api.github.com/repos/simonw/datasette/issues/1476,923979964,IC_kwDOBm6k_c43EtC8,22429695,codecov[bot],2021-09-21T13:18:22Z,2021-09-21T13:18:22Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1476](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2eb01ff) into [main](https://codecov.io/gh/simonw/datasette/commit/b28b6cd2fe97f7e193a235877abeec2c8eb0a821?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b28b6cd) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1476/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1476 +/- ## ======================================= Coverage 91.83% 91.83% ======================================= Files 34 34 Lines 4421 4421 ======================================= Hits 4060 4060 Misses 361 361 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b28b6cd...2eb01ff](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1002459220,"Update pytest-xdist requirement from <2.4,>=2.2.1 to >=2.2.1,<2.5", https://github.com/simonw/datasette/issues/111#issuecomment-924432643,https://api.github.com/repos/simonw/datasette/issues/111,924432643,IC_kwDOBm6k_c43GbkD,9599,simonw,2021-09-21T22:23:23Z,2021-09-21T22:23:23Z,OWNER,I'm going to use https://github.com/dateutil/dateutil for this - it's been maintained constantly (by an evolving team of contributors) [since 2003](https://github.com/dateutil/dateutil/commit/68ae2757ae15c84bf947d47a82a314b3b975bc9b) and is a very trustworthy dependency.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924437942,https://api.github.com/repos/simonw/datasette/issues/111,924437942,IC_kwDOBm6k_c43Gc22,9599,simonw,2021-09-21T22:32:59Z,2021-09-21T22:47:07Z,OWNER,"Side-note: Django 4.0 [will switch](https://docs.djangoproject.com/en/dev/releases/4.0/#zoneinfo-default-timezone-implementation) from using `pytz` to using the standard library `zoneinfo` module introduced in Python 3.9, which has a backport that works as far back as 3.6: https://github.com/pganssle/zoneinfo (https://pypi.org/project/backports.zoneinfo/) If I need to handle timezones I'll use that, but I think I can get away without it? Django does this: https://github.com/django/django/blob/b0ed619303d2fb723330ca9efa3acf23d49f1d19/setup.cfg#L39-L43 ``` install_requires = asgiref >= 3.3.2 backports.zoneinfo; python_version<""3.9"" sqlparse >= 0.2.2 tzdata; sys_platform == 'win32' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924435971,https://api.github.com/repos/simonw/datasette/issues/111,924435971,IC_kwDOBm6k_c43GcYD,9599,simonw,2021-09-21T22:29:15Z,2021-09-21T22:29:49Z,OWNER,"So this is a metadata key called `updated` which can be applied at the table, database or instance level. It is represented as a `.isoformat()` timestamp. Question: should I support just the date - `yyyy-mm-dd` - in addition to the datetime? I think so. I can easily imagine situations where the exact time of day that a change was made hasn't been recorded, but the overall date is known. But in that case, should the `updated` key sometimes be `yyyy-mm-dd` and sometimes be the full isoformat datetime? Or should there be an `updated_date` key that's used for just the date?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924438481,https://api.github.com/repos/simonw/datasette/issues/111,924438481,IC_kwDOBm6k_c43Gc_R,9599,simonw,2021-09-21T22:34:03Z,2021-09-21T22:34:21Z,OWNER,"The simplest possible version of this: it's always represented as a UTC ISO datetime, like this: ""updated"": ""2020-10-31T12:00:00+00:00"" Later versions of Datasette could extend this to handle other timezones or support just the date (though that's a backwards incompatible change so probably better to decide on the date thing right now).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/111#issuecomment-924443089,https://api.github.com/repos/simonw/datasette/issues/111,924443089,IC_kwDOBm6k_c43GeHR,9599,simonw,2021-09-21T22:45:14Z,2021-09-21T22:45:26Z,OWNER,"The audiences I care about here are: - Producers of this timestamp - generally that will be users who are using `datasette publish` to share their data - Human consumers of this timestamp - end users who look at a Datasette site and want to know how recent the data is - Machine consumers of this timestamp - API integrations that might want to check if a Datasette instance has been updated before downloading new data For producers I think there are going to be two categories. The first is users who run ""publish"" and want the site to reflect when they did so (probably using `--updated=now` when they publish). The second are users who are willing to spend more time thinking about this - for example my various git scraping projects where I want to use a date derived from the git history. For humans... I'd like to be able to calculate a relative time (3 hours ago) in addition to showing the display time, because that helps avoid confusion over timezones. For machine consumers, it might be nice to offer the option of a calculated Unix timestamp-since-1970, since those can be easier to work with in some languages than running a full ISO date parser.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/datasette/issues/1470#issuecomment-939386591,https://api.github.com/repos/simonw/datasette/issues/1470,939386591,IC_kwDOBm6k_c43_ebf,9599,simonw,2021-10-10T01:17:34Z,2021-10-10T01:17:34Z,OWNER,I'll open a separate issue for removing `_next=` when running a search.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error, https://github.com/simonw/datasette/issues/1479#issuecomment-930071625,https://api.github.com/repos/simonw/datasette/issues/1479,930071625,IC_kwDOBm6k_c43b8RJ,76450761,kirajano,2021-09-29T11:01:30Z,2021-09-29T11:01:30Z,NONE,"Thanks, but this one has a different error type. Unfortunately, still not working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish", https://github.com/simonw/datasette/issues/1479#issuecomment-929927144,https://api.github.com/repos/simonw/datasette/issues/1479,929927144,IC_kwDOBm6k_c43bY_o,1244799,soobrosa,2021-09-29T07:49:40Z,2021-09-29T07:49:40Z,NONE,"My search yielded these four entries: https://github.com/simonw/datasette/issues?q=PermissionError%3A+%5BWinError+32%5D+ Maybe this is the closet hit? https://github.com/simonw/datasette/issues/744 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish", https://github.com/simonw/datasette/issues/1479#issuecomment-932807859,https://api.github.com/repos/simonw/datasette/issues/1479,932807859,IC_kwDOBm6k_c43mYSz,9599,simonw,2021-10-02T19:22:35Z,2021-10-02T19:22:35Z,OWNER,"I'm pretty sure this is a Windows issue, not a Fly issue. I imagine it affects other forms of `datasette publish` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish", https://github.com/simonw/datasette/issues/1479#issuecomment-932808043,https://api.github.com/repos/simonw/datasette/issues/1479,932808043,IC_kwDOBm6k_c43mYVr,9599,simonw,2021-10-02T19:23:52Z,2021-10-02T19:23:52Z,OWNER,I suspect the root cause of this may be in this code: https://github.com/simonw/datasette/blob/63886178a649586b403966a27a45881709d2b868/datasette/utils/__init__.py#L673-L677,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish", https://github.com/simonw/datasette/issues/1479#issuecomment-932808216,https://api.github.com/repos/simonw/datasette/issues/1479,932808216,IC_kwDOBm6k_c43mYYY,9599,simonw,2021-10-02T19:25:09Z,2021-10-02T19:25:09Z,OWNER,"Actually no, from that stack trace you provided: ``` File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 610, in invoke return callback(*args, **kwargs) File ""c:\users\grott\anaconda3\lib\site-packages\datasette\cli.py"", line 283, in package call(args) File ""c:\users\grott\anaconda3\lib\contextlib.py"", line 119, in __exit__ next(self.gen) File ""c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py"", line 451, in temporary_docker_directory tmp.cleanup() ``` It looks like the problem occurs here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L449-L452","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish", https://github.com/simonw/datasette/issues/1380#issuecomment-953334718,https://api.github.com/repos/simonw/datasette/issues/1380,953334718,IC_kwDOBm6k_c440ru-,813732,glasnt,2021-10-27T21:45:04Z,2021-10-27T21:45:04Z,CONTRIBUTOR,"I am also getting this issue, using the currently most recent version of datasette ``` $ datasette --version datasette, version 0.59.1 ``` If I run `datasette` within just a folder of files, ``` $ datasette serve . ``` Adding new files while datasette is running shows no new files, and removing files causes datasette to return 500 errors. ``` home Error 500 [Errno 2] No such file or directory: 'mydatabase.db' Powered by Datasette ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder, https://github.com/simonw/datasette/issues/1380#issuecomment-953366110,https://api.github.com/repos/simonw/datasette/issues/1380,953366110,IC_kwDOBm6k_c440zZe,813732,glasnt,2021-10-27T22:48:55Z,2021-10-27T22:48:55Z,CONTRIBUTOR,"It looks like if the files argument is a directory, `config_dir` is set, but files in that folder are only loaded into `self.files` at the `Datasette` class initialisation. I tried seeing if I could get `--reload` to work, but I'm getting issues trying to use that command when specifying a directory, as the command `serve` ends up in the files list(?): ``` datasette serve . --reload Error: Invalid value for '[FILES]...': Path 'serve' does not exist. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder, https://github.com/simonw/datasette/issues/1497#issuecomment-953508979,https://api.github.com/repos/simonw/datasette/issues/1497,953508979,IC_kwDOBm6k_c441WRz,9599,simonw,2021-10-28T05:13:49Z,2021-10-28T05:13:49Z,OWNER,Wrote about this in my weeknotes: https://simonwillison.net/2021/Oct/28/weeknotes-kubernetes-web-components/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/pull/1495#issuecomment-954384496,https://api.github.com/repos/simonw/datasette/issues/1495,954384496,IC_kwDOBm6k_c444sBw,536941,fgregg,2021-10-29T03:07:13Z,2021-10-29T03:07:13Z,CONTRIBUTOR,"okay @simonw, made the requested changes. tests are running locally. i think this is ready for you to look at again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options, https://github.com/simonw/datasette/issues/1253#issuecomment-955384545,https://api.github.com/repos/simonw/datasette/issues/1253,955384545,IC_kwDOBm6k_c448gLh,1449512,dufferzafar,2021-10-30T16:00:42Z,2021-10-30T16:00:42Z,NONE,"Yeah, I was pressing Ctrl + Enter as well. Came here to open this issue and found out Shift + Enter works. @simonw Any way to configure this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826064552,"Capture ""Ctrl + Enter"" or ""⌘ + Enter"" to send SQL query?", https://github.com/simonw/datasette/pull/1485#issuecomment-940023938,https://api.github.com/repos/simonw/datasette/issues/1485,940023938,IC_kwDOBm6k_c44B6CC,22429695,codecov[bot],2021-10-11T13:18:16Z,2021-10-11T13:18:16Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1485](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (be9aed9) into [main](https://codecov.io/gh/simonw/datasette/commit/0d5cc20aeffa3537cfc9296d01ec24b9c6e23dcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0d5cc20) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1485/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1485 +/- ## ======================================= Coverage 91.83% 91.83% ======================================= Files 34 34 Lines 4422 4422 ======================================= Hits 4061 4061 Misses 361 361 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0d5cc20...be9aed9](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1022688960,"Update pytest-timeout requirement from <1.5,>=1.4.2 to >=1.4.2,<2.1", https://github.com/simonw/datasette/issues/1432#issuecomment-941002127,https://api.github.com/repos/simonw/datasette/issues/1432,941002127,IC_kwDOBm6k_c44Fo2P,5802411,ashishdotme,2021-10-12T13:14:31Z,2021-10-12T13:14:39Z,NONE,"Any workaround for making it work with datasette-publish-vercel. Currently getting below error module initialization error: __init__() got an unexpected keyword argument 'config' module initialization error __init__() got an unexpected keyword argument 'config'","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/issues/1432#issuecomment-941585767,https://api.github.com/repos/simonw/datasette/issues/1432,941585767,IC_kwDOBm6k_c44H3Vn,5802411,ashishdotme,2021-10-12T21:23:19Z,2021-10-12T21:23:19Z,NONE,"Nevermind, had to remove the branch argument in the workflow to make vercel publish work","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/pull/1487#issuecomment-942722595,https://api.github.com/repos/simonw/datasette/issues/1487,942722595,IC_kwDOBm6k_c44MM4j,9599,simonw,2021-10-13T21:08:53Z,2021-10-13T21:08:53Z,OWNER,Thanks for this!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1023245060,"Added instructions for installing plugins via pipx, #1486", https://github.com/simonw/datasette/issues/1469#issuecomment-942725632,https://api.github.com/repos/simonw/datasette/issues/1469,942725632,IC_kwDOBm6k_c44MNoA,9599,simonw,2021-10-13T21:13:30Z,2021-10-13T21:13:30Z,OWNER,"The core problem here is treating the `?_facet=` query string parameters as the point of truth for which facets are currently enabled. Instead, I could use a `data-` attribute on the displayed facets.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994450961,"Column cog shows ""facet by this"" when already default faceted", https://github.com/simonw/datasette/issues/1488#issuecomment-942779926,https://api.github.com/repos/simonw/datasette/issues/1488,942779926,IC_kwDOBm6k_c44Ma4W,9599,simonw,2021-10-13T22:59:05Z,2021-10-13T22:59:05Z,OWNER,This is weird - as far as I can tell `httpx` has included the query string in `raw_path` for well over a year: https://github.com/encode/httpx/commit/8e4a8a1c73f60fe5754f95b308beaa725cb8791d#diff-c9a78eb3b5f5c4fac4e5552165fbdd5320c7e3fadf9eedabcb5461393466c090R235,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1488#issuecomment-942777414,https://api.github.com/repos/simonw/datasette/issues/1488,942777414,IC_kwDOBm6k_c44MaRG,9599,simonw,2021-10-13T22:52:40Z,2021-10-13T22:52:40Z,OWNER,"Upgrading to 0.20.0 gives me lots of the following errors: '{""ok"": false, ""error"": ""Database not found: .json?_sort=relationships"", ""status"": 404, ""title"": null}' It looks like the full query string is now being treated as the name of the database. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1488#issuecomment-942778382,https://api.github.com/repos/simonw/datasette/issues/1488,942778382,IC_kwDOBm6k_c44MagO,9599,simonw,2021-10-13T22:55:01Z,2021-10-13T22:55:01Z,OWNER,"I think the issue is in `route_path()`: ``` > /Users/simon/Dropbox/Development/datasette/datasette/app.py(1182)route_path() -> response = await view(request, send) (Pdb) path '/_memory.json?sql=select+sqlite_version()' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1488#issuecomment-942778673,https://api.github.com/repos/simonw/datasette/issues/1488,942778673,IC_kwDOBm6k_c44Makx,9599,simonw,2021-10-13T22:55:44Z,2021-10-13T22:55:44Z,OWNER,"``` (Pdb) request.scope['path'] '/_memory.json' (Pdb) request.scope['raw_path'] b'/_memory.json?sql=select+sqlite_version()' ``` So `raw_path` now includes the query string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/issues/1488#issuecomment-942782673,https://api.github.com/repos/simonw/datasette/issues/1488,942782673,IC_kwDOBm6k_c44MbjR,9599,simonw,2021-10-13T23:04:54Z,2021-10-13T23:04:54Z,OWNER,"I think this is the change in `httpx` which is causing the bug for me: https://github.com/encode/httpx/commit/ff9813e84dab56f0f3c4ef3a159a4cce8c644a91#diff-0d0cbe9ebcd03cc8c780b0407762540a082f70cc64257f2fcd588cc30f43c15cR96 Previously it was using `path` from `path, _, query = full_path.partition(b""?"")` to populate the `raw_path` key - but it changed to instead using `request.url.raw_path` which presumably implements the logic that includes the query string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'), https://github.com/simonw/datasette/pull/1489#issuecomment-943594712,https://api.github.com/repos/simonw/datasette/issues/1489,943594712,IC_kwDOBm6k_c44PhzY,9599,simonw,2021-10-14T18:04:11Z,2021-10-14T18:04:11Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026379132,"Update pyyaml requirement from ~=5.3 to >=5.3,<7.0", https://github.com/simonw/datasette/pull/1489#issuecomment-943594735,https://api.github.com/repos/simonw/datasette/issues/1489,943594735,IC_kwDOBm6k_c44Phzv,49699333,dependabot[bot],2021-10-14T18:04:12Z,2021-10-14T18:04:12Z,CONTRIBUTOR,Looks like this PR is closed. If you re-open it I'll rebase it as long as no-one else has edited it (you can use `@dependabot reopen` if the branch has been deleted).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026379132,"Update pyyaml requirement from ~=5.3 to >=5.3,<7.0", https://github.com/simonw/datasette/pull/1489#issuecomment-943594738,https://api.github.com/repos/simonw/datasette/issues/1489,943594738,IC_kwDOBm6k_c44Phzy,49699333,dependabot[bot],2021-10-14T18:04:13Z,2021-10-14T18:04:13Z,CONTRIBUTOR,"OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting `@dependabot ignore this major version` or `@dependabot ignore this minor version`. You can also ignore all major, minor, or patch releases for a dependency by adding an [`ignore` condition](https://docs.github.com/en/code-security/supply-chain-security/configuration-options-for-dependency-updates#ignore) with the desired `update_types` to your config file. If you change your mind, just re-open this PR and I'll resolve any conflicts on it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026379132,"Update pyyaml requirement from ~=5.3 to >=5.3,<7.0", https://github.com/simonw/datasette/pull/1458#issuecomment-943620649,https://api.github.com/repos/simonw/datasette/issues/1458,943620649,IC_kwDOBm6k_c44PoIp,9599,simonw,2021-10-14T18:38:58Z,2021-10-14T18:38:58Z,OWNER,"This is a great idea, thanks.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988555009,Rework the `--static` documentation a bit, https://github.com/simonw/datasette/pull/1467#issuecomment-943623246,https://api.github.com/repos/simonw/datasette/issues/1467,943623246,IC_kwDOBm6k_c44PoxO,9599,simonw,2021-10-14T18:42:19Z,2021-10-14T18:42:19Z,OWNER,This looks like a good fix to me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991575770,Add Authorization header when CORS flag is set, https://github.com/simonw/datasette/pull/1467#issuecomment-943632697,https://api.github.com/repos/simonw/datasette/issues/1467,943632697,IC_kwDOBm6k_c44PrE5,9599,simonw,2021-10-14T18:54:18Z,2021-10-14T18:54:18Z,OWNER,The test there failed because it turns out there's a whole bunch of places that set the `Access-Control-Allow-Origin` header. I'm going to close this PR and ship a fix that refactors those places to use the same code.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",991575770,Add Authorization header when CORS flag is set, https://github.com/simonw/datasette/pull/1481#issuecomment-944986367,https://api.github.com/repos/simonw/datasette/issues/1481,944986367,IC_kwDOBm6k_c44U1j_,9599,simonw,2021-10-16T19:07:38Z,2021-10-16T19:09:02Z,OWNER,This is blocking an upgrade for the Homebrew Datasette package: https://github.com/Homebrew/homebrew-core/pull/86932,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1481#issuecomment-945020210,https://api.github.com/repos/simonw/datasette/issues/1481,945020210,IC_kwDOBm6k_c44U90y,9599,simonw,2021-10-16T23:19:51Z,2021-10-16T23:19:51Z,OWNER,"Since that Janus PR hasn't been merged yet, one temporary option for a fix would be to entirely vendor the fixed Janus - https://github.com/aio-libs/janus/blob/9e13d3fb74e2c93d7501443b370a455d1b302b1f/janus/__init__.py - since it's only a single module.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10, https://github.com/simonw/datasette/pull/1467#issuecomment-945037884,https://api.github.com/repos/simonw/datasette/issues/1467,945037884,IC_kwDOBm6k_c44VCI8,3058200,jameslittle230,2021-10-17T02:29:06Z,2021-10-17T02:29:06Z,NONE,Yay! Thank you @simonw!!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991575770,Add Authorization header when CORS flag is set, https://github.com/simonw/datasette/issues/1432#issuecomment-945639639,https://api.github.com/repos/simonw/datasette/issues/1432,945639639,IC_kwDOBm6k_c44XVDX,5802411,ashishdotme,2021-10-18T10:44:56Z,2021-10-18T10:44:56Z,NONE,"@simonw I am getting the below issue again now, even after removing branch argument from vercel datasette plugin module initialization error: init() got an unexpected keyword argument 'config' module initialization error init() got an unexpected keyword argument 'config'","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/pull/1494#issuecomment-945763015,https://api.github.com/repos/simonw/datasette/issues/1494,945763015,IC_kwDOBm6k_c44XzLH,22429695,codecov[bot],2021-10-18T13:22:56Z,2021-10-18T13:22:56Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1494](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7a3e3c3) into [main](https://codecov.io/gh/simonw/datasette/commit/ff9ccfb0310501a3b4b4ca24d73246a8eb3e7914?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ff9ccfb) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1494/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1494 +/- ## ======================================= Coverage 91.82% 91.82% ======================================= Files 34 34 Lines 4426 4426 ======================================= Hits 4064 4064 Misses 362 362 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [ff9ccfb...7a3e3c3](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1029100823,"Update pytest-asyncio requirement from <0.16,>=0.10 to >=0.10,<0.17", https://github.com/simonw/datasette/issues/1470#issuecomment-946097058,https://api.github.com/repos/simonw/datasette/issues/1470,946097058,IC_kwDOBm6k_c44ZEui,9599,simonw,2021-10-18T19:30:15Z,2021-10-18T19:30:15Z,OWNER,https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid is fixed now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error, https://github.com/simonw/datasette/issues/1432#issuecomment-946255239,https://api.github.com/repos/simonw/datasette/issues/1432,946255239,IC_kwDOBm6k_c44ZrWH,192568,mroswell,2021-10-18T23:55:25Z,2021-10-18T23:55:25Z,CONTRIBUTOR,"I am getting this when I visit my live Datasette page: ``` This Serverless Function has crashed. Your connection is working correctly. Vercel is working correctly. 500: INTERNAL_SERVER_ERROR Code: FUNCTION_INVOCATION_FAILED ID: ... ``` And in the server logs, I'm getting ``` [GET] /disinfectants/listN 19:53:14:23 module initialization error: __init__() got an unexpected keyword argument 'config' module initialization error __init__() got an unexpected keyword argument 'config' ``` Which is the same error that @ashishdotme reported above. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/issues/1432#issuecomment-946287922,https://api.github.com/repos/simonw/datasette/issues/1432,946287922,IC_kwDOBm6k_c44ZzUy,192568,mroswell,2021-10-19T01:16:41Z,2021-10-19T01:16:41Z,CONTRIBUTOR,"Resolved, with assistance from @ashishdotme (Thank you!) Updated requirements.txt to include: ``` datasette==0.59 datasette-publish-vercel==0.11 sqlite-utils==3.6 ``` Ran: ``` $ pip3 install -r requirements.txt ``` The site is back at work! Yay! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=, https://github.com/simonw/datasette/pull/1500#issuecomment-956225475,https://api.github.com/repos/simonw/datasette/issues/1500,956225475,IC_kwDOBm6k_c44_tfD,22429695,codecov[bot],2021-11-01T13:16:36Z,2021-11-01T13:16:36Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1500](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b7f679) into [main](https://codecov.io/gh/simonw/datasette/commit/2c31d1cd9cd3b63458ccbe391866499fa3f44978?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2c31d1c) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1500/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1500 +/- ## ======================================= Coverage 91.82% 91.82% ======================================= Files 34 34 Lines 4426 4426 ======================================= Hits 4064 4064 Misses 362 362 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2c31d1c...1b7f679](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1041158024,Bump black from 21.9b0 to 21.10b0, https://github.com/simonw/datasette/issues/1493#issuecomment-946360891,https://api.github.com/repos/simonw/datasette/issues/1493,946360891,IC_kwDOBm6k_c44aFI7,9599,simonw,2021-10-19T04:37:27Z,2021-10-19T04:37:27Z,OWNER,"I renamed `/:memory:` to `/_memory` in version 0.55 - https://docs.datasette.io/en/stable/changelog.html#v0-55 But... in 0.59 I stopped following HTTP redirects by default, which is why this used to work and no longer does! So the fix is to update the Homebrew regression test to use this instead: datasette --get '/_memory.json?sql=select+3*5' Thanks for catching this!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1028115674,`--get '/:memory:.json?sql=select+3*5'` error with datasette 0.59, https://github.com/simonw/datasette/issues/1396#issuecomment-946467547,https://api.github.com/repos/simonw/datasette/issues/1396,946467547,IC_kwDOBm6k_c44afLb,72577720,MichaelTiemannOSC,2021-10-19T08:10:26Z,2021-10-19T08:10:26Z,CONTRIBUTOR,"Now that 0.59 has excellent annotated release notes, you can re-confirm this is fixed by updating the published Docker image and checking that these fixes still work ;-)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image", https://github.com/simonw/datasette/issues/942#issuecomment-946493045,https://api.github.com/repos/simonw/datasette/issues/942,946493045,IC_kwDOBm6k_c44alZ1,8451755,kokes,2021-10-19T08:42:39Z,2021-10-19T08:42:39Z,NONE,"@simonw I know this is closed, just found this via the annotated release notes, but I wanted to note this one thing: Not sure how widely used this is, but I've seen CSVW a couple times in the wild. It is trying to address these metadata challenges in a standardised way. See e.g. - https://www.w3.org/TR/tabular-data-primer/#h-documentation-columns - https://w3c.github.io/csvw/tests/ I'm not suggesting you change the syntax you've implemented, just letting you know of this effort by W3C.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json, https://github.com/simonw/datasette/issues/1480#issuecomment-947196177,https://api.github.com/repos/simonw/datasette/issues/1480,947196177,IC_kwDOBm6k_c44dRER,110420,ghing,2021-10-20T00:05:10Z,2021-10-20T00:05:10Z,CONTRIBUTOR,"I was looking through the Dockerfile-generation code to see if there was anything that would cause memory usage to be a lot during deployment. I noticed that the Dockerfile [runs `datasette --inspect`](https://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L354). Is it possible that this is using a lot of memory usage? Or would that come into play when running `gcloud builds submit`, not when it's actually deployed?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database, https://github.com/simonw/datasette/issues/1480#issuecomment-947203725,https://api.github.com/repos/simonw/datasette/issues/1480,947203725,IC_kwDOBm6k_c44dS6N,110420,ghing,2021-10-20T00:21:54Z,2021-10-20T00:21:54Z,CONTRIBUTOR,"This StackOverflow post, [sqlite - Cloud Run: Why does my instance need so much RAM?](https://stackoverflow.com/questions/59812405/cloud-run-why-does-my-instance-need-so-much-ram), points to [this section of the Cloud Run docs](https://cloud.google.com/run/docs/troubleshooting) that says: > Note that the Cloud Run container instances run in an environment where the files written to the local filesystem count towards the available memory. This also includes any log files that are not written to /var/log/* or /dev/log. Does datasette write any large files when starting? Or does the [`COPY` command in the Dockerfile](https://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L349) count as writing to the local filesystem?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database, https://github.com/simonw/datasette/issues/1284#issuecomment-949604763,https://api.github.com/repos/simonw/datasette/issues/1284,949604763,IC_kwDOBm6k_c44mdGb,536941,fgregg,2021-10-22T12:54:34Z,2021-10-22T12:54:34Z,CONTRIBUTOR,i'm going to take a swing at this today. we'll see.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",845794436,Feature or Documentation Request: Individual table as home page template, https://github.com/simonw/datasette/issues/1496#issuecomment-949912718,https://api.github.com/repos/simonw/datasette/issues/1496,949912718,IC_kwDOBm6k_c44noSO,9599,simonw,2021-10-22T19:38:23Z,2021-10-22T19:38:23Z,OWNER,https://docs.datasette.io/en/latest/sql_queries.html#named-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033864602,Named parameters docs should include an example of a cast, https://github.com/simonw/datasette/issues/1401#issuecomment-950150483,https://api.github.com/repos/simonw/datasette/issues/1401,950150483,IC_kwDOBm6k_c44oiVT,418191,jaywgraves,2021-10-23T13:09:10Z,2021-10-23T13:09:10Z,CONTRIBUTOR,"I think it's because of this in `app.css` ``` ol, ul { list-style: none; } ``` https://github.com/simonw/datasette/blame/main/datasette/static/app.css#L35-L38 You could probably reinstate that by providing your own CSS. https://docs.datasette.io/en/0.24/custom_templates.html#custom-css-and-javascript","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",950664971,unordered list is not rendering bullet points in description_html on database page, https://github.com/simonw/datasette/issues/1482#issuecomment-950402273,https://api.github.com/repos/simonw/datasette/issues/1482,950402273,IC_kwDOBm6k_c44pfzh,9599,simonw,2021-10-24T22:00:29Z,2021-10-24T22:00:29Z,OWNER,Janus 0.6.2 is out now and should have the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1021550542,Support Python 3.10, https://github.com/simonw/datasette/pull/1495#issuecomment-950403521,https://api.github.com/repos/simonw/datasette/issues/1495,950403521,IC_kwDOBm6k_c44pgHB,9599,simonw,2021-10-24T22:09:18Z,2021-10-24T22:09:18Z,OWNER,"This is a great idea - I've wanted this myself before, but never spent any time thinking about how to achieve it. I think your design here is exactly right - an optional third item in the tuple consisting of a dictionary of options to pass to the view function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options, https://github.com/simonw/datasette/pull/1495#issuecomment-950403692,https://api.github.com/repos/simonw/datasette/issues/1495,950403692,IC_kwDOBm6k_c44pgJs,9599,simonw,2021-10-24T22:10:43Z,2021-10-24T22:10:43Z,OWNER,"To land this change we'll need a unit test that demonstrates the new capability - I suggest putting that next to this test: https://github.com/simonw/datasette/blob/15a9d4abfff0c45dee2a9f851326e1d61b1c678c/tests/test_plugins.py#L648-L659 It will also need documentation, which should be added here: https://github.com/simonw/datasette/blob/15a9d4abfff0c45dee2a9f851326e1d61b1c678c/docs/plugin_hooks.rst#register-routes-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options, https://github.com/simonw/datasette/issues/1497#issuecomment-950410554,https://api.github.com/repos/simonw/datasette/issues/1497,950410554,IC_kwDOBm6k_c44ph06,9599,simonw,2021-10-24T23:01:20Z,2021-10-24T23:01:28Z,OWNER,"I can replicate locally by running: ``` docker build -f Dockerfile \ -t datasetteproject/datasette:0.59.1 \ --build-arg VERSION=0.59.1 . ``` This gives me the same error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950410718,https://api.github.com/repos/simonw/datasette/issues/1497,950410718,IC_kwDOBm6k_c44ph3e,9599,simonw,2021-10-24T23:02:30Z,2021-10-24T23:02:30Z,OWNER,I got the same error publishing 0.59: https://github.com/simonw/datasette/actions/runs/1343251945,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950415129,https://api.github.com/repos/simonw/datasette/issues/1497,950415129,IC_kwDOBm6k_c44pi8Z,9599,simonw,2021-10-24T23:21:33Z,2021-10-24T23:21:33Z,OWNER,That fixed it! Resulting image is 249MB which is a very slight size reduction (I think previous was 259MB (uncompressed).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950411320,https://api.github.com/repos/simonw/datasette/issues/1497,950411320,IC_kwDOBm6k_c44piA4,9599,simonw,2021-10-24T23:06:05Z,2021-10-24T23:06:05Z,OWNER,"Right now the base image is: https://github.com/simonw/datasette/blob/e6e44372b34414eac2f36a4c1120af4f755aa423/Dockerfile#L1 I'm going to try `python:3.9.7-slim-buster` instead: https://hub.docker.com/layers/python/library/python/3.9.7-slim-buster/images/sha256-290b95e4b379762a9bd3d72644598e0972f4e2b5442bba60592c018fadcc744d?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950411417,https://api.github.com/repos/simonw/datasette/issues/1497,950411417,IC_kwDOBm6k_c44piCZ,9599,simonw,2021-10-24T23:06:45Z,2021-10-24T23:11:14Z,OWNER,"Same errors with `3.9.7`: ``` #5 41.46 /usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory #5 41.46 dpkg: error processing package libc6:amd64 (--configure): #5 41.46 installed libc6:amd64 package post-installation script subprocess returned error exit status 127 #5 41.47 Errors were encountered while processing: #5 41.47 libc6:amd64 #5 41.50 E: Sub-process /usr/bin/dpkg returned an error code (1) ``` I'm suspicious of this part of the `Dockerfile`: https://github.com/simonw/datasette/blob/e6e44372b34414eac2f36a4c1120af4f755aa423/Dockerfile#L1-L18 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950411808,https://api.github.com/repos/simonw/datasette/issues/1497,950411808,IC_kwDOBm6k_c44piIg,9599,simonw,2021-10-24T23:08:59Z,2021-10-24T23:08:59Z,OWNER,"Looks like it's this bug, reported on the Debian mailing list: https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1818037.html No obvious workaround there though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950411912,https://api.github.com/repos/simonw/datasette/issues/1497,950411912,IC_kwDOBm6k_c44piKI,9599,simonw,2021-10-24T23:09:41Z,2021-10-24T23:09:41Z,OWNER,Here that is in the Debian bug tracker: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993755,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950412628,https://api.github.com/repos/simonw/datasette/issues/1497,950412628,IC_kwDOBm6k_c44piVU,9599,simonw,2021-10-24T23:13:20Z,2021-10-24T23:13:27Z,OWNER,"I think the root cause here is that I'm using a Debian Buster base image and then installing SpatiaLite from Debian unstable (sid) - as described in this comment: https://github.com/simonw/datasette/issues/1249#issuecomment-804309510 That's has worked fine in the past, but Sid is unstable - and this seems to be one of those instabilities.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950413185,https://api.github.com/repos/simonw/datasette/issues/1497,950413185,IC_kwDOBm6k_c44pieB,9599,simonw,2021-10-24T23:16:25Z,2021-10-24T23:18:30Z,OWNER,"Debian stable these days is ""bullseye"" - https://www.debian.org/releases/ - which has the version of SpatiaLite that I was previously pulling in from Sid: https://packages.debian.org/bullseye/libsqlite3-mod-spatialite So upgrading to the 3.9.7-slim-bullseye base image may help. https://hub.docker.com/layers/python/library/python/3.9.7-slim-bullseye/images/sha256-67af5f544115124dc6d6da1d9d2815aa9825f6fd4aa6710adb0ec1725280fb89?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950415822,https://api.github.com/repos/simonw/datasette/issues/1497,950415822,IC_kwDOBm6k_c44pjHO,9599,simonw,2021-10-24T23:25:45Z,2021-10-24T23:25:45Z,OWNER,I'm going to attempt to publish `0.59` to Docker Hub using https://github.com/simonw/datasette/blob/2c31d1cd9cd3b63458ccbe391866499fa3f44978/.github/workflows/push_docker_tag.yml - if that works I'll push `0.59.1` as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416061,https://api.github.com/repos/simonw/datasette/issues/1497,950416061,IC_kwDOBm6k_c44pjK9,9599,simonw,2021-10-24T23:27:18Z,2021-10-24T23:27:18Z,OWNER,That worked: https://hub.docker.com/layers/datasetteproject/datasette/0.59/images/sha256-038decc28e0ea84b281ecc0058fe8eba7aa99596e5a2177ff714092ad03294ed?context=explore,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416460,https://api.github.com/repos/simonw/datasette/issues/1497,950416460,IC_kwDOBm6k_c44pjRM,9599,simonw,2021-10-24T23:30:10Z,2021-10-24T23:30:10Z,OWNER,"Testing that newly published image: ``` % docker run -p 8002:8001 -v `pwd`:/mnt \ datasetteproject/datasette:0.59 datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db Unable to find image 'datasetteproject/datasette:0.59' locally 0.59: Pulling from datasetteproject/datasette 7d63c13d9b9b: Already exists 6ad2a11ca37b: Already exists e9edbe81a001: Already exists 36629b83aba2: Already exists 7338abefe51c: Already exists 6d71b6b88b82: Pull complete 8c4da3c56bdc: Pull complete Digest: sha256:038decc28e0ea84b281ecc0058fe8eba7aa99596e5a2177ff714092ad03294ed Status: Downloaded newer image for datasetteproject/datasette:0.59 INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ``` and `http://localhost:8002/versions.json` returns: ```json { ""python"": { ""version"": ""3.9.7"", ""full"": ""3.9.7 (default, Oct 12 2021, 02:43:43) \n[GCC 10.2.1 20210110]"" }, ""datasette"": { ""version"": ""0.59"" }, ""asgi"": ""3.0"", ""uvicorn"": ""0.15.0"", ""sqlite"": { ""version"": ""3.34.1"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416659,https://api.github.com/repos/simonw/datasette/issues/1497,950416659,IC_kwDOBm6k_c44pjUT,9599,simonw,2021-10-24T23:31:41Z,2021-10-24T23:31:41Z,OWNER,"Published `0.59.1` as well: https://github.com/simonw/datasette/runs/3991214225?check_suite_focus=true Result: https://hub.docker.com/layers/datasetteproject/datasette/0.59.1/images/sha256-dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416682,https://api.github.com/repos/simonw/datasette/issues/1497,950416682,IC_kwDOBm6k_c44pjUq,9599,simonw,2021-10-24T23:31:51Z,2021-10-24T23:31:51Z,OWNER,One catch: the `latest` tag on Docker Hub is still three months old.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950416802,https://api.github.com/repos/simonw/datasette/issues/1497,950416802,IC_kwDOBm6k_c44pjWi,9599,simonw,2021-10-24T23:32:39Z,2021-10-24T23:32:39Z,OWNER,"That's because the `publish.yml` workflow ends with this, which isn't in the `push_docker_tag.yml` workflow: https://github.com/simonw/datasette/blob/2c31d1cd9cd3b63458ccbe391866499fa3f44978/.github/workflows/publish.yml#L117-L119","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/issues/1497#issuecomment-950417375,https://api.github.com/repos/simonw/datasette/issues/1497,950417375,IC_kwDOBm6k_c44pjff,9599,simonw,2021-10-24T23:36:54Z,2021-10-24T23:36:54Z,OWNER,"Tried fixing this by pushing a new `latest` tag from my laptop: ``` (datasette) datasette % docker pull datasetteproject/datasette:0.59.1 0.59.1: Pulling from datasetteproject/datasette 7d63c13d9b9b: Already exists 6ad2a11ca37b: Already exists e9edbe81a001: Already exists 36629b83aba2: Already exists 7338abefe51c: Already exists 6b825daddc6c: Pull complete d7508b065a21: Pull complete Digest: sha256:dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c Status: Downloaded newer image for datasetteproject/datasette:0.59.1 docker.io/datasetteproject/datasette:0.59.1 (datasette) datasette % docker tag datasetteproject/datasette:0.59.1 datasetteproject/datasette:latest (datasette) datasette % docker push datasetteproject/datasette:latest The push refers to repository [docker.io/datasetteproject/datasette] d668c99b6ff1: Layer already exists aa20c9013575: Layer already exists c97eebf2b227: Layer already exists 284a6c64b82c: Layer already exists 388eedeb736e: Layer already exists 2feece0964b8: Layer already exists e8b689711f21: Layer already exists errors: denied: requested access to the resource is denied unauthorized: authentication required ``` So I logged in with `docker login`: ``` (datasette) datasette % docker login Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username: datasetteproject Password: ``` And ran the push again and it worked: ``` (datasette) datasette % docker push datasetteproject/datasette:latest The push refers to repository [docker.io/datasetteproject/datasette] d668c99b6ff1: Layer already exists aa20c9013575: Layer already exists c97eebf2b227: Layer already exists 284a6c64b82c: Layer already exists 388eedeb736e: Layer already exists 2feece0964b8: Layer already exists e8b689711f21: Layer already exists latest: digest: sha256:dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c size: 1793 ``` https://hub.docker.com/layers/datasetteproject/datasette/latest/images/sha256-dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""", https://github.com/simonw/datasette/pull/1204#issuecomment-951731255,https://api.github.com/repos/simonw/datasette/issues/1204,951731255,IC_kwDOBm6k_c44ukQ3,30934,20after4,2021-10-26T09:01:28Z,2021-10-26T09:01:28Z,NONE,"> Writing the tests will be a bit tricky since we need to confirm that the `include_table_top(datasette, database, actor, table)` arguments were all passed correctly but the only thing we get back from the plugin is a list of templates. Maybe encode those values into the template names somehow? Why not return a data structure instead of just a template name? I've already done some custom hacking to modify datasette but the plugin mechanism you are building here would be much cleaner than what I've built. I'd be happy to help with testing this PR and fleshing it out further if you are still considering merging this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793002853,WIP: Plugin includes, https://github.com/simonw/datasette/issues/878#issuecomment-951740637,https://api.github.com/repos/simonw/datasette/issues/878,951740637,IC_kwDOBm6k_c44umjd,30934,20after4,2021-10-26T09:12:15Z,2021-10-26T09:12:15Z,NONE,"This sounds really ambitious but also really awesome. I like the idea that basically any piece of a page could be selectively replaced. It sort of sounds like a python asyncio version of https://github.com/observablehq/runtime","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1505#issuecomment-970188065,https://api.github.com/repos/simonw/datasette/issues/1505,970188065,IC_kwDOBm6k_c450-Uh,7094907,Segerberg,2021-11-16T11:40:52Z,2021-11-16T11:40:52Z,NONE,A suggestion is to have the option to choose an arbitrary delimiter (and quoting characters ),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052247023,Datasette should have an option to output CSV with semicolons, https://github.com/simonw/datasette/issues/1012#issuecomment-970266123,https://api.github.com/repos/simonw/datasette/issues/1012,970266123,IC_kwDOBm6k_c451RYL,45380,bollwyvl,2021-11-16T13:18:36Z,2021-11-16T13:18:36Z,CONTRIBUTOR,"Congratulations, looks like it went through! There was a bit of a hold-up on the JupyterLab ones, but it's semi automated: a dependabot pr to warehouse and a CI deploy, with a click in between. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py, https://github.com/simonw/datasette/issues/878#issuecomment-970712713,https://api.github.com/repos/simonw/datasette/issues/878,970712713,IC_kwDOBm6k_c452-aJ,9599,simonw,2021-11-16T21:54:33Z,2021-11-16T21:54:33Z,OWNER,I'm going to continue working on this in a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970673085,https://api.github.com/repos/simonw/datasette/issues/878,970673085,IC_kwDOBm6k_c4520u9,9599,simonw,2021-11-16T20:58:24Z,2021-11-16T20:58:24Z,OWNER,"New test: ```python class Complex(AsyncBase): def __init__(self): self.log = [] async def d(self): await asyncio.sleep(random() * 0.1) print(""LOG: d"") self.log.append(""d"") async def c(self): await asyncio.sleep(random() * 0.1) print(""LOG: c"") self.log.append(""c"") async def b(self, c, d): print(""LOG: b"") self.log.append(""b"") async def a(self, b, c): print(""LOG: a"") self.log.append(""a"") async def go(self, a): print(""LOG: go"") self.log.append(""go"") return self.log @pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once assert tuple(result) in ( # c and d could happen in either order (""c"", ""d"", ""b"", ""a"", ""go""), (""d"", ""c"", ""b"", ""a"", ""go""), ) ``` And this code passes it: ```python import asyncio from functools import wraps import inspect try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def __new__(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {""_registry"": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == ""resolve"": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != ""self"" and not p.startswith(""_"") } for key, method in _registry.items() } new_attrs[""_graph""] = graph return super().__new__(cls, name, bases, new_attrs) def make_method(method): parameters = inspect.signature(method).parameters.keys() @wraps(method) async def inner(self, _results=None, **kwargs): print(""\n{}.{}({}) _results={}"".format(self, method.__name__, kwargs, _results)) # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != ""self""] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), ""The following DI parameters could not be found in the registry: {}"".format( missing ) results = {} results.update(kwargs) if to_resolve: resolved_parameters = await self.resolve(to_resolve, _results) results.update(resolved_parameters) return_value = await method(self, **results) if _results is not None: _results[method.__name__] = return_value return return_value return inner class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names, results=None): print(""\n resolve: "", names) if results is None: results = {} # Come up with an execution plan, just for these nodes ts = graphlib.TopologicalSorter() to_do = set(names) done = set() while to_do: item = to_do.pop() dependencies = self._graph[item] ts.add(item, *dependencies) done.add(item) # Add any not-done dependencies to the queue to_do.update({k for k in dependencies if k not in done}) ts.prepare() plan = [] while ts.is_active(): node_group = ts.get_ready() plan.append(node_group) ts.done(*node_group) print(""plan:"", plan) results = {} for node_group in plan: awaitables = [ self._registry[name]( self, _results=results, **{k: v for k, v in results.items() if k in self._graph[name]}, ) for name in node_group ] print("" results = "", results) print("" awaitables: "", awaitables) awaitable_results = await asyncio.gather(*awaitables) results.update( {p[0].__name__: p[1] for p in zip(awaitables, awaitable_results)} ) print("" End of resolve(), returning"", results) return {key: value for key, value in results.items() if key in names} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970705738,https://api.github.com/repos/simonw/datasette/issues/878,970705738,IC_kwDOBm6k_c4528tK,9599,simonw,2021-11-16T21:44:31Z,2021-11-16T21:44:31Z,OWNER,Wrote a TIL about what I learned using `TopologicalSorter`: https://til.simonwillison.net/python/graphlib-topologicalsorter,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1509#issuecomment-970544733,https://api.github.com/repos/simonw/datasette/issues/1509,970544733,IC_kwDOBm6k_c452VZd,9599,simonw,2021-11-16T18:22:32Z,2021-11-16T18:22:32Z,OWNER,"This is mainly happening here: - https://github.com/simonw/datasette/issues/782","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054243511,Datasette 1.0 JSON API (and documentation), https://github.com/simonw/datasette/issues/782#issuecomment-970554697,https://api.github.com/repos/simonw/datasette/issues/782,970554697,IC_kwDOBm6k_c452X1J,9599,simonw,2021-11-16T18:32:03Z,2021-11-16T18:32:03Z,OWNER,"I'm going to take another look at this: - https://github.com/simonw/datasette/issues/878","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/782#issuecomment-970553780,https://api.github.com/repos/simonw/datasette/issues/782,970553780,IC_kwDOBm6k_c452Xm0,9599,simonw,2021-11-16T18:30:51Z,2021-11-16T18:30:58Z,OWNER,"OK, I'm ready to start working on this today. I'm going to go with a default representation that looks like this: ```json { ""rows"": [ {""id"": 1, ""name"": ""One""}, {""id"": 2, ""name"": ""Two""} ], ""next_url"": null } ``` Note that there's no `count` - all it provides is the current selection of results and an indication as to how the next can be retrieved (`null` if there are no more results). I'll implement `?_extra=` to provide everything else.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/pull/1512#issuecomment-970718652,https://api.github.com/repos/simonw/datasette/issues/1512,970718652,IC_kwDOBm6k_c452_28,22429695,codecov[bot],2021-11-16T22:02:59Z,2021-11-16T23:51:48Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1512](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8f757da) into [main](https://codecov.io/gh/simonw/datasette/commit/0156c6b5e52d541e93f0d68e9245f20ae83bc933?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0156c6b) will **decrease** coverage by `2.10%`. > The diff coverage is `36.20%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1512/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1512 +/- ## ========================================== - Coverage 91.82% 89.72% -2.11% ========================================== Files 34 36 +2 Lines 4430 4604 +174 ========================================== + Hits 4068 4131 +63 - Misses 362 473 +111 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/utils/vendored\_graphlib.py](https://codecov.io/gh/simonw/datasette/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3ZlbmRvcmVkX2dyYXBobGliLnB5) | `0.00% <0.00%> (ø)` | | | [datasette/utils/asyncdi.py](https://codecov.io/gh/simonw/datasette/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzeW5jZGkucHk=) | `96.92% <96.92%> (ø)` | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0156c6b...8f757da](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/pull/1512#issuecomment-970718337,https://api.github.com/repos/simonw/datasette/issues/1512,970718337,IC_kwDOBm6k_c452_yB,9599,simonw,2021-11-16T22:02:30Z,2021-11-16T22:02:30Z,OWNER,"I've decided to make the clever `asyncio` dependency injection opt-in - so you can either decorate with `@inject` or you can set `inject_all = True` on the class - for example: ```python import asyncio from datasette.utils.asyncdi import AsyncBase, inject class Simple(AsyncBase): def __init__(self): self.log = [] @inject async def two(self): self.log.append(""two"") @inject async def one(self, two): self.log.append(""one"") return self.log async def not_inject(self, one, two): return one + two class Complex(AsyncBase): inject_all = True def __init__(self): self.log = [] async def b(self): self.log.append(""b"") async def a(self, b): self.log.append(""a"") async def go(self, a): self.log.append(""go"") return self.log ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/issues/878#issuecomment-970624197,https://api.github.com/repos/simonw/datasette/issues/878,970624197,IC_kwDOBm6k_c452ozF,9599,simonw,2021-11-16T19:49:05Z,2021-11-16T19:49:05Z,OWNER,"Here's the latest version of my weird dependency injection async class: ```python import inspect class AsyncMeta(type): def __new__(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {""_registry"": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == ""resolve"": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Topological sort of _registry by parameter dependencies graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != ""self"" and not p.startswith(""_"") } for key, method in _registry.items() } new_attrs[""_graph""] = graph return super().__new__(cls, name, bases, new_attrs) def make_method(method): @wraps(method) async def inner(self, **kwargs): parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != ""self""] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), ""The following DI parameters could not be found in the registry: {}"".format( missing ) results = {} results.update(kwargs) results.update(await self.resolve(to_resolve)) return await method(self, **results) return inner bad = [0] class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names): print("" resolve({})"".format(names)) results = {} # Resolve them in the correct order ts = TopologicalSorter() ts2 = TopologicalSorter() print("" names = "", names) print("" self._graph = "", self._graph) for name in names: if self._graph[name]: ts.add(name, *self._graph[name]) ts2.add(name, *self._graph[name]) print("" static_order ="", tuple(ts2.static_order())) ts.prepare() while ts.is_active(): print("" is_active, i = "", bad[0]) bad[0] += 1 if bad[0] > 20: print("" Infinite loop?"") break nodes = ts.get_ready() print("" Do nodes:"", nodes) awaitables = [self._registry[name](self, **{ k: v for k, v in results.items() if k in self._graph[name] }) for name in nodes] print("" awaitables: "", awaitables) awaitable_results = await asyncio.gather(*awaitables) results.update({ p[0].__name__: p[1] for p in zip(awaitables, awaitable_results) }) print(results) for node in nodes: ts.done(node) return results ``` Example usage: ```python class Foo(AsyncBase): async def graa(self, boff): print(""graa"") return 5 async def boff(self): print(""boff"") return 8 async def other(self, boff, graa): print(""other"") return 5 + boff + graa foo = Foo() await foo.other() ``` Output: ``` resolve(['boff', 'graa']) names = ['boff', 'graa'] self._graph = {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}} static_order = ('boff', 'graa') is_active, i = 0 Do nodes: ('boff',) awaitables: [] resolve([]) names = [] self._graph = {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}} static_order = () boff {'boff': 8} is_active, i = 1 Do nodes: ('graa',) awaitables: [] resolve([]) names = [] self._graph = {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}} static_order = () graa {'boff': 8, 'graa': 5} other 18 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970655304,https://api.github.com/repos/simonw/datasette/issues/878,970655304,IC_kwDOBm6k_c452wZI,9599,simonw,2021-11-16T20:32:16Z,2021-11-16T20:32:16Z,OWNER,"This code is really fiddly. I just got to this version: ```python import asyncio from functools import wraps import inspect try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def __new__(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {""_registry"": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == ""resolve"": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != ""self"" and not p.startswith(""_"") } for key, method in _registry.items() } new_attrs[""_graph""] = graph return super().__new__(cls, name, bases, new_attrs) def make_method(method): @wraps(method) async def inner(self, _results=None, **kwargs): print(""inner - _results="", _results) parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != ""self""] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), ""The following DI parameters could not be found in the registry: {}"".format( missing ) results = {} results.update(kwargs) if to_resolve: resolved_parameters = await self.resolve(to_resolve, _results) results.update(resolved_parameters) return_value = await method(self, **results) if _results is not None: _results[method.__name__] = return_value return return_value return inner class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names, results=None): print(""\n resolve: "", names) if results is None: results = {} # Resolve them in the correct order ts = graphlib.TopologicalSorter() for name in names: ts.add(name, *self._graph[name]) ts.prepare() async def resolve_nodes(nodes): print("" resolve_nodes"", nodes) print("" (current results = {})"".format(repr(results))) awaitables = [ self._registry[name]( self, _results=results, **{k: v for k, v in results.items() if k in self._graph[name]}, ) for name in nodes if name not in results ] print("" awaitables: "", awaitables) awaitable_results = await asyncio.gather(*awaitables) results.update( {p[0].__name__: p[1] for p in zip(awaitables, awaitable_results)} ) if not ts.is_active(): # Nothing has dependencies - just resolve directly print("" no dependencies, resolve directly"") await resolve_nodes(names) else: # Resolve in topological order while ts.is_active(): nodes = ts.get_ready() print("" ts.get_ready() returned nodes:"", nodes) await resolve_nodes(nodes) for node in nodes: ts.done(node) print("" End of resolve(), returning"", results) return {key: value for key, value in results.items() if key in names} ``` With this test: ```python class Complex(AsyncBase): def __init__(self): self.log = [] async def c(self): print(""LOG: c"") self.log.append(""c"") async def b(self, c): print(""LOG: b"") self.log.append(""b"") async def a(self, b, c): print(""LOG: a"") self.log.append(""a"") async def go(self, a): print(""LOG: go"") self.log.append(""go"") return self.log @pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once assert result == [""c"", ""b"", ""a"", ""go""] ``` This test sometimes passes, and sometimes fails! Output for a pass: ``` tests/test_asyncdi.py inner - _results= None resolve: ['a'] ts.get_ready() returned nodes: ('c', 'b') resolve_nodes ('c', 'b') (current results = {}) awaitables: [, ] inner - _results= {} LOG: c inner - _results= {'c': None} resolve: ['c'] ts.get_ready() returned nodes: ('c',) resolve_nodes ('c',) (current results = {'c': None}) awaitables: [] End of resolve(), returning {'c': None} LOG: b ts.get_ready() returned nodes: ('a',) resolve_nodes ('a',) (current results = {'c': None, 'b': None}) awaitables: [] inner - _results= {'c': None, 'b': None} LOG: a End of resolve(), returning {'c': None, 'b': None, 'a': None} LOG: go ``` Output for a fail: ``` tests/test_asyncdi.py inner - _results= None resolve: ['a'] ts.get_ready() returned nodes: ('b', 'c') resolve_nodes ('b', 'c') (current results = {}) awaitables: [, ] inner - _results= {} resolve: ['c'] ts.get_ready() returned nodes: ('c',) resolve_nodes ('c',) (current results = {}) awaitables: [] inner - _results= {} LOG: c inner - _results= {'c': None} LOG: c End of resolve(), returning {'c': None} LOG: b ts.get_ready() returned nodes: ('a',) resolve_nodes ('a',) (current results = {'c': None, 'b': None}) awaitables: [] inner - _results= {'c': None, 'b': None} LOG: a End of resolve(), returning {'c': None, 'b': None, 'a': None} LOG: go F =================================================================================================== FAILURES =================================================================================================== _________________________________________________________________________________________________ test_complex _________________________________________________________________________________________________ @pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once > assert result == [""c"", ""b"", ""a"", ""go""] E AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go'] E At index 1 diff: 'c' != 'b' E Left contains one more item: 'go' E Use -v to get the full diff tests/test_asyncdi.py:48: AssertionError ================== short test summary info ================================ FAILED tests/test_asyncdi.py::test_complex - AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go'] ``` I figured out why this is happening. `a` requires `b` and `c` `b` also requires `c` The code decides to run `b` and `c` in parallel. If `c` completes first, then when `b` runs it gets to use the already-calculated result for `c` - so it doesn't need to call `c` again. If `b` gets to that point before `c` does it also needs to call `c`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970655927,https://api.github.com/repos/simonw/datasette/issues/878,970655927,IC_kwDOBm6k_c452wi3,9599,simonw,2021-11-16T20:33:11Z,2021-11-16T20:33:11Z,OWNER,"What should be happening here instead is it should resolve the full graph and notice that `c` is depended on by both `b` and `a` - so it should run `c` first, then run the next ones in parallel. So maybe the algorithm I'm inheriting from https://docs.python.org/3/library/graphlib.html isn't the correct algorithm?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970657874,https://api.github.com/repos/simonw/datasette/issues/878,970657874,IC_kwDOBm6k_c452xBS,9599,simonw,2021-11-16T20:36:01Z,2021-11-16T20:36:01Z,OWNER,"My goal here is to calculate the most efficient way to resolve the different nodes, running them in parallel where possible. So for this class: ```python class Complex(AsyncBase): async def d(self): pass async def c(self): pass async def b(self, c, d): pass async def a(self, b, c): pass async def go(self, a): pass ``` A call to `go()` should do this: - `c` and `d` in parallel - `b` - `a` - `go`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-970660299,https://api.github.com/repos/simonw/datasette/issues/878,970660299,IC_kwDOBm6k_c452xnL,9599,simonw,2021-11-16T20:39:43Z,2021-11-16T20:42:27Z,OWNER,"But that does seem to be the plan that `TopographicalSorter` provides: ```python graph = {""go"": {""a""}, ""a"": {""b"", ""c""}, ""b"": {""c"", ""d""}} ts = TopologicalSorter(graph) ts.prepare() while ts.is_active(): nodes = ts.get_ready() print(nodes) ts.done(*nodes) ``` Outputs: ``` ('c', 'd') ('b',) ('a',) ('go',) ``` Also: ```python graph = {""go"": {""d"", ""e"", ""f""}, ""d"": {""b"", ""c""}, ""b"": {""c""}} ts = TopologicalSorter(graph) ts.prepare() while ts.is_active(): nodes = ts.get_ready() print(nodes) ts.done(*nodes) ``` Gives: ``` ('e', 'f', 'c') ('b',) ('d',) ('go',) ``` I'm confident that `TopologicalSorter` is the way to do this. I think I need to rewrite my code to call it once to get that plan, then `await asyncio.gather(*nodes)` in turn to execute it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1513#issuecomment-970738130,https://api.github.com/repos/simonw/datasette/issues/1513,970738130,IC_kwDOBm6k_c453EnS,9599,simonw,2021-11-16T22:32:19Z,2021-11-16T22:32:19Z,OWNER,"I came up with the following query which seems to work! ```sql with cte as ( select rowid, country, country_long, name, owner, primary_fuel from [global-power-plants] ), truncated as ( select null as _facet, null as facet_name, null as facet_count, rowid, country, country_long, name, owner, primary_fuel from cte order by rowid limit 4 ), country_long_facet as ( select 'country_long' as _facet, country_long as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), owner_facet as ( select 'owner' as _facet, owner as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), primary_fuel_facet as ( select 'primary_fuel' as _facet, primary_fuel as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ) select * from truncated union all select * from country_long_facet union all select * from owner_facet union all select * from primary_fuel_facet ``` (Limits should be 101, 31, 31, 31 but I reduced size to get a shorter example table). Results [look like this](https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+country_long_facet%0D%0Aunion+all+select+*+from+owner_facet%0D%0Aunion+all+select+*+from+primary_fuel_facet): _facet | facet_name | facet_count | rowid | country | country_long | name | owner | primary_fuel -- | -- | -- | -- | -- | -- | -- | -- | --   |   |   | 1 | AFG | Afghanistan | Kajaki Hydroelectric Power Plant Afghanistan |   | Hydro   |   |   | 2 | AFG | Afghanistan | Kandahar DOG |   | Solar   |   |   | 3 | AFG | Afghanistan | Kandahar JOL |   | Solar   |   |   | 4 | AFG | Afghanistan | Mahipar Hydroelectric Power Plant Afghanistan |   | Hydro country_long | United States of America | 8688 |   |   |   |   |   |   country_long | China | 4235 |   |   |   |   |   |   country_long | United Kingdom | 2603 |   |   |   |   |   |   owner |   | 14112 |   |   |   |   |   |   owner | Lightsource Renewable Energy | 120 |   |   |   |   |   |   owner | Cypress Creek Renewables | 109 |   |   |   |   |   |   primary_fuel | Solar | 9662 |   |   |   |   |   |   primary_fuel | Hydro | 7155 |   |   |   |   |   |   primary_fuel | Wind | 5188 |   |   |   |   |   |   This is a neat proof of concept. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970742415,https://api.github.com/repos/simonw/datasette/issues/1513,970742415,IC_kwDOBm6k_c453FqP,9599,simonw,2021-11-16T22:37:14Z,2021-11-16T22:37:14Z,OWNER,"The query takes 42.794ms to run. Here's the equivalent page using separate queries: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1 Annoyingly I can't disable facet suggestions but keep facets. I'm going to turn on tracing so I can see how long the separate queries took.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970758179,https://api.github.com/repos/simonw/datasette/issues/1513,970758179,IC_kwDOBm6k_c453Jgj,9599,simonw,2021-11-16T22:47:38Z,2021-11-16T22:47:38Z,OWNER,"Trace now enabled: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1&_trace=1 Here are the relevant traces: ```json [ { ""type"": ""sql"", ""start"": 31.214430154, ""end"": 31.214817089, ""duration_ms"": 0.3869350000016425, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/base.py\"", line 262, in get\n return await self.view_get(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/base.py\"", line 477, in view_get\n response_or_template_contexts = await self.data(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 705, in data\n results = await db.execute(sql, params, truncate=True, **extra_args)\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] order by rowid limit 3"", ""params"": {} }, { ""type"": ""sql"", ""start"": 31.215234586, ""end"": 31.220110342, ""duration_ms"": 4.875756000000564, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 760, in data\n ) = await facet.facet_results()\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 212, in facet_results\n facet_rows_results = await self.ds.execute(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select country_long as value, count(*) as count from (\n select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n )\n where country_long is not null\n group by country_long order by count desc, value limit 4"", ""params"": [] }, { ""type"": ""sql"", ""start"": 31.221062485, ""end"": 31.228968364, ""duration_ms"": 7.905878999999061, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 760, in data\n ) = await facet.facet_results()\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 212, in facet_results\n facet_rows_results = await self.ds.execute(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select owner as value, count(*) as count from (\n select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n )\n where owner is not null\n group by owner order by count desc, value limit 4"", ""params"": [] }, { ""type"": ""sql"", ""start"": 31.229809757, ""end"": 31.253902162, ""duration_ms"": 24.09240499999754, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 760, in data\n ) = await facet.facet_results()\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 212, in facet_results\n facet_rows_results = await self.ds.execute(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select primary_fuel as value, count(*) as count from (\n select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n )\n where primary_fuel is not null\n group by primary_fuel order by count desc, value limit 4"", ""params"": [] }, { ""type"": ""sql"", ""start"": 31.255699745, ""end"": 31.256243889, ""duration_ms"": 0.544143999999136, ""traceback"": [ "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 145, in suggest\n row_count = await self.get_row_count()\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 132, in get_row_count\n await self.ds.execute(\n"", "" File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""global-power-plants"", ""sql"": ""select count(*) from (select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] )"", ""params"": [] } ] ``` ``` fetch rows: 0.3869350000016425 ms facet country_long: 4.875756000000564 ms facet owner: 7.905878999999061 ms facet primary_fuel: 24.09240499999754 ms count: 0.544143999999136 ms ``` Total = 37.8ms I modified the query to include the total count as well: https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+country_long_facet%0D%0Aunion+all+select+*+from+owner_facet%0D%0Aunion+all+select+*+from+primary_fuel_facet%0D%0Aunion+all+select+*+from+total_count&_trace=1 ```sql with cte as ( select rowid, country, country_long, name, owner, primary_fuel from [global-power-plants] ), truncated as ( select null as _facet, null as facet_name, null as facet_count, rowid, country, country_long, name, owner, primary_fuel from cte order by rowid limit 4 ), country_long_facet as ( select 'country_long' as _facet, country_long as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), owner_facet as ( select 'owner' as _facet, owner as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), primary_fuel_facet as ( select 'primary_fuel' as _facet, primary_fuel as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), total_count as ( select 'COUNT' as _facet, '' as facet_name, count(*) as facet_count, null, null, null, null, null, null from cte ) select * from truncated union all select * from country_long_facet union all select * from owner_facet union all select * from primary_fuel_facet union all select * from total_count ``` The trace says that query took 34.801436999998714 ms. To my huge surprise, this convoluted optimization only shaves the sum query time down from 37.8ms to 34.8ms! That entire database file is just 11.1 MB though. Maybe it would make a meaningful difference on something larger?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970767952,https://api.github.com/repos/simonw/datasette/issues/1513,970767952,IC_kwDOBm6k_c453L5Q,9599,simonw,2021-11-16T22:53:52Z,2021-11-16T22:53:52Z,OWNER,It's going to take another 15 minutes for the build to finish and deploy the version with `_trace=1`: https://github.com/simonw/covid-19-datasette/actions/runs/1469150112,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970766486,https://api.github.com/repos/simonw/datasette/issues/1513,970766486,IC_kwDOBm6k_c453LiW,9599,simonw,2021-11-16T22:52:56Z,2021-11-16T22:56:07Z,OWNER,"https://covid-19.datasettes.com/covid is 805.2MB https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2 Equivalent SQL: https://covid-19.datasettes.com/covid?sql=with+cte+as+%28%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A%29%2C%0D%0Astate_facet+as+%28%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Afips_facet+as+%28%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Acounty_facet+as+%28%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet%0D%0Aunion+all+select+*+from+total_count ```sql with cte as ( select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties ), truncated as ( select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths from cte order by date desc limit 4 ), state_facet as ( select 'state' as _facet, state as facet_name, count(*) as facet_count, null, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), fips_facet as ( select 'fips' as _facet, fips as facet_name, count(*) as facet_count, null, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), county_facet as ( select 'county' as _facet, county as facet_name, count(*) as facet_count, null, null, null, null, null, null, null from cte group by facet_name order by facet_count desc limit 3 ), total_count as ( select 'COUNT' as _facet, '' as facet_name, count(*) as facet_count, null, null, null, null, null, null, null from cte ) select * from truncated union all select * from state_facet union all select * from fips_facet union all select * from county_facet union all select * from total_count ``` _facet | facet_name | facet_count | rowid | date | county | state | fips | cases | deaths -- | -- | -- | -- | -- | -- | -- | -- | -- | --   |   |   | 1917344 | 2021-11-15 | Autauga | Alabama | 1001 | 10407 | 154   |   |   | 1917345 | 2021-11-15 | Baldwin | Alabama | 1003 | 37875 | 581   |   |   | 1917346 | 2021-11-15 | Barbour | Alabama | 1005 | 3648 | 79   |   |   | 1917347 | 2021-11-15 | Bibb | Alabama | 1007 | 4317 | 92 state | Texas | 148028 |   |   |   |   |   |   |   state | Georgia | 96249 |   |   |   |   |   |   |   state | Virginia | 79315 |   |   |   |   |   |   |   fips |   | 17580 |   |   |   |   |   |   |   fips | 53061 | 665 |   |   |   |   |   |   |   fips | 17031 | 662 |   |   |   |   |   |   |   county | Washington | 18666 |   |   |   |   |   |   |   county | Unknown | 15840 |   |   |   |   |   |   |   county | Jefferson | 15637 |   |   |   |   |   |   |   COUNT |   | 1920593 |   |   |   |   |   |   |  ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970770304,https://api.github.com/repos/simonw/datasette/issues/1513,970770304,IC_kwDOBm6k_c453MeA,9599,simonw,2021-11-16T22:55:19Z,2021-11-16T22:55:19Z,OWNER,(One thing I really like about this pattern is that it should work exactly the same when used to facet the results of arbitrary SQL queries as it does when faceting results from the table page.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970780866,https://api.github.com/repos/simonw/datasette/issues/1513,970780866,IC_kwDOBm6k_c453PDC,9599,simonw,2021-11-16T23:01:57Z,2021-11-16T23:01:57Z,OWNER,"One disadvantage to this approach: if you have a SQL time limit of 1s and it takes 0.9s to return the rows but then 0.5s to calculate each of the requested facets the entire query will exceed the time limit. Could work around this by catching that error and then re-running the query just for the rows, but that would result in the user having to wait longer for the results. Could try to remember if that has happened using an in-memory Python data structure and skip the faceting optimization if it's caused problems in the past? That seems a bit gross. Maybe this becomes an opt-in optimization you can request in your `metadata.json` setting for that table, which massively increases the time limit? That's a bit weird too - now there are two separate implementations of the faceting logic, which had better have a REALLY big pay-off to be worth maintaining. What if we kept the query that returns the rows to be displayed on the page separate from the facets, but then executed all of the facets together using this method such that the `cte` only (presumably) has to be calculated once? That would still lead to multiple facets potentially exceeding the SQL time limit when single facets would not have. Maybe a better optimization would be to move facets to happening via `fetch()` calls from the client, so the user gets to see their rows instantly and the facets then appear as and when they are available (though it would cause page jank). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970827674,https://api.github.com/repos/simonw/datasette/issues/1513,970827674,IC_kwDOBm6k_c453aea,9599,simonw,2021-11-16T23:26:58Z,2021-11-16T23:26:58Z,OWNER,"With trace. https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&_trace=1 shows the following: ``` fetch rows: 0.41762600005768036 ms facet state: 284.30423800000426 ms facet county: 273.2565999999679 ms facet fips: 197.80996999998024 ms ``` = 755.78843400001ms total It didn't run a count because that's the homepage and the count is cached. So I dropped the count from the query and ran it: https://covid-19.datasettes.com/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1 Shows 649.4359889999259 ms for the query - compared to 755.78843400001ms for the separate. So it saved about 100ms. Still not a huge difference though! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970828568,https://api.github.com/repos/simonw/datasette/issues/1513,970828568,IC_kwDOBm6k_c453asY,9599,simonw,2021-11-16T23:27:11Z,2021-11-16T23:27:11Z,OWNER,One last experiment: I'm going to try running an expensive query in the CTE portion.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970845844,https://api.github.com/repos/simonw/datasette/issues/1513,970845844,IC_kwDOBm6k_c453e6U,9599,simonw,2021-11-16T23:35:38Z,2021-11-16T23:35:38Z,OWNER,"I tried adding `cases > 10000` but the SQL query now takes too long - so moving this to my laptop. ``` cd /tmp wget https://covid-19.datasettes.com/covid.db datasette covid.db \ --setting facet_time_limit_ms 10000 \ --setting sql_time_limit_ms 10000 \ --setting trace_debug 1 ``` `http://127.0.0.1:8006/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&cases__gt=10000` shows in the traces: ```json [ { ""type"": ""sql"", ""start"": 12.693033525, ""end"": 12.694056904, ""duration_ms"": 1.0233789999993803, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 262, in get\n return await self.view_get(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 477, in view_get\n response_or_template_contexts = await self.data(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 705, in data\n results = await db.execute(sql, params, truncate=True, **extra_args)\n"" ], ""database"": ""covid"", ""sql"": ""select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 order by rowid limit 3"", ""params"": { ""p0"": 10000 } }, { ""type"": ""sql"", ""start"": 12.694285093, ""end"": 12.814936275, ""duration_ms"": 120.65118200000136, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 262, in get\n return await self.view_get(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 477, in view_get\n response_or_template_contexts = await self.data(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 723, in data\n count_rows = list(await db.execute(count_sql, from_sql_params))\n"" ], ""database"": ""covid"", ""sql"": ""select count(*) from ny_times_us_counties where \""cases\"" > :p0"", ""params"": { ""p0"": 10000 } }, { ""type"": ""sql"", ""start"": 12.818812089, ""end"": 12.851172544, ""duration_ms"": 32.360455000000954, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 856, in data\n suggested_facets.extend(await facet.suggest())\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\"", line 164, in suggest\n distinct_values = await self.ds.execute(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""covid"", ""sql"": ""select county, count(*) as n from (\n select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 \n ) where county is not null\n group by county\n limit 4"", ""params"": { ""p0"": 10000 } }, { ""type"": ""sql"", ""start"": 12.851418868, ""end"": 12.871268359, ""duration_ms"": 19.84949100000044, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 856, in data\n suggested_facets.extend(await facet.suggest())\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\"", line 164, in suggest\n distinct_values = await self.ds.execute(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""covid"", ""sql"": ""select state, count(*) as n from (\n select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 \n ) where state is not null\n group by state\n limit 4"", ""params"": { ""p0"": 10000 } }, { ""type"": ""sql"", ""start"": 12.871497655, ""end"": 12.897715027, ""duration_ms"": 26.217371999999628, ""traceback"": [ "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 856, in data\n suggested_facets.extend(await facet.suggest())\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\"", line 164, in suggest\n distinct_values = await self.ds.execute(\n"", "" File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\"", line 634, in execute\n return await self.databases[db_name].execute(\n"" ], ""database"": ""covid"", ""sql"": ""select fips, count(*) as n from (\n select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 \n ) where fips is not null\n group by fips\n limit 4"", ""params"": { ""p0"": 10000 } } ] ``` So that's: ``` fetch rows: 1.0233789999993803 ms count: 120.65118200000136 ms facet county: 32.360455000000954 ms facet state: 19.84949100000044 ms facet fips: 26.217371999999628 ms ``` = 200.1 ms total Compared to: `http://127.0.0.1:8006/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1` Which is 353ms total. The separate queries ran faster! Really surprising result there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970853917,https://api.github.com/repos/simonw/datasette/issues/1513,970853917,IC_kwDOBm6k_c453g4d,9599,simonw,2021-11-16T23:41:01Z,2021-11-16T23:41:01Z,OWNER,"One very interesting difference between the two: on the single giant query page: ```json { ""request_duration_ms"": 376.4317020000476, ""sum_trace_duration_ms"": 370.0828700000329, ""num_traces"": 5 } ``` And on the page that uses separate queries: ```json { ""request_duration_ms"": 819.012272000009, ""sum_trace_duration_ms"": 201.52852100000018, ""num_traces"": 19 } ``` The separate pages page takes 819ms total to render the page, but spends 201ms across 19 SQL queries. The single big query takes 376ms total to render the page, spending 370ms in 5 queries
    Those 5 queries, if you're interested ```sql select database_name, schema_version from databases PRAGMA schema_version PRAGMA schema_version explain with cte as (\r\n select rowid, date, county, state, fips, cases, deaths\r\n from ny_times_us_counties\r\n),\r\ntruncated as (\r\n select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet with cte as (\r\n select rowid, date, county, state, fips, cases, deaths\r\n from ny_times_us_counties\r\n),\r\ntruncated as (\r\n select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet ```
    All of that additional non-SQL overhead must be stuff relating to Python and template rendering code running on the page. I'm really surprised at how much overhead that is! This is worth researching separately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/issues/1513#issuecomment-970855084,https://api.github.com/repos/simonw/datasette/issues/1513,970855084,IC_kwDOBm6k_c453hKs,9599,simonw,2021-11-16T23:41:46Z,2021-11-16T23:41:46Z,OWNER,Conclusion: using a giant convoluted CTE and UNION ALL query to attempt to calculate facets at the same time as retrieving rows is a net LOSS for performance! Very surprised to see that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time, https://github.com/simonw/datasette/pull/1512#issuecomment-970857411,https://api.github.com/repos/simonw/datasette/issues/1512,970857411,IC_kwDOBm6k_c453hvD,9599,simonw,2021-11-16T23:43:21Z,2021-11-16T23:43:21Z,OWNER,"``` E File ""/home/runner/work/datasette/datasette/datasette/utils/vendored_graphlib.py"", line 56 E if (result := self._node2info.get(node)) is None: E ^ E SyntaxError: invalid syntax ``` Oh no - the vendored code I use has `:=` so doesn't work on Python 3.6! Will have to backport it more thoroughly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/pull/1512#issuecomment-970861628,https://api.github.com/repos/simonw/datasette/issues/1512,970861628,IC_kwDOBm6k_c453iw8,9599,simonw,2021-11-16T23:46:07Z,2021-11-16T23:46:07Z,OWNER,"I made the changes locally and tested them with Python 3.6 like so: ``` cd /tmp mkdir v cd v pipenv shell --python=python3.6 cd ~/Dropbox/Development/datasette pip install -e '.[test]' pytest tests/test_asyncdi.py ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/issues/878#issuecomment-971209475,https://api.github.com/repos/simonw/datasette/issues/878,971209475,IC_kwDOBm6k_c4543sD,9599,simonw,2021-11-17T05:41:42Z,2021-11-17T05:41:42Z,OWNER,"I'm going to build a brand new implementation of the `TableView` class that doesn't subclass `BaseView` at all, instead using `asyncinject`. If I'm lucky that will clean up the grungiest part of the codebase. I can maybe even run the tests against old `TableView` and `TableView2` to check that they behave the same.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/pull/1512#issuecomment-971010724,https://api.github.com/repos/simonw/datasette/issues/1512,971010724,IC_kwDOBm6k_c454HKk,9599,simonw,2021-11-17T01:12:22Z,2021-11-17T01:12:22Z,OWNER,I'm going to extract out the `asyncinject` stuff into a separate library.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/pull/1512#issuecomment-971055677,https://api.github.com/repos/simonw/datasette/issues/1512,971055677,IC_kwDOBm6k_c454SI9,9599,simonw,2021-11-17T01:39:25Z,2021-11-17T01:39:25Z,OWNER,https://github.com/simonw/asyncinject version 0.1a0 is now live on PyPI: https://pypi.org/project/asyncinject/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/pull/1512#issuecomment-971056169,https://api.github.com/repos/simonw/datasette/issues/1512,971056169,IC_kwDOBm6k_c454SQp,9599,simonw,2021-11-17T01:39:44Z,2021-11-17T01:39:44Z,OWNER,Closing this PR because I shipped the code in it as a separate library instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes, https://github.com/simonw/datasette/issues/878#issuecomment-971057553,https://api.github.com/repos/simonw/datasette/issues/878,971057553,IC_kwDOBm6k_c454SmR,9599,simonw,2021-11-17T01:40:45Z,2021-11-17T01:40:45Z,OWNER,"I shipped that code as a new library, `asyncinject`: https://pypi.org/project/asyncinject/ - I'll open a new PR to attempt to refactor `TableView` to use it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/pull/1500#issuecomment-971568829,https://api.github.com/repos/simonw/datasette/issues/1500,971568829,IC_kwDOBm6k_c456Pa9,49699333,dependabot[bot],2021-11-17T13:13:58Z,2021-11-17T13:13:58Z,CONTRIBUTOR,Superseded by #1514.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1041158024,Bump black from 21.9b0 to 21.10b0, https://github.com/simonw/datasette/pull/1514#issuecomment-971575746,https://api.github.com/repos/simonw/datasette/issues/1514,971575746,IC_kwDOBm6k_c456RHC,22429695,codecov[bot],2021-11-17T13:18:58Z,2021-11-17T13:18:58Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1514](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b02c35a) into [main](https://codecov.io/gh/simonw/datasette/commit/0156c6b5e52d541e93f0d68e9245f20ae83bc933?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0156c6b) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1514/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1514 +/- ## ======================================= Coverage 91.82% 91.82% ======================================= Files 34 34 Lines 4430 4430 ======================================= Hits 4068 4068 Misses 362 362 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0156c6b...b02c35a](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1056117435,Bump black from 21.9b0 to 21.11b0, https://github.com/simonw/datasette/pull/1495#issuecomment-960420237,https://api.github.com/repos/simonw/datasette/issues/1495,960420237,IC_kwDOBm6k_c45PtmN,192568,mroswell,2021-11-04T03:12:01Z,2021-11-04T03:12:01Z,CONTRIBUTOR,"This all looks promising! I will need detailed documentation on how to upgrade datasette once it's available, and how to implement. (@fgregg example looks very straightforward on the plugin front.) I'll be so excited if I can get: https://list.saferdisinfectants.org/ instead of https://list.saferdisinfectants.org/disinfectants/listN ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options, https://github.com/simonw/datasette/pull/1514#issuecomment-972852184,https://api.github.com/repos/simonw/datasette/issues/1514,972852184,IC_kwDOBm6k_c45_IvY,49699333,dependabot[bot],2021-11-18T13:11:15Z,2021-11-18T13:11:15Z,CONTRIBUTOR,Superseded by #1516.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1056117435,Bump black from 21.9b0 to 21.11b0, https://github.com/simonw/datasette/pull/1516#issuecomment-972858458,https://api.github.com/repos/simonw/datasette/issues/1516,972858458,IC_kwDOBm6k_c45_KRa,22429695,codecov[bot],2021-11-18T13:19:01Z,2021-11-18T13:19:01Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1516](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a82c620) into [main](https://codecov.io/gh/simonw/datasette/commit/0156c6b5e52d541e93f0d68e9245f20ae83bc933?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0156c6b) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1516/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1516 +/- ## ======================================= Coverage 91.82% 91.82% ======================================= Files 34 34 Lines 4430 4430 ======================================= Hits 4068 4068 Misses 362 362 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0156c6b...a82c620](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1057340779,Bump black from 21.9b0 to 21.11b1, https://github.com/simonw/datasette/issues/1380#issuecomment-967181828,https://api.github.com/repos/simonw/datasette/issues/1380,967181828,IC_kwDOBm6k_c45pgYE,7094907,Segerberg,2021-11-12T15:00:18Z,2021-11-12T20:02:29Z,NONE,"There is no such option see https://github.com/simonw/datasette/issues/43. But you could write a plugin using the datasette.add_database(db, name=None) https://docs.datasette.io/en/stable/internals.html#add-database-db-name-none ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder, https://github.com/simonw/datasette/issues/1380#issuecomment-967801997,https://api.github.com/repos/simonw/datasette/issues/1380,967801997,IC_kwDOBm6k_c45r3yN,7094907,Segerberg,2021-11-13T08:05:37Z,2021-11-13T08:09:11Z,NONE,"@glasnt yeah I guess that could be an option. I run datasette on large databases > 75gb and the startup time is a bit slow for me even with -i --inspect-file options. Here's a quick sketch for a plugin that will reload db's in a folder that you set for the plugin in metadata.json. If you request /-reload-db new db's will be added. (You probably want to implement some authentication for this =) ) https://gist.github.com/Segerberg/b96a0e0a5389dce2396497323cda7042 ","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder, https://github.com/simonw/datasette/issues/1380#issuecomment-967747190,https://api.github.com/repos/simonw/datasette/issues/1380,967747190,IC_kwDOBm6k_c45rqZ2,813732,glasnt,2021-11-13T00:47:26Z,2021-11-13T00:47:26Z,CONTRIBUTOR,"Would it make sense to run datasette with a fswatch/inotifywait on a folder, then? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder, https://github.com/simonw/datasette/issues/1506#issuecomment-968192980,https://api.github.com/repos/simonw/datasette/issues/1506,968192980,IC_kwDOBm6k_c45tXPU,9599,simonw,2021-11-14T02:22:40Z,2021-11-14T02:22:40Z,OWNER,"I think the answer is to spot this case and link to `?_item_exact=x` instead of `?_item=x` - it looks like that's already recommended in the documentation here: https://docs.datasette.io/en/stable/json_api.html#column-filter-arguments > **?column__exact=value** or **?_column=value** > Returns rows where the specified column exactly matches the value. So maybe the facet selection rendering logic needs to spot this and link correctly to it?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052826038,Columns beginning with an underscore do not facet correctly, https://github.com/simonw/datasette/issues/1503#issuecomment-968207906,https://api.github.com/repos/simonw/datasette/issues/1503,968207906,IC_kwDOBm6k_c45ta4i,9599,simonw,2021-11-14T05:08:26Z,2021-11-14T05:08:26Z,OWNER,"Error: ``` def test_table_html_filter_form_column_options( path, expected_column_options, app_client ): response = app_client.get(path) assert response.status == 200 form = Soup(response.body, ""html.parser"").find(""form"") column_options = [ o.attrs.get(""value"") or o.string for o in form.select(""select[name=_filter_column] option"") ] > assert expected_column_options == column_options E AssertionError: assert ['- column -'...wid', 'value'] == ['- column -', 'value'] E At index 1 diff: 'rowid' != 'value' E Left contains one more item: 'value' E Use -v to get the full diff ``` This is because `rowid` isn't a table column but IS returned by the query used on that page. My solution: start with the query columns, but then add any table columns that were not already returned by the query to the end of the `filter_columns` list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1050163432,`?_nocol=` removes that column from the filter interface, https://github.com/simonw/datasette/issues/1507#issuecomment-968209560,https://api.github.com/repos/simonw/datasette/issues/1507,968209560,IC_kwDOBm6k_c45tbSY,9599,simonw,2021-11-14T05:26:36Z,2021-11-14T05:26:36Z,OWNER,"It looks like my builds there still run on Python 2! ``` git clone --no-single-branch --depth 50 https://github.com/simonw/datasette . git checkout --force de1e031713f47fbd51eb7239db3e7e6025fbf81a git clean -d -f -f python2.7 -mvirtualenv /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2 /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/bin/python -m pip install --upgrade --no-cache-dir pip setuptools /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/bin/python -m pip install --upgrade --no-cache-dir mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 sphinx<2 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<2.2 cat docs/conf.py ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968209616,https://api.github.com/repos/simonw/datasette/issues/1507,968209616,IC_kwDOBm6k_c45tbTQ,9599,simonw,2021-11-14T05:27:22Z,2021-11-14T05:27:22Z,OWNER,https://blog.readthedocs.com/default-python-3/ they started defaulting new projects to Python 3 back in Feb 2019 but clearly my project was created before then.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968209731,https://api.github.com/repos/simonw/datasette/issues/1507,968209731,IC_kwDOBm6k_c45tbVD,9599,simonw,2021-11-14T05:28:41Z,2021-11-14T05:28:41Z,OWNER,"I will try adding a `.readthedocs.yml` file: https://docs.readthedocs.io/en/stable/config-file/v2.html#python-version This might work: ``` version: 2 build: os: ubuntu-20.04 tools: python: ""3.9"" sphinx: configuration: docs/conf.py ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968209957,https://api.github.com/repos/simonw/datasette/issues/1507,968209957,IC_kwDOBm6k_c45tbYl,9599,simonw,2021-11-14T05:31:07Z,2021-11-14T05:31:07Z,OWNER,"Looks like ReadTheDocs builds started failing for `latest` a few weeks ago: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968210222,https://api.github.com/repos/simonw/datasette/issues/1507,968210222,IC_kwDOBm6k_c45tbcu,9599,simonw,2021-11-14T05:34:14Z,2021-11-14T05:34:14Z,OWNER,"Here's the new build using Python 3: https://readthedocs.org/projects/datasette/builds/15268482/ It's still broken. Here's one of many issue threads about it, this one has a workaround fix: https://github.com/readthedocs/readthedocs.org/issues/8616#issuecomment-952034858 > For future readers, the solution for this problem is to pin `docutils<0.18` in your `requirements.txt` file, and have a `.readthedocs.yaml` file with these contents: > > ``` > version: 2 > > python: > install: > - requirements: docs/requirements.txt > ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/issues/1507#issuecomment-968210842,https://api.github.com/repos/simonw/datasette/issues/1507,968210842,IC_kwDOBm6k_c45tbma,9599,simonw,2021-11-14T05:41:55Z,2021-11-14T05:41:55Z,OWNER,"Here's the build with that fix: https://readthedocs.org/projects/datasette/builds/15268498/ It passed and published the docs: https://docs.datasette.io/en/latest/changelog.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release, https://github.com/simonw/datasette/pull/1508#issuecomment-968904414,https://api.github.com/repos/simonw/datasette/issues/1508,968904414,IC_kwDOBm6k_c45wE7e,22429695,codecov[bot],2021-11-15T13:20:49Z,2021-11-15T13:20:49Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1508](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (299774b) into [main](https://codecov.io/gh/simonw/datasette/commit/502c02fa6dde6a8bb840af6c4c8cf858aa1db687?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (502c02f) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1508/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1508 +/- ## ======================================= Coverage 91.82% 91.82% ======================================= Files 34 34 Lines 4430 4430 ======================================= Hits 4068 4068 Misses 362 362 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [502c02f...299774b](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053655062,Update docutils requirement from <0.18 to <0.19, https://github.com/simonw/datasette/issues/448#issuecomment-969621662,https://api.github.com/repos/simonw/datasette/issues/448,969621662,IC_kwDOBm6k_c45y0Ce,9599,simonw,2021-11-16T01:32:04Z,2021-11-16T01:32:04Z,OWNER,"Tests are failing and I think it's because the facets come back in different orders, need a tie-breaker. https://github.com/simonw/datasette/runs/4219325197?check_suite_focus=true","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969436930,https://api.github.com/repos/simonw/datasette/issues/448,969436930,IC_kwDOBm6k_c45yG8C,9599,simonw,2021-11-15T23:31:58Z,2021-11-15T23:31:58Z,OWNER,"I think this SQL recipe may work instead: ```sql select * from ads_with_targets where 'people_who_match:interests:African-American Civil Rights Movement (1954—68)' in ( select value from json_each(target_names) ) and 'interests:Martin Luther King III' in ( select value from json_each(target_names) ) ``` https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select%0D%0A++*%0D%0Afrom%0D%0A++ads_with_targets%0D%0Awhere%0D%0A++%27people_who_match%3Ainterests%3AAfrican-American+Civil+Rights+Movement+%281954%E2%80%9468%29%27+in+%28%0D%0A++++select%0D%0A++++++value%0D%0A++++from%0D%0A++++++json_each%28target_names%29%0D%0A++%29%0D%0A++and+%27interests%3AMartin+Luther+King+III%27+in+%28%0D%0A++++select%0D%0A++++++value%0D%0A++++from%0D%0A++++++json_each%28target_names%29%0D%0A++%29&interests=&African=&Martin=","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/519#issuecomment-969433734,https://api.github.com/repos/simonw/datasette/issues/519,969433734,IC_kwDOBm6k_c45yGKG,9599,simonw,2021-11-15T23:26:11Z,2021-11-15T23:26:11Z,OWNER,"I'm happy with this as the goals for 1.0. I'm going to close this issue and create three tracking tickets for the three key themes: - https://github.com/simonw/datasette/issues/1509 - https://github.com/simonw/datasette/issues/1510 - https://github.com/simonw/datasette/issues/1511","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459590021,Decide what goes into Datasette 1.0, https://github.com/simonw/datasette/issues/448#issuecomment-969440918,https://api.github.com/repos/simonw/datasette/issues/448,969440918,IC_kwDOBm6k_c45yH6W,9599,simonw,2021-11-15T23:40:17Z,2021-11-15T23:40:35Z,OWNER,"Applied that fix to the `arraycontains` filter but I'm still getting bad results for the faceting: Should never get 182 results on a page that faceting against only 172 items. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969442215,https://api.github.com/repos/simonw/datasette/issues/448,969442215,IC_kwDOBm6k_c45yIOn,9599,simonw,2021-11-15T23:42:03Z,2021-11-15T23:42:03Z,OWNER,I think this code is wrong in the `ArrayFacet` class: https://github.com/simonw/datasette/blob/502c02fa6dde6a8bb840af6c4c8cf858aa1db687/datasette/facets.py#L357-L364,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969446972,https://api.github.com/repos/simonw/datasette/issues/448,969446972,IC_kwDOBm6k_c45yJY8,9599,simonw,2021-11-15T23:46:13Z,2021-11-15T23:46:13Z,OWNER,"It looks like the problem here is that some of the tags occur more than once in the documents: So they get counted more than once, hence the 182 count for something that couldn't possibly return more than 172 documents.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969449772,https://api.github.com/repos/simonw/datasette/issues/448,969449772,IC_kwDOBm6k_c45yKEs,9599,simonw,2021-11-15T23:48:37Z,2021-11-15T23:48:37Z,OWNER,"Given this query: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select%0D%0A++j.value+as+value%2C%0D%0A++count%28*%29+as+count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++id%2C%0D%0A++++++file%2C%0D%0A++++++clicks%2C%0D%0A++++++impressions%2C%0D%0A++++++text%2C%0D%0A++++++url%2C%0D%0A++++++spend_amount%2C%0D%0A++++++spend_currency%2C%0D%0A++++++created%2C%0D%0A++++++ended%2C%0D%0A++++++target_names%0D%0A++++from%0D%0A++++++ads_with_targets%0D%0A++++where%0D%0A++++++%3Ap0+in+%28%0D%0A++++++++select%0D%0A++++++++++value%0D%0A++++++++from%0D%0A++++++++++json_each%28%5Bads_with_targets%5D.%5Btarget_names%5D%29%0D%0A++++++%29%0D%0A++%29%0D%0A++join+json_each%28target_names%29+j%0D%0Agroup+by%0D%0A++j.value%0D%0Aorder+by%0D%0A++count+desc%2C%0D%0A++value%0D%0Alimit%0D%0A++31&p0=people_who_match%3Ainterests%3AAfrican-American+culture ```sql select j.value as value, count(*) as count from ( select id, file, clicks, impressions, text, url, spend_amount, spend_currency, created, ended, target_names from ads_with_targets where :p0 in ( select value from json_each([ads_with_targets].[target_names]) ) ) join json_each(target_names) j group by j.value order by count desc, value limit 31 ``` How can I return a count of the number of documents containing each tag, but not the number of total tags that match including duplicates?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969557008,https://api.github.com/repos/simonw/datasette/issues/448,969557008,IC_kwDOBm6k_c45ykQQ,9599,simonw,2021-11-16T00:56:09Z,2021-11-16T00:59:59Z,OWNER,"This looks like it might work: ```sql with inner as ( select * from ads_with_targets where :p0 in ( select value from json_each([ads_with_targets].[target_names]) ) ), deduped_array_items as ( select distinct j.value, inner.* from json_each([inner].[target_names]) j join inner ) select value, count(*) from deduped_array_items group by value order by count(*) desc ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969557972,https://api.github.com/repos/simonw/datasette/issues/448,969557972,IC_kwDOBm6k_c45ykfU,9599,simonw,2021-11-16T00:56:58Z,2021-11-16T00:56:58Z,OWNER,It uses a CTE which were introduced in SQLite 3.8 - and AWS Lambda Python 3.9 still provides 3.7 - but I've checked and I can use `pysqlite3-binary` to work around that there so I'm OK relying on CTEs for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969572281,https://api.github.com/repos/simonw/datasette/issues/448,969572281,IC_kwDOBm6k_c45yn-5,9599,simonw,2021-11-16T01:05:11Z,2021-11-16T01:05:11Z,OWNER,"I tried this and it seems to work correctly: ```python for source_and_config in self.get_configs(): config = source_and_config[""config""] source = source_and_config[""source""] column = config.get(""column"") or config[""simple""] facet_sql = """""" with inner as ({sql}), deduped_array_items as ( select distinct j.value, inner.* from json_each([inner].{col}) j join inner ) select value as value, count(*) as count from deduped_array_items group by value order by count(*) desc limit {limit} """""".format( col=escape_sqlite(column), sql=self.sql, limit=facet_size + 1 ) ``` The queries are _very_ slow though - I had to bump up to 2s time limit even against only a view returning 3,499 rows.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969578466,https://api.github.com/repos/simonw/datasette/issues/448,969578466,IC_kwDOBm6k_c45ypfi,9599,simonw,2021-11-16T01:08:29Z,2021-11-16T01:08:29Z,OWNER,Actually with the cache warmed up it looks like the facet query is taking 150ms which is good enough.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/448#issuecomment-969582098,https://api.github.com/repos/simonw/datasette/issues/448,969582098,IC_kwDOBm6k_c45yqYS,9599,simonw,2021-11-16T01:10:28Z,2021-11-16T01:10:28Z,OWNER,"Also note that this demo data is using a SQL view to create the JSON arrays - the view is defined as such: ```sql CREATE VIEW ads_with_targets as select ads.*, json_group_array(targets.name) as target_names from ads join ad_targets on ad_targets.ad_id = ads.id join targets on ad_targets.target_id = targets.id group by ad_targets.ad_id; ``` So running JSON faceting on top of that view is a pretty big ask!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views, https://github.com/simonw/datasette/issues/1511#issuecomment-969600859,https://api.github.com/repos/simonw/datasette/issues/1511,969600859,IC_kwDOBm6k_c45yu9b,9599,simonw,2021-11-16T01:20:13Z,2021-11-16T01:20:13Z,OWNER,"See: - #830","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054246919,Review plugin hooks for Datasette 1.0, https://github.com/simonw/datasette/issues/1012#issuecomment-969602825,https://api.github.com/repos/simonw/datasette/issues/1012,969602825,IC_kwDOBm6k_c45yvcJ,9599,simonw,2021-11-16T01:21:14Z,2021-11-16T01:21:14Z,OWNER,"I'd been wondering how to get new classifiers into Trove - thanks, I'll give this a go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py, https://github.com/simonw/datasette/issues/1012#issuecomment-969613166,https://api.github.com/repos/simonw/datasette/issues/1012,969613166,IC_kwDOBm6k_c45yx9u,9599,simonw,2021-11-16T01:27:25Z,2021-11-16T01:27:25Z,OWNER,"Requested here: - https://github.com/pypa/trove-classifiers/pull/85","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py, https://github.com/simonw/datasette/issues/1176#issuecomment-969616626,https://api.github.com/repos/simonw/datasette/issues/1176,969616626,IC_kwDOBm6k_c45yyzy,9599,simonw,2021-11-16T01:29:13Z,2021-11-16T01:29:13Z,OWNER,"I'm inclined to create a Sphinx reference documentation page for this, as I did for `sqlite-utils` here: https://sqlite-utils.datasette.io/en/stable/reference.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/pull/1543#issuecomment-986768401,https://api.github.com/repos/simonw/datasette/issues/1543,986768401,IC_kwDOBm6k_c460OQR,22429695,codecov[bot],2021-12-06T13:18:48Z,2021-12-06T13:18:48Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1543](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (353a851) into [main](https://codecov.io/gh/simonw/datasette/commit/7c02be2ee94cc64b120cc58b7a72cd387031f287?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7c02be2) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1543/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1543 +/- ## ======================================= Coverage 91.84% 91.84% ======================================= Files 34 34 Lines 4438 4438 ======================================= Hits 4076 4076 Misses 362 362 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [7c02be2...353a851](https://codecov.io/gh/simonw/datasette/pull/1543?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072135269,Bump black from 21.11b1 to 21.12b0, https://github.com/simonw/datasette/issues/1527#issuecomment-988154238,https://api.github.com/repos/simonw/datasette/issues/1527,988154238,IC_kwDOBm6k_c465gl-,9599,simonw,2021-12-07T18:05:26Z,2021-12-07T18:05:26Z,OWNER,"Found a new case of this bug: click the ""Apply"" button on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1 ![apply-bug](https://user-images.githubusercontent.com/9599/145082760-6947c769-480f-45c7-9916-b6cc7f5834f8.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1544#issuecomment-988226523,https://api.github.com/repos/simonw/datasette/issues/1544,988226523,IC_kwDOBm6k_c465yPb,9599,simonw,2021-12-07T20:02:00Z,2021-12-07T20:02:00Z,OWNER,Here's the code at fault: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/database.py#L288-L291,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1073712378,Code that detects the label column for a table is case-sensitive, https://github.com/simonw/datasette/issues/1544#issuecomment-988226938,https://api.github.com/repos/simonw/datasette/issues/1544,988226938,IC_kwDOBm6k_c465yV6,9599,simonw,2021-12-07T20:02:44Z,2021-12-07T20:02:44Z,OWNER,I'm feeling rushed today so I'm going to fix this without adding a test!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1073712378,Code that detects the label column for a table is case-sensitive, https://github.com/simonw/datasette/issues/1304#issuecomment-988459453,https://api.github.com/repos/simonw/datasette/issues/1304,988459453,IC_kwDOBm6k_c466rG9,9308268,rayvoelker,2021-12-08T03:15:27Z,2021-12-08T03:15:27Z,NONE,"I was thinking if there were a way to use some sort of sting function to ""unpack"" the values and convert them into ints... hm","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",863884805,"Document how to send multiple values for ""Named parameters"" ", https://github.com/simonw/datasette/issues/1304#issuecomment-988461884,https://api.github.com/repos/simonw/datasette/issues/1304,988461884,IC_kwDOBm6k_c466rs8,30934,20after4,2021-12-08T03:20:26Z,2021-12-08T03:20:26Z,NONE,"The easiest or most straightforward thing to do is to use named parameters like: ```sql select * where key IN (:p1, :p2, :p3) ``` And simply construct the list of placeholders dynamically based on the number of values. Doing this is possible with datasette if you forgo ""canned queries"" and just use the raw query endpoint and pass the query sql, along with p1, p2 ... in the request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",863884805,"Document how to send multiple values for ""Named parameters"" ", https://github.com/simonw/datasette/issues/1304#issuecomment-988463455,https://api.github.com/repos/simonw/datasette/issues/1304,988463455,IC_kwDOBm6k_c466sFf,30934,20after4,2021-12-08T03:23:14Z,2021-12-08T03:23:14Z,NONE,I actually think it would be a useful thing to add support for in datasette. It wouldn't be difficult to unwind an array of params and add the placeholders automatically.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",863884805,"Document how to send multiple values for ""Named parameters"" ", https://github.com/simonw/datasette/issues/1528#issuecomment-988468238,https://api.github.com/repos/simonw/datasette/issues/1528,988468238,IC_kwDOBm6k_c466tQO,30934,20after4,2021-12-08T03:35:45Z,2021-12-08T03:35:45Z,NONE,"FWIW I implemented something similar with a bit of plugin code: ```python @hookimpl def canned_queries(datasette: Datasette, database: str) -> Mapping[str, str]: # load ""canned queries"" from the filesystem under # www/sql/db/query_name.sql queries = {} sqldir = Path(__file__).parent.parent / ""sql"" if database: sqldir = sqldir / database if not sqldir.is_dir(): return queries for f in sqldir.glob('*.sql'): try: sql = f.read_text('utf8').strip() if not len(sql): log(f""Skipping empty canned query file: {f}"") continue queries[f.stem] = { ""sql"": sql } except OSError as err: log(err) return queries ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1060631257,"Add new `""sql_file""` key to Canned Queries in metadata?", https://github.com/simonw/datasette/issues/878#issuecomment-973554024,https://api.github.com/repos/simonw/datasette/issues/878,973554024,IC_kwDOBm6k_c46B0Fo,9599,simonw,2021-11-19T00:21:20Z,2021-11-19T00:21:20Z,OWNER,"That's annoying: it looks like plugins can't use `register_routes()` to over-ride default routes within Datasette itself. This didn't work: ```python from datasette.utils.asgi import Response from datasette import hookimpl import html async def table(request): return Response.html(""Hello from {}"".format( html.escape(repr(request.url_vars)) )) @hookimpl def register_routes(): return [ (r""/(?P[^/]+)/(?P[^/]+?$)"", table), ] ``` I'll use a `/t/` prefix for the moment, but this is probably something I'll fix in Datasette itself later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973564260,https://api.github.com/repos/simonw/datasette/issues/878,973564260,IC_kwDOBm6k_c46B2lk,9599,simonw,2021-11-19T00:27:06Z,2021-11-19T00:27:06Z,OWNER,"Problem: the fancy `asyncinject` stuff inteferes with the fancy Datasette thing that introspects view functions to look for what parameters they take: ```python class Table(asyncinject.AsyncInjectAll): async def view(self, request): return Response.html(""Hello from {}"".format( html.escape(repr(request.url_vars)) )) @hookimpl def register_routes(): return [ (r""/t/(?P[^/]+)/(?P[^/]+?$)"", Table().view), ] ``` This failed with error: ""Table.view() takes 1 positional argument but 2 were given"" So I'm going to use `AsyncInject` and have the `view` function NOT use the `@inject` decorator.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973568285,https://api.github.com/repos/simonw/datasette/issues/878,973568285,IC_kwDOBm6k_c46B3kd,9599,simonw,2021-11-19T00:29:20Z,2021-11-19T00:29:20Z,OWNER,"This is working! ```python from datasette.utils.asgi import Response from datasette import hookimpl import html from asyncinject import AsyncInject, inject class Table(AsyncInject): @inject async def database(self, request): return request.url_vars[""db_name""] @inject async def main(self, request, database): return Response.html(""Database: {}"".format( html.escape(database) )) async def view(self, request): return await self.main(request=request) @hookimpl def register_routes(): return [ (r""/t/(?P[^/]+)/(?P[^/]+?$)"", Table().view), ] ``` This project will definitely show me if I actually like the `asyncinject` patterns or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973527870,https://api.github.com/repos/simonw/datasette/issues/878,973527870,IC_kwDOBm6k_c46Bts-,9599,simonw,2021-11-19T00:13:43Z,2021-11-19T00:13:43Z,OWNER,"New plan: I'm going to build a brand new implementation of `TableView` starting out as a plugin, using the `register_routes()` plugin hook. It will reuse the existing HTML template but will be a completely new Python implementation, based on `asyncinject`. I'm going to start by just getting the table to show up on the page - then I'll add faceting, suggested facets, filters and so-on. Bonus: I'm going to see if I can get it to work for arbitrary SQL queries too (stretch goal).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973542284,https://api.github.com/repos/simonw/datasette/issues/878,973542284,IC_kwDOBm6k_c46BxOM,9599,simonw,2021-11-19T00:16:44Z,2021-11-19T00:16:44Z,OWNER,"``` Development % cookiecutter gh:simonw/datasette-plugin You've downloaded /Users/simon/.cookiecutters/datasette-plugin before. Is it okay to delete and re-download it? [yes]: yes plugin_name []: table-new description []: New implementation of TableView, see https://github.com/simonw/datasette/issues/878 hyphenated [table-new]: underscored [table_new]: github_username []: simonw author_name []: Simon Willison include_static_directory []: include_templates_directory []: ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973635157,https://api.github.com/repos/simonw/datasette/issues/878,973635157,IC_kwDOBm6k_c46CH5V,9599,simonw,2021-11-19T01:07:08Z,2021-11-19T01:07:08Z,OWNER,"This exercise is proving so useful in getting my head around how the enormous and complex `TableView` class works again. Here's where I've got to now - I'm systematically working through the variables that are returned for HTML and for JSON copying across code to get it to work: ```python from datasette.database import QueryInterrupted from datasette.utils import escape_sqlite from datasette.utils.asgi import Response, NotFound, Forbidden from datasette.views.base import DatasetteError from datasette import hookimpl from asyncinject import AsyncInject, inject from pprint import pformat class Table(AsyncInject): @inject async def database(self, request, datasette): # TODO: all that nasty hash resolving stuff can go here db_name = request.url_vars[""db_name""] try: db = datasette.databases[db_name] except KeyError: raise NotFound(f""Database '{db_name}' does not exist"") return db @inject async def table_and_format(self, request, database, datasette): table_and_format = request.url_vars[""table_and_format""] # TODO: be a lot smarter here if ""."" in table_and_format: return table_and_format.split(""."", 2) else: return table_and_format, ""html"" @inject async def main(self, request, database, table_and_format, datasette): # TODO: if this is actually a canned query, dispatch to it table, format = table_and_format is_view = bool(await database.get_view_definition(table)) table_exists = bool(await database.table_exists(table)) if not is_view and not table_exists: raise NotFound(f""Table not found: {table}"") await check_permissions( datasette, request, [ (""view-table"", (database.name, table)), (""view-database"", database.name), ""view-instance"", ], ) private = not await datasette.permission_allowed( None, ""view-table"", (database.name, table), default=True ) pks = await database.primary_keys(table) table_columns = await database.table_columns(table) specified_columns = await columns_to_select(datasette, database, table, request) select_specified_columns = "", "".join( escape_sqlite(t) for t in specified_columns ) select_all_columns = "", "".join(escape_sqlite(t) for t in table_columns) use_rowid = not pks and not is_view if use_rowid: select_specified_columns = f""rowid, {select_specified_columns}"" select_all_columns = f""rowid, {select_all_columns}"" order_by = ""rowid"" order_by_pks = ""rowid"" else: order_by_pks = "", "".join([escape_sqlite(pk) for pk in pks]) order_by = order_by_pks if is_view: order_by = """" nocount = request.args.get(""_nocount"") nofacet = request.args.get(""_nofacet"") if request.args.get(""_shape"") in (""array"", ""object""): nocount = True nofacet = True # Next, a TON of SQL to build where_params and filters and suchlike # skipping that and jumping straight to... where_clauses = [] where_clause = """" if where_clauses: where_clause = f""where {' and '.join(where_clauses)} "" from_sql = ""from {table_name} {where}"".format( table_name=escape_sqlite(table), where=(""where {} "".format("" and "".join(where_clauses))) if where_clauses else """", ) from_sql_params ={} params = {} count_sql = f""select count(*) {from_sql}"" sql_no_order_no_limit = ( ""select {select_all_columns} from {table_name} {where}"".format( select_all_columns=select_all_columns, table_name=escape_sqlite(table), where=where_clause, ) ) page_size = 100 offset = "" offset 0"" sql = ""select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}"".format( select_specified_columns=select_specified_columns, table_name=escape_sqlite(table), where=where_clause, order_by=order_by, page_size=page_size + 1, offset=offset, ) # Fetch rows results = await database.execute(sql, params, truncate=True) columns = [r[0] for r in results.description] rows = list(results.rows) # Fetch count filtered_table_rows_count = None if count_sql: try: count_rows = list(await database.execute(count_sql, from_sql_params)) filtered_table_rows_count = count_rows[0][0] except QueryInterrupted: pass vars = { ""json"": { # THIS STUFF is from the regular JSON ""database"": database.name, ""table"": table, ""is_view"": is_view, # ""human_description_en"": human_description_en, ""rows"": rows[:page_size], ""truncated"": results.truncated, ""filtered_table_rows_count"": filtered_table_rows_count, # ""expanded_columns"": expanded_columns, # ""expandable_columns"": expandable_columns, ""columns"": columns, ""primary_keys"": pks, # ""units"": units, ""query"": {""sql"": sql, ""params"": params}, # ""facet_results"": facet_results, # ""suggested_facets"": suggested_facets, # ""next"": next_value and str(next_value) or None, # ""next_url"": next_url, ""private"": private, ""allow_execute_sql"": await datasette.permission_allowed( request.actor, ""execute-sql"", database, default=True ), }, ""html"": { # ... this is the HTML special stuff # ""table_actions"": table_actions, # ""supports_search"": bool(fts_table), # ""search"": search or """", ""use_rowid"": use_rowid, # ""filters"": filters, # ""display_columns"": display_columns, # ""filter_columns"": filter_columns, # ""display_rows"": display_rows, # ""facets_timed_out"": facets_timed_out, # ""sorted_facet_results"": sorted( # facet_results.values(), # key=lambda f: (len(f[""results""]), f[""name""]), # reverse=True, # ), # ""show_facet_counts"": special_args.get(""_facet_size"") == ""max"", # ""extra_wheres_for_ui"": extra_wheres_for_ui, # ""form_hidden_args"": form_hidden_args, # ""is_sortable"": any(c[""sortable""] for c in display_columns), # ""path_with_replaced_args"": path_with_replaced_args, # ""path_with_removed_args"": path_with_removed_args, # ""append_querystring"": append_querystring, ""request"": request, # ""sort"": sort, # ""sort_desc"": sort_desc, ""disable_sort"": is_view, # ""custom_table_templates"": [ # f""_table-{to_css_class(database)}-{to_css_class(table)}.html"", # f""_table-table-{to_css_class(database)}-{to_css_class(table)}.html"", # ""_table.html"", # ], # ""metadata"": metadata, # ""view_definition"": await db.get_view_definition(table), # ""table_definition"": await db.get_table_definition(table), }, } # I'm just trying to get HTML to work for the moment if format == ""json"": return Response.json(dict(vars, locals=locals()), default=repr) else: return Response.html(repr(vars[""html""])) async def view(self, request, datasette): return await self.main(request=request, datasette=datasette) @hookimpl def register_routes(): return [ (r""/t/(?P[^/]+)/(?P[^/]+?$)"", Table().view), ] async def check_permissions(datasette, request, permissions): """"""permissions is a list of (action, resource) tuples or 'action' strings"""""" for permission in permissions: if isinstance(permission, str): action = permission resource = None elif isinstance(permission, (tuple, list)) and len(permission) == 2: action, resource = permission else: assert ( False ), ""permission should be string or tuple of two items: {}"".format( repr(permission) ) ok = await datasette.permission_allowed( request.actor, action, resource=resource, default=None, ) if ok is not None: if ok: return else: raise Forbidden(action) async def columns_to_select(datasette, database, table, request): table_columns = await database.table_columns(table) pks = await database.primary_keys(table) columns = list(table_columns) if ""_col"" in request.args: columns = list(pks) _cols = request.args.getlist(""_col"") bad_columns = [column for column in _cols if column not in table_columns] if bad_columns: raise DatasetteError( ""_col={} - invalid columns"".format("", "".join(bad_columns)), status=400, ) # De-duplicate maintaining order: columns.extend(dict.fromkeys(_cols)) if ""_nocol"" in request.args: # Return all columns EXCEPT these bad_columns = [ column for column in request.args.getlist(""_nocol"") if (column not in table_columns) or (column in pks) ] if bad_columns: raise DatasetteError( ""_nocol={} - invalid columns"".format("", "".join(bad_columns)), status=400, ) tmp_columns = [ column for column in columns if column not in request.args.getlist(""_nocol"") ] columns = tmp_columns return columns ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/878#issuecomment-973678931,https://api.github.com/repos/simonw/datasette/issues/878,973678931,IC_kwDOBm6k_c46CSlT,9599,simonw,2021-11-19T02:51:17Z,2021-11-19T02:51:17Z,OWNER,"OK, I managed to get a table to render! Here's the code I used - I had to copy a LOT of stuff. https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 I'm going to move this work into a new, separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1518#issuecomment-973681970,https://api.github.com/repos/simonw/datasette/issues/1518,973681970,IC_kwDOBm6k_c46CTUy,9599,simonw,2021-11-19T02:56:31Z,2021-11-19T02:56:53Z,OWNER,"Here's where I got to with my hacked-together initial plugin prototype - it managed to render the table page with some rows on it (and a bunch of missing functionality such as filters): https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-973682389,https://api.github.com/repos/simonw/datasette/issues/1518,973682389,IC_kwDOBm6k_c46CTbV,9599,simonw,2021-11-19T02:57:39Z,2021-11-19T02:57:39Z,OWNER,"Ideally I'd like to execute the existing test suite against the new implementation - that would require me to solve this so I can replace the view with the plugin version though: - #1517 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1517#issuecomment-973686874,https://api.github.com/repos/simonw/datasette/issues/1517,973686874,IC_kwDOBm6k_c46CUha,9599,simonw,2021-11-19T03:06:58Z,2021-11-19T03:06:58Z,OWNER,"I made a mistake: I just wrote a test that proves that plugins CAN over-ride default routes, plus if you look at the code here the plugins get to register themselves first: https://github.com/simonw/datasette/blob/0156c6b5e52d541e93f0d68e9245f20ae83bc933/datasette/app.py#L965-L981","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1057996111,Let `register_routes()` over-ride default routes within Datasette, https://github.com/simonw/datasette/issues/1518#issuecomment-973687978,https://api.github.com/repos/simonw/datasette/issues/1518,973687978,IC_kwDOBm6k_c46CUyq,9599,simonw,2021-11-19T03:07:47Z,2021-11-19T03:07:47Z,OWNER,"I was wrong about that, you CAN over-ride default routes already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1517#issuecomment-973696604,https://api.github.com/repos/simonw/datasette/issues/1517,973696604,IC_kwDOBm6k_c46CW5c,9599,simonw,2021-11-19T03:20:00Z,2021-11-19T03:20:00Z,OWNER,Confirmed - my test plugin is indeed correctly over-riding the table page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1057996111,Let `register_routes()` over-ride default routes within Datasette, https://github.com/simonw/datasette/issues/1518#issuecomment-973700549,https://api.github.com/repos/simonw/datasette/issues/1518,973700549,IC_kwDOBm6k_c46CX3F,9599,simonw,2021-11-19T03:31:20Z,2021-11-19T03:31:26Z,OWNER,"... and while I'm doing all of this I can rewrite the templates to not use those cheating magical functions AND document the template context at the same time, refs: - #1510.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-973698917,https://api.github.com/repos/simonw/datasette/issues/1518,973698917,IC_kwDOBm6k_c46CXdl,9599,simonw,2021-11-19T03:26:18Z,2021-11-19T03:29:03Z,OWNER,"A (likely incomplete) list of features on the table page: - [ ] Display table/database/instance metadata - [ ] Show count of all results - [ ] Display table of results - [ ] Special table display treatment for URLs, numbers - [ ] Allow plugins to modify table cells - [ ] Respect `?_col=` and `?_nocol=` - [ ] Show interface for filtering by columns and operations - [ ] Show search box, support executing FTS searches - [ ] Sort table by specified column - [ ] Paginate table - [ ] Show facet results - [ ] Show suggested facets - [ ] Link to available exports - [ ] Display schema for table - [ ] Maybe it should show the SQL for the query too? - [ ] Handle various non-obvious querystring options, like `?_where=` and `?_through=`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-973699424,https://api.github.com/repos/simonw/datasette/issues/1518,973699424,IC_kwDOBm6k_c46CXlg,9599,simonw,2021-11-19T03:27:49Z,2021-11-19T03:27:49Z,OWNER,"My goal is to break up a lot of this functionality into separate methods. These methods can be executed in parallel by `asyncinject`, but more importantly they can be used to build a much better JSON representation, where the default representation is lighter and `?_extra=x` options can be used to execute more expensive portions and add them to the response. So the HTML version itself needs to be re-written to use those JSON extras.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-973700322,https://api.github.com/repos/simonw/datasette/issues/1518,973700322,IC_kwDOBm6k_c46CXzi,9599,simonw,2021-11-19T03:30:30Z,2021-11-19T03:30:30Z,OWNER,"Right now the HTML version gets to cheat - it passes through objects that are not JSON serializable, including custom functions that can then be called by Jinja. I'm interested in maybe removing this cheating - if the HTML version could only request JSON-serializable extras those could be exposed in the API as well. It would also help cleanup the kind-of-nasty pattern I use in the current `BaseView` where everything returns both a bunch of JSON-serializable data AND an awaitable function that then gets to add extra things to the HTML context.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/pull/1495#issuecomment-974108455,https://api.github.com/repos/simonw/datasette/issues/1495,974108455,IC_kwDOBm6k_c46D7cn,192568,mroswell,2021-11-19T14:14:35Z,2021-11-19T14:14:35Z,CONTRIBUTOR,A nudge on this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options, https://github.com/simonw/datasette/issues/1521#issuecomment-974371116,https://api.github.com/repos/simonw/datasette/issues/1521,974371116,IC_kwDOBm6k_c46E7ks,9599,simonw,2021-11-19T19:45:47Z,2021-11-19T19:45:47Z,OWNER,"https://github.com/krallin/tini says: > *NOTE: If you are using Docker 1.13 or greater, Tini is included in Docker itself. This includes all versions of Docker CE. To enable Tini, just [pass the `--init` flag to `docker run`](https://docs.docker.com/engine/reference/commandline/run/).*","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974380798,https://api.github.com/repos/simonw/datasette/issues/1521,974380798,IC_kwDOBm6k_c46E97-,9599,simonw,2021-11-19T19:54:26Z,2021-11-19T19:54:26Z,OWNER,"Got it working! Here's a `Dockerfile` which runs completely stand-alone (thanks to using the `echo $'` trick to write out the config files it needs) and successfully serves Datasette behind Apache and `mod_proxy`: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini # Append this to the end of the default httpd.conf file RUN echo $'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPass / http://localhost:9000/\n\ ProxyPassReverse / http://localhost:9000/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf WORKDIR /app RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette -p 9000 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""] ``` Run it like this: ``` docker build -t datasette-apache2-proxy . docker run -p 5000:80 --rm datasette-apache2-proxy ``` Then run this to confirm: ``` ~ % curl -i 'http://localhost:5000/-/versions.json' HTTP/1.1 200 OK Date: Fri, 19 Nov 2021 19:54:05 GMT Server: uvicorn content-type: application/json; charset=utf-8 X-Proxied-By: Apache2 Transfer-Encoding: chunked {""python"": {""version"": ""3.10.0"", ""full"": ""3.10.0 (default, Nov 13 2021, 03:23:03) [GCC 10.3.1 20210424]""}, ""datasette"": {""version"": ""0.59.2""}, ""asgi"": ""3.0"", ""uvicorn"": ""0.15.0"", ""sqlite"": {""version"": ""3.35.5"", ""fts_versions"": [""FTS5"", ""FTS4"", ""FTS3""], ""extensions"": {""json1"": null}, ""compile_options"": [""COMPILER=gcc-10.3.1 20210424"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_DBSTAT_VTAB"", ""ENABLE_FTS3"", ""ENABLE_FTS3_PARENTHESIS"", ""ENABLE_FTS4"", ""ENABLE_FTS5"", ""ENABLE_GEOPOLY"", ""ENABLE_JSON1"", ""ENABLE_MATH_FUNCTIONS"", ""ENABLE_RTREE"", ""ENABLE_UNLOCK_NOTIFY"", ""MAX_VARIABLE_NUMBER=250000"", ""SECURE_DELETE"", ""THREADSAFE=1"", ""USE_URI""]}} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974388295,https://api.github.com/repos/simonw/datasette/issues/1521,974388295,IC_kwDOBm6k_c46E_xH,9599,simonw,2021-11-19T20:00:06Z,2021-11-19T20:00:06Z,OWNER,"And this is the version that proxies to a `base_url` of `/foo/bar/`: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini # Append this to the end of the default httpd.conf file RUN echo $'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPass /foo/bar/ http://localhost:9000/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo $'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette fixtures.db --setting base_url ""/foo/bar/"" -p 9000 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1518#issuecomment-974285803,https://api.github.com/repos/simonw/datasette/issues/1518,974285803,IC_kwDOBm6k_c46Emvr,9599,simonw,2021-11-19T17:56:48Z,2021-11-19T18:14:30Z,OWNER,"Very confused by this piece of code here: https://github.com/simonw/datasette/blob/1c13e1af0664a4dfb1e69714c56523279cae09e4/datasette/views/table.py#L37-L63 I added it in https://github.com/simonw/datasette/commit/754836eef043676e84626c4fd3cb993eed0d2976 - in the new world that should probably be replaced by pure JSON. Aha - this comment explains it: https://github.com/simonw/datasette/issues/521#issuecomment-505279560 > I think the trick is to redefine what a ""cell_row"" is. Each row is currently a list of cells: > > https://github.com/simonw/datasette/blob/6341f8cbc7833022012804dea120b838ec1f6558/datasette/views/table.py#L159-L163 > > I can redefine the row (the `cells` variable in the above example) as a thing-that-iterates-cells (hence behaving like a list) but that also supports `__getitem__` access for looking up cell values if you know the name of the column. The goal was to support neater custom templates like this: ```html+jinja {% for row in display_rows %}

    {{ row[""First_Name""] }} {{ row[""Last_Name""] }}

    ... ``` This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-974287570,https://api.github.com/repos/simonw/datasette/issues/1518,974287570,IC_kwDOBm6k_c46EnLS,9599,simonw,2021-11-19T17:59:33Z,2021-11-19T17:59:33Z,OWNER,"I'm going to try leaning into the `asyncinject` mechanism a bit here. One method can execute and return the raw rows. Another can turn that into the default minimal JSON representation. Then a third can take that (or take both) and use it to inflate out the JSON that the HTML template needs, with those extras and with the rendered cells from plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-974300823,https://api.github.com/repos/simonw/datasette/issues/1518,974300823,IC_kwDOBm6k_c46EqaX,9599,simonw,2021-11-19T18:18:32Z,2021-11-19T18:18:32Z,OWNER,"> This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more. I can definitely support this using pure-JSON - I could make two versions of the row available, one that's an array of cell objects and the other that's an object mapping column names to column raw values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1520#issuecomment-974308215,https://api.github.com/repos/simonw/datasette/issues/1520,974308215,IC_kwDOBm6k_c46EsN3,9599,simonw,2021-11-19T18:29:26Z,2021-11-19T18:29:26Z,OWNER,"The solution that jumps to mind first is that it would be neat if routes could return something that meant ""actually my bad, I can't handle this after all - move to the next one in the list"". A related idea: it might be useful for custom views like my one here to say ""no actually call the default view for this, but give me back the response so I can modify it in some way"". Kind of like Django or ASGI middleware.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058803238,Pattern for avoiding accidental URL over-rides, https://github.com/simonw/datasette/issues/1519#issuecomment-974309591,https://api.github.com/repos/simonw/datasette/issues/1519,974309591,IC_kwDOBm6k_c46EsjX,9599,simonw,2021-11-19T18:31:32Z,2021-11-19T18:31:32Z,OWNER,"`base_url` has been a source of so many bugs like this! I often find them quite hard to replicate, likely because I haven't made myself a good Apache `mod_proxy` testing environment yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974310208,https://api.github.com/repos/simonw/datasette/issues/1519,974310208,IC_kwDOBm6k_c46EstA,9599,simonw,2021-11-19T18:32:31Z,2021-11-19T18:32:31Z,OWNER,Having a live demo running on Cloud Run that proxies through Apache and uses `base_url` would be incredibly useful for replicating and debugging this kind of thing. I wonder how hard it is to run Apache and `mod_proxy` in the same Docker container as Datasette?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1521#issuecomment-974321391,https://api.github.com/repos/simonw/datasette/issues/1521,974321391,IC_kwDOBm6k_c46Evbv,9599,simonw,2021-11-19T18:49:15Z,2021-11-19T18:57:18Z,OWNER,"This pattern looks like it can help: https://ahmet.im/blog/cloud-run-multiple-processes-easy-way/ - see example in https://github.com/ahmetb/multi-process-container-lazy-solution I got that demo working locally like this: ```bash cd /tmp git clone https://github.com/ahmetb/multi-process-container-lazy-solution cd multi-process-container-lazy-solution docker build -t multi-process-container-lazy-solution . docker run -p 5000:8080 --rm multi-process-container-lazy-solution ``` I want to use `apache2` rather than `nginx` though. I found a few relevant examples of Apache in Alpine: - https://github.com/Hacking-Lab/alpine-apache2-reverse-proxy/blob/master/Dockerfile - https://www.sentiatechblog.com/running-apache-in-a-docker-container - https://github.com/search?l=Dockerfile&q=alpine+apache2&type=code ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974322178,https://api.github.com/repos/simonw/datasette/issues/1521,974322178,IC_kwDOBm6k_c46EvoC,9599,simonw,2021-11-19T18:50:22Z,2021-11-19T18:50:22Z,OWNER,"I'll get this working on my laptop first, but then I want to get it up and running on Cloud Run - maybe with a GitHub Actions workflow in this repo that re-deploys it on manual execution.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974327812,https://api.github.com/repos/simonw/datasette/issues/1521,974327812,IC_kwDOBm6k_c46ExAE,9599,simonw,2021-11-19T18:58:49Z,2021-11-19T18:59:55Z,OWNER,"From this example: https://github.com/tigelane/dockerfiles/blob/06cff2ac8cdc920ebd64f50965115eaa3d0afb84/Alpine-Apache2/Dockerfile#L25-L31 it looks like running `apk add apache2` installs a config file at `/etc/apache2/httpd.conf` - so one approach is to then modify that file. ``` # APACHE - Alpine ################# RUN apk --update add apache2 php5-apache2 && \ #apk add openrc --no-cache && \ rm -rf /var/cache/apk/* && \ sed -i 's/#ServerName www.example.com:80/ServerName localhost/' /etc/apache2/httpd.conf && \ mkdir -p /run/apache2/ # Upload our files from folder ""dist"". COPY dist /var/www/localhost/htdocs # Manually set up the apache environment variables ENV APACHE_RUN_USER www-data ENV APACHE_RUN_GROUP www-data ENV APACHE_LOG_DIR /var/log/apache2 ENV APACHE_LOCK_DIR /var/lock/apache2 ENV APACHE_PID_FILE /var/run/apache2.pid # Execute apache2 on run ######################## EXPOSE 80 ENTRYPOINT [""httpd""] CMD [""-D"", ""FOREGROUND""] ``` I think I'll create my own separate copy and modify that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974332787,https://api.github.com/repos/simonw/datasette/issues/1521,974332787,IC_kwDOBm6k_c46EyNz,9599,simonw,2021-11-19T19:05:52Z,2021-11-19T19:05:52Z,OWNER,"Made myself this Dockerfile to let me explore a bit: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 CMD [""sh""] ``` Then: ``` % docker run alpine-apache2-sh % docker run -it alpine-apache2-sh / # ls /etc/apache2/httpd.conf /etc/apache2/httpd.conf / # cat /etc/apache2/httpd.conf # # This is the main Apache HTTP server configuration file. It contains the # configuration directives that give the server its instructions. ... ``` Copying that into a GIST like so: ``` docker run -it --entrypoint sh alpine-apache2-sh -c ""cat /etc/apache2/httpd.conf"" | pbcopy ``` Gist here: https://gist.github.com/simonw/5ea0db6049192cb9f761fbd6beb3a84a","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974334278,https://api.github.com/repos/simonw/datasette/issues/1521,974334278,IC_kwDOBm6k_c46EylG,9599,simonw,2021-11-19T19:08:09Z,2021-11-19T19:08:09Z,OWNER,"Stripping comments using this StackOverflow recipe: https://unix.stackexchange.com/a/157619 docker run -it --entrypoint sh alpine-apache2-sh \ -c ""cat /etc/apache2/httpd.conf"" | sed '/^[[:blank:]]*#/d;s/#.*//' Result is here: https://gist.github.com/simonw/0a05090df5fcff8e8b3334621fa17976","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1521#issuecomment-974336020,https://api.github.com/repos/simonw/datasette/issues/1521,974336020,IC_kwDOBm6k_c46EzAU,9599,simonw,2021-11-19T19:10:48Z,2021-11-19T19:10:48Z,OWNER,"There's a promising looking minimal Apache 2 proxy config here: https://stackoverflow.com/questions/26474476/minimal-configuration-for-apache-reverse-proxy-in-docker-container ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974602459,https://api.github.com/repos/simonw/datasette/issues/1522,974602459,IC_kwDOBm6k_c46F0Db,9599,simonw,2021-11-20T06:15:58Z,2021-11-20T06:15:58Z,OWNER,First I'm going to try using Debian Buster as the base image instead of Alpine.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974605128,https://api.github.com/repos/simonw/datasette/issues/1522,974605128,IC_kwDOBm6k_c46F0tI,9599,simonw,2021-11-20T06:47:59Z,2021-11-20T06:47:59Z,OWNER,"I managed to port the whole thing over to Debian - which took a lot of work because their packaged Apache 2 works very differently from the Alpine one. Once again... I got it working fine on my laptop, but the image deployed to Cloud Run throws 503 errors! ```dockerfile FROM python:3.9.7-slim-bullseye RUN apt-get update && \ apt-get install -y apache2 supervisor && \ apt clean && \ rm -rf /var/lib/apt && \ rm -rf /var/lib/dpkg/info/* # Apache environment, copied from # https://github.com/ijklim/laravel-benfords-law-app/blob/e9bf385dcaddb62ea466a7b245ab6e4ef708c313/docker/os/Dockerfile ENV APACHE_DOCUMENT_ROOT=/var/www/html/public ENV APACHE_RUN_USER www-data ENV APACHE_RUN_GROUP www-data ENV APACHE_PID_FILE /var/run/apache2.pid ENV APACHE_RUN_DIR /var/run/apache2 ENV APACHE_LOCK_DIR /var/lock/apache2 ENV APACHE_LOG_DIR /var/log RUN ln -sf /dev/stdout /var/log/apache2-access.log RUN ln -sf /dev/stderr /var/log/apache2-error.log RUN mkdir -p $APACHE_RUN_DIR $APACHE_LOCK_DIR RUN a2enmod proxy RUN a2enmod proxy_http RUN a2enmod headers ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip # Append this to the end of the default httpd.conf file RUN echo '\n\ \n\ Options Indexes FollowSymLinks\n\ AllowOverride None\n\ Require all granted\n\ \n\ \n\ \n\ ServerName localhost\n\ DocumentRoot /app/html\n\ ProxyPreserveHost On\n\ ProxyPass /prefix/ http://127.0.0.1:8001/\n\ Header add X-Proxied-By ""Apache2""\n\ \n\ ' > /etc/apache2/sites-enabled/000-default.conf WORKDIR /app RUN mkdir -p /app/html RUN echo 'Datasette' > /app/html/index.html ADD https://latest.datasette.io/fixtures.db /app/fixtures.db EXPOSE 80 RUN echo ""[supervisord]"" >> /app/supervisord.conf RUN echo ""nodaemon=true"" >> /app/supervisord.conf RUN echo """" >> /app/supervisord.conf RUN echo ""[program:apache2]"" >> /app/supervisord.conf RUN echo ""command=apache2 -D FOREGROUND"" >> /app/supervisord.conf RUN echo """" >> /app/supervisord.conf RUN echo ""[program:datasette]"" >> /app/supervisord.conf RUN echo ""command=datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001"" >> /app/supervisord.conf CMD [""/usr/bin/supervisord"", ""-c"", ""/app/supervisord.conf""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974605529,https://api.github.com/repos/simonw/datasette/issues/1522,974605529,IC_kwDOBm6k_c46F0zZ,9599,simonw,2021-11-20T06:52:21Z,2021-11-20T06:52:21Z,OWNER,"I've now tried both Debian and Alpine, and I've tried both `tini` and `supervisord`. Each time I get the same result - I get 503 errors for the first dozen or so refreshes of `/prefix/` followed by it intermittently working. Absolutely stumped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974607456,https://api.github.com/repos/simonw/datasette/issues/1522,974607456,IC_kwDOBm6k_c46F1Rg,17906,mrchrisadams,2021-11-20T07:10:11Z,2021-11-20T07:10:11Z,NONE,"As a a sanity check, would it be worth looking at trying to push the multi-process container on another provider of a knative / cloud run / tekton ? I have a somewhat similar use case for a future proejct, so i'm been very grateful to you sharing all the progress in this issue. As I understand it, Scaleway also offer a very similar offering using what appear to be many similar components that might at least see if it's an issue with more than one knative based FaaS provider https://www.scaleway.com/en/serverless-containers/ https://developers.scaleway.com/en/products/containers/api/#main-features ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974389472,https://api.github.com/repos/simonw/datasette/issues/1519,974389472,IC_kwDOBm6k_c46FADg,9599,simonw,2021-11-19T20:01:02Z,2021-11-19T20:01:02Z,OWNER,I now have a `Dockerfile` in https://github.com/simonw/datasette/issues/1521#issuecomment-974388295 that I can use to run a local Apache 2 with `mod_proxy` to investigate this class of bugs!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974391204,https://api.github.com/repos/simonw/datasette/issues/1519,974391204,IC_kwDOBm6k_c46FAek,9599,simonw,2021-11-19T20:02:41Z,2021-11-19T20:02:41Z,OWNER,"Bug confirmed: ![proxy-bug](https://user-images.githubusercontent.com/9599/142684666-112136bf-9243-4b6e-8202-339fcfe91bcc.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974398399,https://api.github.com/repos/simonw/datasette/issues/1519,974398399,IC_kwDOBm6k_c46FCO_,9599,simonw,2021-11-19T20:08:20Z,2021-11-19T20:22:02Z,OWNER,"The relevant test is this one: https://github.com/simonw/datasette/blob/30255055150d7bc0affc8156adc18295495020ff/tests/test_html.py#L1608-L1649 I modified that test to add `""/fixtures/facetable?sql=select+1""` as one of the tested paths, and dropped in an `assert False` to pause it in the debugger: ``` @pytest.mark.parametrize( ""path"", [ ""/"", ""/fixtures"", ""/fixtures/compound_three_primary_keys"", ""/fixtures/compound_three_primary_keys/a,a,a"", ""/fixtures/paginated_view"", ""/fixtures/facetable"", ""/fixtures?sql=select+1"", ], ) def test_base_url_config(app_client_base_url_prefix, path): client = app_client_base_url_prefix response = client.get(""/prefix/"" + path.lstrip(""/"")) soup = Soup(response.body, ""html.parser"") if path == ""/fixtures?sql=select+1"": > assert False E assert False ``` BUT... in the debugger: ``` (Pdb) print(soup) ...

    This data as json, testall, testnone, testresponse, CSV

    ``` Those all have the correct prefix! But that's not what I'm seeing in my `Dockerfile` reproduction of the issue. Something very weird is going on here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974405016,https://api.github.com/repos/simonw/datasette/issues/1519,974405016,IC_kwDOBm6k_c46FD2Y,9599,simonw,2021-11-19T20:14:19Z,2021-11-19T20:15:05Z,OWNER,"I added `template_debug` in the Dockerfile: ``` datasette fixtures.db --setting template_debug 1 --setting base_url ""/foo/bar/"" -p 9000 &\n\ ``` And then hit `http://localhost:5000/foo/bar/fixtures?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1` to view the template context - and it showed the bug, output edited to just show relevant keys: ```json { ""edit_sql_url"": ""/foo/bar/fixtures?sql=select+%2A+from+compound_three_primary_keys+limit+1"", ""settings"": { ""force_https_urls"": false, ""template_debug"": true, ""trace_debug"": false, ""base_url"": ""/foo/bar/"" }, ""show_hide_link"": ""/fixtures?sql=select+%2A+from+compound_three_primary_keys+limit+1&_context=1&_hide_sql=1"", ""show_hide_text"": ""hide"", ""show_hide_hidden"": """", ""renderers"": { ""json"": ""/fixtures.json?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1"" }, ""url_csv"": ""/fixtures.csv?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1&_size=max"", ""url_csv_path"": ""/fixtures.csv"", ""base_url"": ""/foo/bar/"" } ``` This is so strange. `edit_sql_url` and `base_url` are correct, but `show_hide_link` and `url_csv` and `renderers.json` are not. And it's _really strange_ that the bug doesn't show up in the tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974418496,https://api.github.com/repos/simonw/datasette/issues/1519,974418496,IC_kwDOBm6k_c46FHJA,9599,simonw,2021-11-19T20:24:16Z,2021-11-19T20:24:16Z,OWNER,"Here's the code that generates `edit_sql_url` correctly: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/database.py#L416-L420 And here's the code for `show_hide_link`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/database.py#L432-L433 And for `url_csv`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/base.py#L600-L602","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974420619,https://api.github.com/repos/simonw/datasette/issues/1519,974420619,IC_kwDOBm6k_c46FHqL,9599,simonw,2021-11-19T20:25:19Z,2021-11-19T20:25:19Z,OWNER,"The implementations of `path_with_removed_args` and `path_with_format`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L228-L254 https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L710-L729","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974422829,https://api.github.com/repos/simonw/datasette/issues/1519,974422829,IC_kwDOBm6k_c46FIMt,9599,simonw,2021-11-19T20:26:35Z,2021-11-19T20:26:35Z,OWNER,"In the `?_context=` debug view the request looks like this: ``` ""request"": """", ``` I'm going to add a `repr()` to it such that it's a bit more useful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974433206,https://api.github.com/repos/simonw/datasette/issues/1519,974433206,IC_kwDOBm6k_c46FKu2,9599,simonw,2021-11-19T20:31:52Z,2021-11-19T20:31:52Z,OWNER,"Modified my `Dockerfile` to do this: RUN pip install https://github.com/simonw/datasette/archive/ff0dd4da38d48c2fa9250ecf336002c9ed724e36.zip And now the `request` in that debug `?_context=1` looks like this: ``` ""request"": """" ``` That explains the bug - that request doesn't maintain the original path prefix of `http://localhost:5000/foo/bar/fixtures?sql=` (also it's been rewritten to `localhost:9000` instead of `localhost:5000`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974433320,https://api.github.com/repos/simonw/datasette/issues/1519,974433320,IC_kwDOBm6k_c46FKwo,9599,simonw,2021-11-19T20:32:04Z,2021-11-19T20:32:04Z,OWNER,Still not clear why the tests pass but the live example fails.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1521#issuecomment-974433520,https://api.github.com/repos/simonw/datasette/issues/1521,974433520,IC_kwDOBm6k_c46FKzw,9599,simonw,2021-11-19T20:32:29Z,2021-11-19T20:32:29Z,OWNER,This configuration works great.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974435661,https://api.github.com/repos/simonw/datasette/issues/1522,974435661,IC_kwDOBm6k_c46FLVN,9599,simonw,2021-11-19T20:33:42Z,2021-11-19T20:33:42Z,OWNER,"Should just be a case of deploying this `Dockerfile`: ```Dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini # Append this to the end of the default httpd.conf file RUN echo $'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPass /foo/bar/ http://localhost:9000/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo $'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette fixtures.db --setting base_url ""/foo/bar/"" -p 9000 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""] ``` I can follow this TIL: https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974450232,https://api.github.com/repos/simonw/datasette/issues/1519,974450232,IC_kwDOBm6k_c46FO44,9599,simonw,2021-11-19T20:41:53Z,2021-11-19T20:42:19Z,OWNER,https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration says I should use `ProxyPreserveHost on`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974447950,https://api.github.com/repos/simonw/datasette/issues/1519,974447950,IC_kwDOBm6k_c46FOVO,9599,simonw,2021-11-19T20:40:19Z,2021-11-19T20:40:19Z,OWNER,"Figured it out! The test is not an accurate recreation of what is happening, because it doesn't simulate a request with a path of `/fixtures` that has been redirected by the proxy to `/prefix/fixtures`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974477465,https://api.github.com/repos/simonw/datasette/issues/1519,974477465,IC_kwDOBm6k_c46FViZ,9599,simonw,2021-11-19T21:15:30Z,2021-11-19T21:15:30Z,OWNER,"I think what's happening here is Apache is actually making a request to `/fixtures` rather than making a request to `/prefix/fixtures` - and Datasette is replying to requests on both the prefixed and the non-prefixed paths. This is pretty confusing! I think Datasette should ONLY reply to `/prefix/fixtures` instead and return a 404 for `/fixtures` - this would make things a whole lot easier to debug. But shipping that change could break existing deployments. Maybe that should be a breaking change for 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974478126,https://api.github.com/repos/simonw/datasette/issues/1519,974478126,IC_kwDOBm6k_c46FVsu,9599,simonw,2021-11-19T21:16:36Z,2021-11-19T21:16:36Z,OWNER,"In the meantime I can catch these errors by changing the test to run each path twice, once with and once without the prefix. This should accurately simulate how Apache is working here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1522#issuecomment-974506401,https://api.github.com/repos/simonw/datasette/issues/1522,974506401,IC_kwDOBm6k_c46Fcmh,9599,simonw,2021-11-19T22:11:51Z,2021-11-19T22:11:51Z,OWNER,"This is frustrating: I have the following Dockerfile: ```dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ bash RUN pip install datasette ENV TINI_VERSION v0.18.0 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini RUN chmod +x /tini # Append this to the end of the default httpd.conf file RUN echo $'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPass /prefix/ http://localhost:8001/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo $'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db RUN echo $'#!/usr/bin/env bash\n\ set -e\n\ \n\ httpd -D FOREGROUND &\n\ datasette fixtures.db --setting base_url ""/prefix/"" -h 0.0.0.0 -p 8001 &\n\ \n\ wait -n' > /app/start.sh RUN chmod +x /app/start.sh EXPOSE 80 ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""] ``` It works fine when I run it locally: ``` docker build -t datasette-apache-proxy-demo . docker run -p 5000:80 datasette-apache-proxy-demo ``` But when I deploy it to Cloud Run with the following script: ```bash #!/bin/bash # https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run NAME=""datasette-apache-proxy-demo"" PROJECT=$(gcloud config get-value project) IMAGE=""gcr.io/$PROJECT/$NAME"" gcloud builds submit --tag $IMAGE gcloud run deploy \ --allow-unauthenticated \ --platform=managed \ --image $IMAGE $NAME \ --port 80 ``` It serves the `/` page successfully, but hits to `/prefix/` return the following 503 error: > Service Unavailable > > The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. > > Apache/2.4.51 (Unix) Server at datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app Port 80 Cloud Run logs: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974521687,https://api.github.com/repos/simonw/datasette/issues/1522,974521687,IC_kwDOBm6k_c46FgVX,9599,simonw,2021-11-19T22:46:26Z,2021-11-19T22:46:26Z,OWNER,"Oh weird, it started working: https://datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app/prefix/fixtures/sortable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974523297,https://api.github.com/repos/simonw/datasette/issues/1522,974523297,IC_kwDOBm6k_c46Fguh,9599,simonw,2021-11-19T22:50:31Z,2021-11-19T22:50:31Z,OWNER,Demo code is now at: https://github.com/simonw/datasette/tree/main/demos/apache-proxy,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974523569,https://api.github.com/repos/simonw/datasette/issues/1522,974523569,IC_kwDOBm6k_c46Fgyx,9599,simonw,2021-11-19T22:51:10Z,2021-11-19T22:51:10Z,OWNER,I wan a GitHub Action which I can manually activate to deploy a new version of that demo... and I want it to bake in the latest release of Datasette so I can use it to demonstrate bug fixes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974541971,https://api.github.com/repos/simonw/datasette/issues/1522,974541971,IC_kwDOBm6k_c46FlST,9599,simonw,2021-11-19T23:40:32Z,2021-11-19T23:40:32Z,OWNER,"I want to be able to use build arguments to specify which commit version or branch of Datasette to deploy. This is proving hard to work out. I have this in my Dockerfile now: ``` ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip ``` Which works locally: docker build -t datasette-apache-proxy-demo . \ --build-arg DATASETTE_REF=c617e1769ea27e045b0f2907ef49a9a1244e577d But I can't figure out the right incantation to pass to `gcloud build submit`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974542348,https://api.github.com/repos/simonw/datasette/issues/1522,974542348,IC_kwDOBm6k_c46FlYM,9599,simonw,2021-11-19T23:41:47Z,2021-11-19T23:44:07Z,OWNER,Do I have to use `cloudbuild.yml` to specify these? https://stackoverflow.com/a/58327340/6083 and https://stackoverflow.com/a/66232670/6083 suggest I do.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974557766,https://api.github.com/repos/simonw/datasette/issues/1522,974557766,IC_kwDOBm6k_c46FpJG,9599,simonw,2021-11-20T00:35:25Z,2021-11-20T00:35:25Z,OWNER,Wrote a TIL about `--build-arg` and Cloud Run: https://til.simonwillison.net/cloudrun/using-build-args-with-cloud-run,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974558076,https://api.github.com/repos/simonw/datasette/issues/1522,974558076,IC_kwDOBm6k_c46FpN8,9599,simonw,2021-11-20T00:36:56Z,2021-11-20T00:36:56Z,OWNER,That 503 error is _really_ frustrating: I have a deploy running at https://apache-proxy-demo.datasette.io/prefix/ and after a fresh deploy it serves 503 errors for quite a while - then eventually starts working.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974558267,https://api.github.com/repos/simonw/datasette/issues/1519,974558267,IC_kwDOBm6k_c46FpQ7,9599,simonw,2021-11-20T00:37:57Z,2021-11-20T00:37:57Z,OWNER,Thanks to #1522 I have a live demo that exhibits this bug now: https://apache-proxy-demo.datasette.io/prefix/fixtures/attraction_characteristic,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974559176,https://api.github.com/repos/simonw/datasette/issues/1519,974559176,IC_kwDOBm6k_c46FpfI,9599,simonw,2021-11-20T00:42:08Z,2021-11-20T00:42:08Z,OWNER,"> In the meantime I can catch these errors by changing the test to run each path twice, once with and once without the prefix. This should accurately simulate how Apache is working here. This worked, I managed to get the tests to fail! Here's the change I made: ```diff diff --git a/tests/test_html.py b/tests/test_html.py index f24165b..dbdfe59 100644 --- a/tests/test_html.py +++ b/tests/test_html.py @@ -1614,12 +1614,19 @@ def test_metadata_sort_desc(app_client): ""/fixtures/compound_three_primary_keys/a,a,a"", ""/fixtures/paginated_view"", ""/fixtures/facetable"", + ""/fixtures?sql=select+1"", ], ) -def test_base_url_config(app_client_base_url_prefix, path): +@pytest.mark.parametrize(""use_prefix"", (True, False)) +def test_base_url_config(app_client_base_url_prefix, path, use_prefix): client = app_client_base_url_prefix - response = client.get(""/prefix/"" + path.lstrip(""/"")) + path_to_get = path + if use_prefix: + path_to_get = ""/prefix/"" + path.lstrip(""/"") + response = client.get(path_to_get) soup = Soup(response.body, ""html.parser"") + if path == ""/fixtures?sql=select+1"": + assert False for el in soup.findAll([""a"", ""link"", ""script""]): if ""href"" in el.attrs: href = el[""href""] @@ -1642,11 +1649,12 @@ def test_base_url_config(app_client_base_url_prefix, path): # If this has been made absolute it may start http://localhost/ if href.startswith(""http://localhost/""): href = href[len(""http://localost/"") :] - assert href.startswith(""/prefix/""), { + assert href.startswith(""/prefix/""), json.dumps({ ""path"": path, + ""path_to_get"": path_to_get, ""href_or_src"": href, ""element_parent"": str(el.parent), - } + }, indent=4, default=repr) def test_base_url_affects_metadata_extra_css_urls(app_client_base_url_prefix): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1522#issuecomment-974564712,https://api.github.com/repos/simonw/datasette/issues/1522,974564712,IC_kwDOBm6k_c46Fq1o,9599,simonw,2021-11-20T01:07:49Z,2021-11-20T01:10:48Z,OWNER,https://apache-proxy-demo.datasette.io/prefix/fixtures/compound_three_primary_keys has broken suggested facet links - they go to `https://localhost:8001/prefix/fixtures/compound_three_primary_keys?_facet=pk1#facet-pk1` - but I think that's because I'm missing the `ProxyPreserveHost On` setting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974561593,https://api.github.com/repos/simonw/datasette/issues/1519,974561593,IC_kwDOBm6k_c46FqE5,9599,simonw,2021-11-20T00:53:19Z,2021-11-20T00:53:19Z,OWNER,Adding that test found (I hope!) all of the remaining `base_url` bugs. There were a bunch! I think I finally get to close #838 too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1519#issuecomment-974562942,https://api.github.com/repos/simonw/datasette/issues/1519,974562942,IC_kwDOBm6k_c46FqZ-,9599,simonw,2021-11-20T00:59:32Z,2021-11-20T00:59:32Z,OWNER,"Ouch a nasty bug crept through there - https://datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app/prefix/fixtures/compound_three_primary_keys says > 500: name 'ds' is not defined","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1522#issuecomment-974565392,https://api.github.com/repos/simonw/datasette/issues/1522,974565392,IC_kwDOBm6k_c46FrAQ,9599,simonw,2021-11-20T01:11:20Z,2021-11-20T01:11:20Z,OWNER,"Yup, that fixed it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974565816,https://api.github.com/repos/simonw/datasette/issues/1522,974565816,IC_kwDOBm6k_c46FrG4,9599,simonw,2021-11-20T01:13:39Z,2021-11-20T01:13:39Z,OWNER,"I have a hunch that running `httpd -D FOREGROUND` doesn't show error logs, which would explain why I can't use the Cloud Run logs to figure out the reason for the 503s.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974572505,https://api.github.com/repos/simonw/datasette/issues/1522,974572505,IC_kwDOBm6k_c46FsvZ,9599,simonw,2021-11-20T01:50:48Z,2021-11-20T01:51:41Z,OWNER,"I figured out a recipe to run `httpd` as a service inside Alpine - works great on my laptop, here's my new `Dockerfile`: ```dockerfile FROM python:3-alpine # openrc gives us rc-service RUN apk add --no-cache \ openrc \ apache2 \ apache2-proxy \ bash ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip # Append this to the end of the default httpd.conf file RUN echo -e 'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPreserveHost On\n\ ProxyPass /prefix/ http://127.0.0.1:8001/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo 'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db EXPOSE 80 # RUN echo -e ""#!/bin/bash\nopenrc default\nrc-service apache2 start;\ndatasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001"" > /app/start.sh RUN echo ""#!/bin/bash"" >> start.sh RUN echo ""openrc default"" >> start.sh RUN echo ""rc-service apache2 start"" >> start.sh RUN echo ""datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001"" >> /app/start.sh RUN chmod +x /app/start.sh CMD /app/start.sh ``` I'm going to try this on Cloud Run and see if it fixes the 503s One annoying thing about this: Ctrl+C on my laptop no longer stops the container, I have to `docker ps` and then `docker kill xxx` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974577565,https://api.github.com/repos/simonw/datasette/issues/1522,974577565,IC_kwDOBm6k_c46Ft-d,9599,simonw,2021-11-20T02:23:07Z,2021-11-20T02:23:07Z,OWNER,"OK, that works on my laptop - and Ctrl+C quits it, which is nice: ``` apache-proxy % docker run -p 5000:80 --rm datasette-build-arg-demo 2021-11-20 02:22:13,925 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2021-11-20 02:22:13,927 INFO supervisord started with pid 1 2021-11-20 02:22:14,931 INFO spawned: 'datasette' with pid 7 2021-11-20 02:22:14,934 INFO spawned: 'httpd' with pid 8 2021-11-20 02:22:16,484 INFO success: datasette entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-11-20 02:22:16,484 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) ^C 2021-11-20 02:22:26,285 WARN received SIGINT indicating exit request 2021-11-20 02:22:26,286 INFO waiting for datasette, httpd to die 2021-11-20 02:22:26,315 INFO stopped: httpd (exit status 0) 2021-11-20 02:22:26,540 INFO stopped: datasette (exit status 0) ``` Here's my new Dockerfile: ```dockerfile FROM python:3-alpine RUN apk add --no-cache \ apache2 \ apache2-proxy \ supervisor \ bash ARG DATASETTE_REF RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip # Append this to the end of the default httpd.conf file RUN echo -e 'ServerName localhost\n\ \n\ \n\ Order deny,allow\n\ Allow from all\n\ \n\ \n\ ProxyPreserveHost On\n\ ProxyPass /prefix/ http://127.0.0.1:8001/\n\ Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf RUN echo 'Datasette' > /var/www/localhost/htdocs/index.html WORKDIR /app ADD https://latest.datasette.io/fixtures.db /app/fixtures.db EXPOSE 80 RUN echo ""[supervisord]"" >> /app/supervisord.conf RUN echo ""nodaemon=true"" >> /app/supervisord.conf RUN echo """" >> /app/supervisord.conf RUN echo ""[program:httpd]"" >> /app/supervisord.conf RUN echo ""command=httpd -D FOREGROUND"" >> /app/supervisord.conf RUN echo """" >> /app/supervisord.conf RUN echo ""[program:datasette]"" >> /app/supervisord.conf RUN echo ""command=datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001"" >> /app/supervisord.conf CMD [""/usr/bin/supervisord"", ""-c"", ""/app/supervisord.conf""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974577082,https://api.github.com/repos/simonw/datasette/issues/1522,974577082,IC_kwDOBm6k_c46Ft26,9599,simonw,2021-11-20T02:19:27Z,2021-11-20T02:19:27Z,OWNER,"https://docs.docker.com/config/containers/multi-service_container/ suggests `supervisord` as a last resort. https://stackoverflow.com/a/49100302/6083 has a neat looking recipe for than in Alpine: > **1.** `Dockerfile` is: > > FROM alpine:latest > RUN apk update && apk add --no-cache supervisor openssh nginx > COPY supervisord.conf /etc/supervisord.conf > CMD [""/usr/bin/supervisord"", ""-c"", ""/etc/supervisord.conf""] > > **2.** `supervisord.conf` is: > > [supervisord] > nodaemon=true > > [program:sshd] > command=/usr/sbin/sshd -D > > [program:nginx] > command=nginx -c /etc/nginx/nginx.conf ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974573616,https://api.github.com/repos/simonw/datasette/issues/1522,974573616,IC_kwDOBm6k_c46FtAw,9599,simonw,2021-11-20T01:58:44Z,2021-11-20T01:58:44Z,OWNER,"Deploy to Cloud Run appears to hang here: ``` Deploying container to Cloud Run service [datasette-apache-proxy-demo] in project [datasette-222320] region [us-central1] ⠧ Deploying... Revision deployment finished. Waiting for health check to begin. ⠧ Creating Revision... . Routing traffic... ✓ Setting IAM Policy... ``` **Waiting for health check to begin** makes it sound like the container didn't start properly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974575512,https://api.github.com/repos/simonw/datasette/issues/1522,974575512,IC_kwDOBm6k_c46FteY,9599,simonw,2021-11-20T02:09:20Z,2021-11-20T02:09:20Z,OWNER,"> **Waiting for health check to begin** makes it sound like the container didn't start properly. That eventually failed, but I did get these in the build logs: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974576436,https://api.github.com/repos/simonw/datasette/issues/1522,974576436,IC_kwDOBm6k_c46Fts0,9599,simonw,2021-11-20T02:14:45Z,2021-11-20T02:14:45Z,OWNER,I'm going to try running Apache with `httpd -D FOREGROUND` while running `datasette &`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974576624,https://api.github.com/repos/simonw/datasette/issues/1522,974576624,IC_kwDOBm6k_c46Ftvw,9599,simonw,2021-11-20T02:16:12Z,2021-11-20T02:16:12Z,OWNER,"Again, that approach worked on my laptop but when deployed to Cloud Run mostly gave me 503 errors for the `/prefix/` page, with the occasional 200. I did this: ```Dockerfile RUN echo ""#!/bin/bash"" >> start.sh # Start Datasette running in background with & RUN echo ""datasette /app/fixtures.db --setting base_url '/prefix/' --version-note '${DATASETTE_REF}' -h 0.0.0.0 -p 8001 &"" >> /app/start.sh RUN echo ""httpd -D FOREGROUND"" >> /app/start.sh RUN chmod +x /app/start.sh CMD /app/start.sh ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974577949,https://api.github.com/repos/simonw/datasette/issues/1522,974577949,IC_kwDOBm6k_c46FuEd,9599,simonw,2021-11-20T02:26:09Z,2021-11-20T02:26:17Z,OWNER,"So frustrating, that's giving me the same problem after being deployed! 503 errors for the first while, then it starts working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974578141,https://api.github.com/repos/simonw/datasette/issues/1522,974578141,IC_kwDOBm6k_c46FuHd,9599,simonw,2021-11-20T02:27:23Z,2021-11-20T02:27:23Z,OWNER,"Aha! This could be the clue I was looking for: https://www.reddit.com/r/googlecloud/comments/fmkx63/comment/fl5csty/?utm_source=reddit&utm_medium=web2x&context=3 > Are you processing on a background thread in your container? If so, it's likely your problem, because cloud run will put your app into a low power state between http requests. For long running tasks in cloud run, you need to keep the http connection open, and not return until you are done. Maybe the `datasette &` process is being affected by that in some way?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974585374,https://api.github.com/repos/simonw/datasette/issues/1522,974585374,IC_kwDOBm6k_c46Fv4e,9599,simonw,2021-11-20T03:28:58Z,2021-11-20T03:28:58Z,OWNER,Based on https://medium.com/google-cloud/init-process-for-containers-d03a471fa0cc I might try s6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974682507,https://api.github.com/repos/simonw/datasette/issues/1522,974682507,IC_kwDOBm6k_c46GHmL,9599,simonw,2021-11-20T17:24:13Z,2021-11-20T17:24:13Z,OWNER,"I'm going to leave this issue open, tag it as ""help wanted"" and cross my fingers that someone with Cloud Run deep expertise takes an interest in figuring out what's going wrong here!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974683220,https://api.github.com/repos/simonw/datasette/issues/1522,974683220,IC_kwDOBm6k_c46GHxU,9599,simonw,2021-11-20T17:29:12Z,2021-11-20T17:29:12Z,OWNER,"> As a a sanity check, would it be worth looking at trying to push the multi-process container on another provider of a knative / cloud run / tekton ? I have a somewhat similar use case for a future proejct, so i'm been very grateful to you sharing all the progress in this issue. That's a great idea. I'll try running on a non-Knative host too (probably Fly - though they actually run containers using Firecracker which ends up being completely different). Cloud Run are the only Knative host I've used, know of any others aside from Scaleway? They look like they're worth getting familiar with.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974685095,https://api.github.com/repos/simonw/datasette/issues/1522,974685095,IC_kwDOBm6k_c46GIOn,9599,simonw,2021-11-20T17:42:25Z,2021-11-20T17:42:25Z,OWNER,"I tried to deploy it to Fly - initially using `flyctl launch` but then switching to `flyctl deploy` so I could use the `--build-arg` option (posted [a feature request](https://community.fly.io/t/feature-request-flyctl-launch-build-arg/3209) here). Almost got it working, but it failed the health check: ``` % cd datasette/demos/apache-proxy apache-proxy % flyctl launch Creating app in /Users/simon/Dropbox/Development/datasette/demos/apache-proxy Scanning source code Detected Dockerfile app Automatically selected personal organization: Simon Willison ? Select region: sjc (Sunnyvale, California (US)) Created app floral-dust-4577 in organization personal Wrote config file fly.toml Your app is ready. Deploy with `flyctl deploy` ? Would you like to deploy now? Yes Deploying floral-dust-4577 ==> Validating app configuration --> Validating app configuration done Services TCP 80/443 ⇢ 8080 ==> Creating build context --> Creating build context done ==> Building image with Docker Sending build context to Docker daemon 8.704kB ... Error error building: executor failed running [/bin/sh -c pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip]: exit code: 1 # I didn't pass the build argument, trying again with flyctl deploy apache-proxy % flyctl deploy --build-arg DATASETTE_REF=main Update available 0.0.229 -> v0.0.255 Run ""flyctl version update"" to upgrade Deploying floral-dust-4577 ==> Validating app configuration --> Validating app configuration done Services TCP 80/443 ⇢ 8080 ==> Creating build context --> Creating build context done ==> Building image with Docker Sending build context to Docker daemon 8.704kB [+] Building 15.7s (27/27) ... 0.0s ==> Pushing image to fly The push refers to repository [registry.fly.io/floral-dust-4577] 9bf88c92aa2a: Pushed 3d61728b8391: Pushed ... --> Pushing image done Image: registry.fly.io/floral-dust-4577:deployment-1637429501 Image size: 276 MB ==> Creating release Release v2 created You can detach the terminal anytime without stopping the deployment Monitoring Deployment 1 desired, 1 placed, 0 healthy, 0 unhealthy [health checks: 1 total, 1 critical] 1 desired, 1 placed, 0 healthy, 1 unhealthy [health checks: 1 total, 1 critical] v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to Failed Instances ==> Failure #1 Instance ID = 36adac86 Version = 0 Region = sjc Desired = run Status = running Health Checks = 1 total, 1 critical Restarts = 0 Created = 4m52s ago Recent Events TIMESTAMP TYPE MESSAGE 2021-11-20T17:32:52Z Received Task received by client 2021-11-20T17:32:52Z Task Setup Building Task Directory 2021-11-20T17:33:02Z Started Task started by client Recent Logs 2021-11-20T17:32:56Z [info] Unpacking image 2021-11-20T17:33:01Z [info] Preparing kernel init 2021-11-20T17:33:01Z [info] Configuring firecracker 2021-11-20T17:33:02Z [info] Starting virtual machine 2021-11-20T17:33:02Z [info] Starting init (commit: 7943db6)... 2021-11-20T17:33:02Z [info] Preparing to run: `/usr/bin/supervisord -c /app/supervisord.conf` as root 2021-11-20T17:33:02Z [info] 2021/11/20 17:33:02 listening on [fdaa:0:4ef:a7b:2295:36ad:ac86:2]:22 (DNS: [fdaa::3]:53) 2021-11-20T17:33:02Z [info] 2021-11-20 17:33:02,374 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2021-11-20T17:33:02Z [info] 2021-11-20 17:33:02,376 INFO supervisord started with pid 510 2021-11-20T17:33:03Z [info] 2021-11-20 17:33:03,379 INFO spawned: 'apache2' with pid 515 2021-11-20T17:33:03Z [info] 2021-11-20 17:33:03,381 INFO spawned: 'datasette' with pid 516 2021-11-20T17:33:05Z [info] 2021-11-20 17:33:05,068 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-11-20T17:33:05Z [info] 2021-11-20 17:33:05,068 INFO success: datasette entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-11-20T17:33:28Z [error] Health check status changed 'warning' => 'critical' ***v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974692546,https://api.github.com/repos/simonw/datasette/issues/1522,974692546,IC_kwDOBm6k_c46GKDC,9599,simonw,2021-11-20T18:34:13Z,2021-11-20T18:34:13Z,OWNER,"Here's why it failed - the `fly.toml` file that was generated when I ran `flyctl launch` had this section: ```toml [[services]] http_checks = [] internal_port = 8080 processes = [""app""] protocol = ""tcp"" script_checks = [] ``` But I need `internal_port` to be 80 for Apache, so I changed that and ran `flyctl deploy --build-arg DATASETTE_REF=main` again - and it worked! https://floral-dust-4577.fly.dev/prefix/ - not seeing any 503 errors there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974693350,https://api.github.com/repos/simonw/datasette/issues/1522,974693350,IC_kwDOBm6k_c46GKPm,9599,simonw,2021-11-20T18:39:27Z,2021-11-20T18:39:27Z,OWNER,"I'm going to go with Fly instead for this, especially as I can keep it within their free tier (and iI want to get more familiar with their platform).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-974695111,https://api.github.com/repos/simonw/datasette/issues/1522,974695111,IC_kwDOBm6k_c46GKrH,9599,simonw,2021-11-20T18:52:11Z,2021-11-20T18:52:11Z,OWNER,The demo is now live on https://datasette-apache-proxy-demo.fly.dev/prefix/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1519#issuecomment-974697824,https://api.github.com/repos/simonw/datasette/issues/1519,974697824,IC_kwDOBm6k_c46GLVg,9599,simonw,2021-11-20T19:11:21Z,2021-11-20T19:11:21Z,OWNER,"OK, i think I got all of them this time! The latest demo is now live at https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures/sortable?_facet=pk2 I'm closing this issue, but feel free to re-open it if you spot any that I missed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1524#issuecomment-974704254,https://api.github.com/repos/simonw/datasette/issues/1524,974704254,IC_kwDOBm6k_c46GM5-,9599,simonw,2021-11-20T20:03:51Z,2021-11-20T20:22:52Z,OWNER,"I'm also going to extract the Apache config files from https://github.com/simonw/datasette/blob/250db8192cb8aba5eb8cd301ccc2a49525bc3d24/demos/apache-proxy/Dockerfile into a separate file to make it easier to read. (The supervisor config needs to be dynamically constructed to include $DATASETTE_REF so I will leave it where it is.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059219106,"Improve Apache proxy documentation, link to demo", https://github.com/simonw/datasette/issues/1519#issuecomment-974701788,https://api.github.com/repos/simonw/datasette/issues/1519,974701788,IC_kwDOBm6k_c46GMTc,9599,simonw,2021-11-20T19:42:29Z,2021-11-20T19:42:29Z,OWNER,"> I think what's happening here is Apache is actually making a request to `/fixtures` rather than making a request to `/prefix/fixtures` - and Datasette is replying to requests on both the prefixed and the non-prefixed paths. > > This is pretty confusing! I think Datasette should ONLY reply to `/prefix/fixtures` instead and return a 404 for `/fixtures` - this would make things a whole lot easier to debug. > > But shipping that change could break existing deployments. Maybe that should be a breaking change for 1.0. On further thought I'm not going to do this. Having Datasette work behind a proxy the way it does right now is clearly easy for people to deploy (now that I've fixed the bugs) and I trust my improved tests to catch problems in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1524#issuecomment-974707878,https://api.github.com/repos/simonw/datasette/issues/1524,974707878,IC_kwDOBm6k_c46GNym,9599,simonw,2021-11-20T20:34:51Z,2021-11-20T20:38:29Z,OWNER,"I pointed `CNAME` of `datasette-apache-proxy-demo.datasette.io` at `datasette-apache-proxy-demo.fly.dev.` using Vercel DNS: Then I asked Fly to issue a LetsEncrypt certificate for that: ``` % flyctl certs create datasette-apache-proxy-demo.datasette.io # About 53 seconds later: % flyctl certs show datasette-apache-proxy-demo.datasette.io The certificate for datasette-apache-proxy-demo.datasette.io has been issued. Hostname = datasette-apache-proxy-demo.datasette.io DNS Provider = constellix Certificate Authority = Let's Encrypt Issued = ecdsa,rsa Added to App = 53 seconds ago Source = fly ``` https://datasette-apache-proxy-demo.datasette.io/ works now - I'll use that in the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059219106,"Improve Apache proxy documentation, link to demo", https://github.com/simonw/datasette/issues/1426#issuecomment-974711959,https://api.github.com/repos/simonw/datasette/issues/1426,974711959,IC_kwDOBm6k_c46GOyX,52649,tannewt,2021-11-20T21:11:51Z,2021-11-20T21:11:51Z,NONE,I think another thing would be to make `/pages/robots.txt` work. That way you can use jinja to generate a desired robots.txt. I'm using it to allow the main index and what it links to to be crawled (but not the database pages directly.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1524#issuecomment-974721652,https://api.github.com/repos/simonw/datasette/issues/1524,974721652,IC_kwDOBm6k_c46GRJ0,9599,simonw,2021-11-20T22:41:03Z,2021-11-20T22:41:03Z,OWNER,New TIL: https://til.simonwillison.net/fly/custom-subdomain-fly,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059219106,"Improve Apache proxy documentation, link to demo", https://github.com/simonw/datasette/issues/1524#issuecomment-974725814,https://api.github.com/repos/simonw/datasette/issues/1524,974725814,IC_kwDOBm6k_c46GSK2,9599,simonw,2021-11-20T23:24:01Z,2021-11-20T23:24:01Z,OWNER,"I noticed that `http://datasette-apache-proxy-demo.datasette.io/` wasn't redirecting to `https` so I built a new plugin: https://github.com/simonw/datasette-redirect-to-https ``` % curl -i 'http://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key' HTTP/1.1 301 Moved Permanently date: Sat, 20 Nov 2021 23:22:50 GMT server: Fly/51d150d (2021-11-19) location: https://datasette-apache-proxy-demo.datasette.io/fixtures/no_primary_key x-proxied-by: Apache2 Debian transfer-encoding: chunked via: 1.1 fly.io fly-request-id: 01FMZTHTHVPC8BZY0625D7JV4B ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059219106,"Improve Apache proxy documentation, link to demo", https://github.com/simonw/datasette/issues/1525#issuecomment-974912985,https://api.github.com/repos/simonw/datasette/issues/1525,974912985,IC_kwDOBm6k_c46G_3Z,9599,simonw,2021-11-21T22:55:42Z,2021-11-21T22:55:42Z,OWNER,Here's the template: https://github.com/simonw/datasette/blob/48f11998b73350057b74fe6ab464d4ac3071637c/datasette/templates/row.html#L39-L41,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059509927,"""Links from other tables"" broken for columns starting with underscore", https://github.com/simonw/datasette/issues/1525#issuecomment-974913180,https://api.github.com/repos/simonw/datasette/issues/1525,974913180,IC_kwDOBm6k_c46G_6c,9599,simonw,2021-11-21T22:57:08Z,2021-11-21T22:57:08Z,OWNER,https://latest.datasette.io/fixtures/facetable can't quite demonstrate the bug because `_neighborhood` isn't a foreign key - I should rename `city_id` to `_city_id`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059509927,"""Links from other tables"" broken for columns starting with underscore", https://github.com/simonw/datasette/issues/93#issuecomment-974765825,https://api.github.com/repos/simonw/datasette/issues/93,974765825,IC_kwDOBm6k_c46Gb8B,9599,simonw,2021-11-21T07:00:21Z,2021-11-21T07:00:21Z,OWNER,Closing this in favour of Datasette Desktop: https://datasette.io/desktop,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary, https://github.com/simonw/datasette/issues/1527#issuecomment-974979785,https://api.github.com/repos/simonw/datasette/issues/1527,974979785,IC_kwDOBm6k_c46HQLJ,9599,simonw,2021-11-22T01:02:57Z,2021-11-22T01:03:19Z,OWNER,"I think the root cause is this hidden form field on https://latest.datasette.io/fixtures/facetable?_facet=_neighborhood&_neighborhood__exact=Downtown ```html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1526#issuecomment-975073308,https://api.github.com/repos/simonw/datasette/issues/1526,975073308,IC_kwDOBm6k_c46HnAc,9599,simonw,2021-11-22T04:13:46Z,2021-11-22T04:13:46Z,OWNER,"Addressing that over here (hadn't seen that issue yet, thanks for the prod): https://github.com/simonw/datasette-publish-vercel/issues/51#issuecomment-975073026","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059549523,"Add to vercel.json, rather than overwriting it.", https://github.com/simonw/datasette/issues/1526#issuecomment-975110692,https://api.github.com/repos/simonw/datasette/issues/1526,975110692,IC_kwDOBm6k_c46HwIk,9599,simonw,2021-11-22T04:49:44Z,2021-11-22T04:49:44Z,OWNER,Fixed in the 0.12 release of that plugin: https://github.com/simonw/datasette-publish-vercel/releases/tag/0.12,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059549523,"Add to vercel.json, rather than overwriting it.", https://github.com/simonw/datasette/issues/1528#issuecomment-975955589,https://api.github.com/repos/simonw/datasette/issues/1528,975955589,IC_kwDOBm6k_c46K-aF,15178711,asg017,2021-11-22T22:00:30Z,2021-11-22T22:00:30Z,CONTRIBUTOR,"Oh, another thing to consider: I believe this would be the first `""_file""` key in datasette's metadata, compared to other `""_url""` keys like `""license_url""` or `""about_url""`. Not too sure what considerations to include with this (ex should missing files cause Datasette to stop before starting, should build scripts bundle these sql files somewhere during `datasette package`, etc.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1060631257,"Add new `""sql_file""` key to Canned Queries in metadata?", https://github.com/simonw/datasette/issues/1522#issuecomment-976023405,https://api.github.com/repos/simonw/datasette/issues/1522,976023405,IC_kwDOBm6k_c46LO9t,360895,steren,2021-11-23T00:08:07Z,2021-11-23T00:08:07Z,NONE,"If you suspect that Cloud Run throttled CPU could be the cause, you can request to have CPU always allocated with `gcloud beta run deploy --no-cpu-throttling` ([read more](https://cloud.google.com/blog/products/serverless/cloud-run-gets-always-on-cpu-allocation)) It could also be the Cloud Run sandbox that somehow gets in the way here, in which case I recommend testing with the second generation execution environment: `gcloud beta run deploy --execution-environment gen2`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/issues/1522#issuecomment-976117989,https://api.github.com/repos/simonw/datasette/issues/1522,976117989,IC_kwDOBm6k_c46LmDl,813732,glasnt,2021-11-23T03:00:34Z,2021-11-23T03:00:34Z,CONTRIBUTOR,"I tried deploying the most recent version of the Dockerfile in this thread ([link to comment](https://github.com/simonw/datasette/issues/1522#issuecomment-974605128)), and after trying a few different different combinations, I was only successful when I used `--no-cpu-throttling` (""CPU Is always allocated"" in the UI) Using this method, I got a very similar issue to you: The first time I'd load the site I'd get a 503. But after that first load, I didn't get the issue again. It would re-occur if the service started from cold boot. I suspect this is a race condition in the supervisord configuration. The errors I got were the same `Connection refused: AH00957: http: attempt to connect to 127.0.0.1:8001 (127.0.0.1) failed`, and that seems to indicate that `datasette` hadn't yet started. Looking at the order of logs getting back, the processes reported successfully completing loading after the first 503 was returned, so that makes me think race condition. I can replicate this locally, if I `docker run` and request `localhost:5000/prefix` _before_ I get the `datasette entered RUNNING state` message. Cloud Run wakes up when requests are received, so this test would semi-replicate that, but local docker would be the equivalent of a persistent process, hence it doesn't normally exhibit the same issues. Unfortunately supervisor/supervisor issue 122 (not linking as to prevent cross-project link spam) seems to say that dependency chaining is a feature that's been asked for for a long time, but hasn't been implemented. You could try some suggestions in that thread. ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy, https://github.com/simonw/datasette/pull/1529#issuecomment-977870699,https://api.github.com/repos/simonw/datasette/issues/1529,977870699,IC_kwDOBm6k_c46SR9r,22429695,codecov[bot],2021-11-24T13:18:52Z,2021-11-30T02:30:46Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1529](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2cb3dbb) into [main](https://codecov.io/gh/simonw/datasette/commit/48f11998b73350057b74fe6ab464d4ac3071637c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (48f1199) will **decrease** coverage by `0.06%`. > The diff coverage is `n/a`. > :exclamation: Current head 2cb3dbb differs from pull request most recent head 495b726. Consider uploading reports for the commit 495b726 to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1529/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1529 +/- ## ========================================== - Coverage 91.90% 91.83% -0.07% ========================================== Files 34 34 Lines 4434 4434 ========================================== - Hits 4075 4072 -3 - Misses 359 362 +3 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.42% <0.00%> (-1.79%)` | :arrow_down: | | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <0.00%> (-0.75%)` | :arrow_down: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1beb7d9...495b726](https://codecov.io/gh/simonw/datasette/pull/1529?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1062414013,"Update janus requirement from <0.7,>=0.6.2 to >=0.6.2,<0.8", https://github.com/simonw/datasette/issues/1518#issuecomment-981172385,https://api.github.com/repos/simonw/datasette/issues/1518,981172385,IC_kwDOBm6k_c46e4Ch,9599,simonw,2021-11-28T23:21:26Z,2021-11-28T23:21:26Z,OWNER,"Aside: is there any reason this work can't complete the long-running goal of merging the TableView and QueryView, such that most of the features available for tables become available for arbitrary queries too? I had already mentally committed to implementing facets for queries, but I just realized that filters could work too - using either a CTE or a nested query. Pagination is the one holdout here, since table pagination uses keyset pagination over a known order. But maybe arbitrary queries can only be paginated off you order them first?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-981172801,https://api.github.com/repos/simonw/datasette/issues/1518,981172801,IC_kwDOBm6k_c46e4JB,9599,simonw,2021-11-28T23:23:51Z,2021-11-28T23:23:51Z,OWNER,"(I could experiment with merging the two tables by adding a temporary undocumented `?_sql=` parameter to the in-progress table view that sets an alternative query instead of `select cols from table` - added bonus, this will force me to use introspection against the returned columns rather than mixing in the known columns for the specified table)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1533#issuecomment-981149039,https://api.github.com/repos/simonw/datasette/issues/1533,981149039,IC_kwDOBm6k_c46eyVv,9599,simonw,2021-11-28T20:45:36Z,2021-11-28T20:45:36Z,OWNER,I built an initial prototype of this in a branch: https://github.com/simonw/datasette/commit/e0a84691c2959f2d1d76948574c9c4a910c7556c - which exposed even more flaws in the way `TableView` is structured (adding custom HTTP headers to the response is way harder than it should be) which I should address in the refactor in #617.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065431383,"Add `Link: rel=""alternate""` header pointing to JSON for a table/query", https://github.com/simonw/datasette/issues/1534#issuecomment-981149531,https://api.github.com/repos/simonw/datasette/issues/1534,981149531,IC_kwDOBm6k_c46eydb,9599,simonw,2021-11-28T20:48:54Z,2021-11-28T20:48:54Z,OWNER,"If I'm going to do this, is there value in also spotting `Accept: text/csv` and returning CSV for that? I'm pretty sure no client has EVER implemented this though, so it feels like it would be showboating.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/issues/1518#issuecomment-981153060,https://api.github.com/repos/simonw/datasette/issues/1518,981153060,IC_kwDOBm6k_c46ezUk,9599,simonw,2021-11-28T21:13:09Z,2021-12-17T23:37:08Z,OWNER,"Two new requirements inspired by work on the `datasette-table` (and `datasette-notebook`) projects: - #1533 - #1534","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-981153186,https://api.github.com/repos/simonw/datasette/issues/1518,981153186,IC_kwDOBm6k_c46ezWi,9599,simonw,2021-11-28T21:13:50Z,2021-11-28T21:13:50Z,OWNER,"I'm also going to use the new `datasette-table` Web Component to help guide the design of the new API, which relates directly to this issue too: - #1532","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/pull/1537#issuecomment-981631026,https://api.github.com/repos/simonw/datasette/issues/1537,981631026,IC_kwDOBm6k_c46goAy,22429695,codecov[bot],2021-11-29T13:23:20Z,2021-11-29T13:23:20Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1537](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fcfaec1) into [main](https://codecov.io/gh/simonw/datasette/commit/48f11998b73350057b74fe6ab464d4ac3071637c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (48f1199) will **decrease** coverage by `0.06%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1537/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1537 +/- ## ========================================== - Coverage 91.90% 91.83% -0.07% ========================================== Files 34 34 Lines 4434 4434 ========================================== - Hits 4075 4072 -3 - Misses 359 362 +3 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1537/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.42% <0.00%> (-1.79%)` | :arrow_down: | | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1537/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <0.00%> (-0.75%)` | :arrow_down: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [48f1199...fcfaec1](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066023866,"Update aiofiles requirement from <0.8,>=0.4 to >=0.4,<0.9", https://github.com/simonw/datasette/issues/1532#issuecomment-981966693,https://api.github.com/repos/simonw/datasette/issues/1532,981966693,IC_kwDOBm6k_c46h59l,30934,20after4,2021-11-29T19:56:52Z,2021-11-29T19:56:52Z,NONE,FWIW I've written some web components that consume the json api and I think it's a really nice way to work with datasette. I like the combination with datasette+sqlite as a back-end feeding data to a front-end that's entirely javascript + html.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936,Use datasette-table Web Component to guide the design of the JSON API for 1.0, https://github.com/simonw/datasette/issues/1304#issuecomment-981980048,https://api.github.com/repos/simonw/datasette/issues/1304,981980048,IC_kwDOBm6k_c46h9OQ,30934,20after4,2021-11-29T20:13:53Z,2021-11-29T20:14:11Z,NONE,There isn't any way to do this with sqlite as far as I know. The only option is to insert the right number of ? placeholders into the sql template and then provide an array of values.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",863884805,"Document how to send multiple values for ""Named parameters"" ", https://github.com/simonw/datasette/issues/1538#issuecomment-981849494,https://api.github.com/repos/simonw/datasette/issues/1538,981849494,IC_kwDOBm6k_c46hdWW,9599,simonw,2021-11-29T17:23:52Z,2021-11-29T17:23:52Z,OWNER,"Just trying to use `git_history_file` produces this error: > `TypeError: Attempted to convert a callback into a command twice.` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066288689,Research pattern for re-registering existing Click tools with register_commands, https://github.com/simonw/datasette/issues/1538#issuecomment-981852280,https://api.github.com/repos/simonw/datasette/issues/1538,981852280,IC_kwDOBm6k_c46heB4,9599,simonw,2021-11-29T17:27:12Z,2021-11-29T17:27:12Z,OWNER,"Thanks to https://stackoverflow.com/a/45514541/6083 I found the right pattern: ```python from datasette import hookimpl from git_history.cli import cli as git_history_cli @hookimpl def register_commands(cli): cli.add_command(git_history_cli, name=""git-history"") ``` I think this is a little bit too obscure to add to the Datasette documentation - I'll turn it into a TIL instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066288689,Research pattern for re-registering existing Click tools with register_commands, https://github.com/simonw/datasette/issues/1538#issuecomment-981856895,https://api.github.com/repos/simonw/datasette/issues/1538,981856895,IC_kwDOBm6k_c46hfJ_,9599,simonw,2021-11-29T17:32:44Z,2021-11-29T17:32:44Z,OWNER,TIL: https://til.simonwillison.net/datasette/reuse-click-for-register-commands,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066288689,Research pattern for re-registering existing Click tools with register_commands, https://github.com/simonw/datasette/issues/1527#issuecomment-982235541,https://api.github.com/repos/simonw/datasette/issues/1527,982235541,IC_kwDOBm6k_c46i7mV,9599,simonw,2021-11-30T02:57:34Z,2021-11-30T02:58:44Z,OWNER,"I started fiddling with a test for this which extracts the `` fields, but I probably won't use it: ```python def test_exact_parameter_results_in_correct_hidden_fields(app_client): # https://github.com/simonw/datasette/issues/1527 response = app_client.get( ""/fixtures/facetable?_facet=_neighborhood&_neighborhood__exact=Downtown"" ) # In this case we should NOT have a hidden _neighborhood__exact=Downtown field form = Soup(response.body, ""html.parser"").find(""form"") selects = [ { ""name"": select[""name""], ""value"": select.select(""option[selected]"")[0].text if select.select(""option[selected]"") else """", } for select in form.findAll(""select"") ] inputs = [input.attrs for input in form.findAll(""input"")] # Turn those both into a {name: (value, type)} array form_inputs = {} form_inputs.update( {select[""name""]: (select[""value""], ""select"") for select in selects} ) form_inputs.update( { input[""name""]: (input.get(""value""), input[""type""]) for input in inputs if input.get(""name"") } ) assert form_inputs == { ""_filter_column_1"": (""_neighborhood"", ""select""), ""_filter_op_1"": (""="", ""select""), ""_filter_value_1"": (""Downtown"", ""text""), ""_filter_column"": ("""", ""select""), ""_filter_op"": ("""", ""select""), ""_filter_value"": (None, ""text""), ""_sort"": (""Sort by pk"", ""select""), ""_sort_by_desc"": (None, ""checkbox""), ""_facet"": (""_neighborhood"", ""hidden""), ""_neighborhood__exact"": (""Downtown"", ""hidden""), } ``` The problem is that last hidden field, `_neighborhood__exact=Downtown` - which should not be there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1527#issuecomment-982318745,https://api.github.com/repos/simonw/datasette/issues/1527,982318745,IC_kwDOBm6k_c46jP6Z,9599,simonw,2021-11-30T06:11:21Z,2021-11-30T06:11:21Z,OWNER,"Manually tested this too, looks like that fixed it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1532#issuecomment-982319210,https://api.github.com/repos/simonw/datasette/issues/1532,982319210,IC_kwDOBm6k_c46jQBq,9599,simonw,2021-11-30T06:12:19Z,2021-11-30T06:12:19Z,OWNER,That's really cool to hear - I've not seen many people actively building on top of the JSON API.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936,Use datasette-table Web Component to guide the design of the JSON API for 1.0, https://github.com/simonw/datasette/issues/1525#issuecomment-982331602,https://api.github.com/repos/simonw/datasette/issues/1525,982331602,IC_kwDOBm6k_c46jTDS,9599,simonw,2021-11-30T06:39:00Z,2021-11-30T06:39:00Z,OWNER,"These two pages now help demonstrate the fix: - https://latest.datasette.io/fixtures/facet_cities/1 - https://latest.datasette.io/fixtures/attraction_characteristic/2 I added a new test for these here: https://github.com/simonw/datasette/blob/35b12746ba2bf9f254791bddac03d25b19be9b77/tests/test_html.py#L823-L848 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059509927,"""Links from other tables"" broken for columns starting with underscore", https://github.com/simonw/datasette/issues/1532#issuecomment-982745406,https://api.github.com/repos/simonw/datasette/issues/1532,982745406,IC_kwDOBm6k_c46k4E-,30934,20after4,2021-11-30T15:28:57Z,2021-11-30T15:28:57Z,NONE,"It's a really great API and the documentation is really great too. Honestly, in more than 20 years of professional experience, I haven't worked with any software API that was more of a joy to use. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936,Use datasette-table Web Component to guide the design of the JSON API for 1.0, https://github.com/simonw/datasette/issues/1540#issuecomment-984048965,https://api.github.com/repos/simonw/datasette/issues/1540,984048965,IC_kwDOBm6k_c46p2VF,9599,simonw,2021-12-01T20:59:26Z,2021-12-01T21:02:58Z,OWNER,"This is a bit of a mess but it does keep the hovercard around for a moment and then fade it away when you mouse out of it: ```html+jinja {% extends ""base.html"" %} {% block content %}

    Hovercards demo

    Here is a link to a row {% endblock %} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1540#issuecomment-984051925,https://api.github.com/repos/simonw/datasette/issues/1540,984051925,IC_kwDOBm6k_c46p3DV,9599,simonw,2021-12-01T21:03:16Z,2021-12-01T21:03:16Z,OWNER,Needs `pageX` not `clientX` because otherwise it doesn't work when you scroll down the page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1540#issuecomment-984053760,https://api.github.com/repos/simonw/datasette/issues/1540,984053760,IC_kwDOBm6k_c46p3gA,9599,simonw,2021-12-01T21:05:20Z,2021-12-01T21:05:20Z,OWNER,"I realized you couldn't click the links any more because the hovercard overlapped them, so I changed it to this instead. Need to reconsider the when-to-hide logic though. ```javascript hovercard.style.top = (ev.pageY + 15) + 'px'; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1519#issuecomment-983890815,https://api.github.com/repos/simonw/datasette/issues/1519,983890815,IC_kwDOBm6k_c46pPt_,157158,phubbard,2021-12-01T17:50:09Z,2021-12-01T17:50:09Z,NONE,"thanks so very much for the prompt attention and fix! Plus, the animated GIF showing the bug is just extra and I love it. Interactions like this are why I love open source.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views, https://github.com/simonw/datasette/issues/1540#issuecomment-983985330,https://api.github.com/repos/simonw/datasette/issues/1540,983985330,IC_kwDOBm6k_c46pmyy,9599,simonw,2021-12-01T19:29:05Z,2021-12-01T19:29:05Z,OWNER,"The layout of the hover card could be similar to the one used by `datasette-cluster-map`: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1540#issuecomment-984037711,https://api.github.com/repos/simonw/datasette/issues/1540,984037711,IC_kwDOBm6k_c46pzlP,9599,simonw,2021-12-01T20:42:17Z,2021-12-01T20:43:14Z,OWNER,"A first prototype (saved as `templates/pages/hovercard.html` and run with `datasette fixtures.db --template-dir=templates`): ```html+jinja {% extends ""base.html"" %} {% block content %}

    Hovercards demo

    Here is a link to a row {% endblock %} ``` ![hovercard](https://user-images.githubusercontent.com/9599/144310888-6db71bad-b6f6-4d8a-a737-81a618022bbe.gif) Lots of decisions to make here. Most importantly, when should it be hidden again?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1175#issuecomment-984569477,https://api.github.com/repos/simonw/datasette/issues/1175,984569477,IC_kwDOBm6k_c46r1aF,24821294,AnkitKundariya,2021-12-02T12:09:30Z,2021-12-02T12:09:30Z,NONE,"@hannseman I have tried the above suggestion given by you but somehow I'm getting the below error. _note : I'm running my application with Docker._ `app_1 | {""event"": ""Exception in ASGI application\n"", ""exc_info"": ["""", ""RuntimeError('no running event loop')"", """"], ""logger"": ""uvicorn.error"", ""level"": ""error"", ""timestamp"": ""2021-12-02T12:06:36.011448Z""} `","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779156520,Use structlog for logging, https://github.com/simonw/datasette/issues/1540#issuecomment-984801331,https://api.github.com/repos/simonw/datasette/issues/1540,984801331,IC_kwDOBm6k_c46suAz,9599,simonw,2021-12-02T16:42:02Z,2021-12-09T23:38:39Z,OWNER,"I'm going to wrap this up in a plugin for the moment - I want it in Datasette core but I'd like to improve the implementation first with things like support for `base_url` which will likely depend on #1533 or similar. Here's the plugin: https://github.com/simonw/datasette-hovercards","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1068791148,Idea: hover to reveal details of linked row, https://github.com/simonw/datasette/issues/1541#issuecomment-984908185,https://api.github.com/repos/simonw/datasette/issues/1541,984908185,IC_kwDOBm6k_c46tIGZ,9599,simonw,2021-12-02T18:56:54Z,2021-12-02T18:56:54Z,OWNER,Also it should link to foreign keys like the table page does.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1069881276,Different default layout for row page, https://github.com/simonw/datasette/issues/1426#issuecomment-985982668,https://api.github.com/repos/simonw/datasette/issues/1426,985982668,IC_kwDOBm6k_c46xObM,95520595,knowledgecamp12,2021-12-04T07:11:29Z,2021-12-04T07:11:29Z,NONE,You can generate xml site map from the online tools using https://tools4seo.site/xml-sitemap-generator. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default", https://github.com/simonw/datasette/issues/1585#issuecomment-1003575286,https://api.github.com/repos/simonw/datasette/issues/1585,1003575286,IC_kwDOBm6k_c470Vf2,9599,simonw,2022-01-01T15:40:38Z,2022-01-01T15:40:38Z,OWNER,API tutorial: https://firebase.google.com/docs/hosting/api-deploy,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1091838742,Fire base caching for `publish cloudrun`, https://github.com/simonw/datasette/issues/1534#issuecomment-1005975080,https://api.github.com/repos/simonw/datasette/issues/1534,1005975080,IC_kwDOBm6k_c479fYo,9599,simonw,2022-01-05T18:29:06Z,2022-01-05T18:29:06Z,OWNER,"A really big downside to this is that it turns out many CDNs - apparently including Cloudflare - don't support the Vary header at all! More in this thread: https://twitter.com/simonw/status/1478470282931163137","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/pull/1548#issuecomment-990967417,https://api.github.com/repos/simonw/datasette/issues/1548,990967417,IC_kwDOBm6k_c47EPZ5,22429695,codecov[bot],2021-12-10T13:19:00Z,2021-12-10T13:19:00Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1548](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (68383f5) into [main](https://codecov.io/gh/simonw/datasette/commit/737115ea14cd51ffb55dea886e6a684c148db2c9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (737115e) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1548/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1548 +/- ## ======================================= Coverage 91.84% 91.84% ======================================= Files 34 34 Lines 4438 4438 ======================================= Hits 4076 4076 Misses 362 362 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [737115e...68383f5](https://codecov.io/gh/simonw/datasette/pull/1548?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076834768,"Update pytest-xdist requirement from <2.5,>=2.2.1 to >=2.2.1,<2.6", https://github.com/simonw/datasette/issues/1518#issuecomment-991285527,https://api.github.com/repos/simonw/datasette/issues/1518,991285527,IC_kwDOBm6k_c47FdEX,9599,simonw,2021-12-10T20:52:00Z,2021-12-10T20:52:00Z,OWNER,"If I break this up into `@inject` methods, what methods could I have and what would they do? - `resolve_path`: Use request path to resolve the database and table. Could handle hash URLs too (if I don't manage to extract those to a plugin) - would be nice if this could raise a redirect, but I think that will instead have to be one of the things it returns - `build_sql`: Builds the SQL query based on the querystring (and some DB introspection) - `execute_count`: Execute the `count(*)` - `execute_rows`: Execute the `limit 101` to fetch the rows - `execute_facets`: Execute all requested facets (could this do its own `asyncio.gather()` to run facets in parallel?) - `suggest_facets`: Execute facet suggestions Are there any plugin hooks that would make sense to execute in parallel? Actually there might be: I don't think `extra_template_vars`, `extra_css_urls`, `extra_js_urls`, `extra_body_script` depend on each other so it might be possible to execute them in a parallel chunk (at least any of them that return awaitables).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1549#issuecomment-991752486,https://api.github.com/repos/simonw/datasette/issues/1549,991752486,IC_kwDOBm6k_c47HPEm,9599,simonw,2021-12-11T19:09:15Z,2021-12-11T19:09:15Z,OWNER,"That's what this option does: ![EAB1B9E8-38E9-4C6D-8854-BD1935F163D9](https://user-images.githubusercontent.com/9599/145688531-668bafa1-e287-4bbd-84d6-157241fb1f68.jpeg) The usability of this is pretty terrible though (including ""stream all rows"" - how are people meant to understand what that does?) so it can definitely do with some rethinking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability, https://github.com/simonw/datasette/issues/1549#issuecomment-991754237,https://api.github.com/repos/simonw/datasette/issues/1549,991754237,IC_kwDOBm6k_c47HPf9,536941,fgregg,2021-12-11T19:14:39Z,2021-12-11T19:14:39Z,CONTRIBUTOR,"that option is not available on [custom queries](https://labordata.bunkum.us/odpr-962a140?sql=with+local_union_filings+as+%28%0D%0A++select+*+from+lm_data+%0D%0A++where%0D%0A++++yr_covered+%3E+cast%28strftime%28%27%25Y%27%2C+%27now%27%2C+%27-5+years%27%29+as+int%29%0D%0A++++and+desig_name+%3D+%27LU%27%0D%0A++order+by+yr_covered+desc%0D%0A%29%2C%0D%0Amost_recent_filing+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from+local_union_filings%0D%0A++group+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++most_recent_filing%0D%0Awhere%0D%0A++next_election+%3E%3D+strftime%28%27%25Y-%25m%27%2C+%27now%27%29%0D%0A++and+next_election+%3C+strftime%28%27%25Y-%25m%27%2C+%27now%27%2C+%27%2B1+year%27%29%0D%0Aorder+by%0D%0A++members+desc%3B). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability, https://github.com/simonw/datasette/issues/1549#issuecomment-991754794,https://api.github.com/repos/simonw/datasette/issues/1549,991754794,IC_kwDOBm6k_c47HPoq,9599,simonw,2021-12-11T19:16:33Z,2021-12-11T19:16:33Z,OWNER,Good call! I'm doing a refactor #1518 right now which will hopefully bring the functionality of those two much closer - I'll make a note to consider this there too.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability, https://github.com/simonw/datasette/issues/617#issuecomment-991755013,https://api.github.com/repos/simonw/datasette/issues/617,991755013,IC_kwDOBm6k_c47HPsF,9599,simonw,2021-12-11T19:17:11Z,2021-12-11T19:17:11Z,OWNER,This work is now happening in #1518 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519613116,Refactor TableView.data() method, https://github.com/simonw/datasette/issues/1549#issuecomment-991755245,https://api.github.com/repos/simonw/datasette/issues/1549,991755245,IC_kwDOBm6k_c47HPvt,9599,simonw,2021-12-11T19:17:54Z,2021-12-11T19:17:54Z,OWNER,"Also relevant: - #1062 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability, https://github.com/simonw/datasette/issues/1550#issuecomment-991761635,https://api.github.com/repos/simonw/datasette/issues/1550,991761635,IC_kwDOBm6k_c47HRTj,9599,simonw,2021-12-11T19:39:01Z,2021-12-11T19:39:01Z,OWNER,"I wonder if this could work for public instances too with some kind of queuing mechanism? I really need to use benchmarking to figure out what the right number of maximum SQLite connections is. I'm just guessing at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077628073,Research option for returning all rows from arbitrary query, https://github.com/simonw/datasette/issues/1550#issuecomment-991805516,https://api.github.com/repos/simonw/datasette/issues/1550,991805516,IC_kwDOBm6k_c47HcBM,9599,simonw,2021-12-11T23:43:24Z,2021-12-11T23:43:24Z,OWNER,"I built a tiny Starlette app to experiment with this a bit: ```python import asyncio import janus from starlette.applications import Starlette from starlette.responses import JSONResponse, HTMLResponse, StreamingResponse from starlette.routing import Route import sqlite3 from concurrent import futures executor = futures.ThreadPoolExecutor(max_workers=10) async def homepage(request): return HTMLResponse( """""" SQL CSV Server

    SQL CSV Server

    """""" ) def run_query_in_thread(sql, sync_q): db = sqlite3.connect(""../datasette/covid.db"") cursor = db.cursor() cursor.arraysize = 100 # Default is 1 apparently? cursor.execute(sql) columns = [d[0] for d in cursor.description] sync_q.put([columns]) # Now start putting batches of rows while True: rows = cursor.fetchmany() if rows: sync_q.put(rows) else: break # Let queue know we are finished\ sync_q.put(None) async def csv_query(request): sql = request.query_params[""sql""] queue = janus.Queue() loop = asyncio.get_running_loop() async def csv_generator(): loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q) while True: rows = await queue.async_q.get() if rows is not None: for row in rows: yield "","".join(map(str, row)) + ""\n "" queue.async_q.task_done() else: # Cleanup queue.close() await queue.wait_closed() break return StreamingResponse(csv_generator(), media_type='text/plain') app = Starlette( debug=True, routes=[ Route(""/"", homepage), Route(""/csv"", csv_query), ], ) ``` But.. if I run this in a terminal window: ``` /tmp % wget 'http://127.0.0.1:8000/csv?sql=select+*+from+ny_times_us_counties' ``` it takes about 20 seconds to run and returns a 50MB file - but while it is running no other requests can be served by that server - not even the homepage! So something is blocking the event loop. Maybe I should be using `fut = loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)` and then awaiting `fut` somewhere, like in the Janus documentation? Don't think that's needed though. Needs more work to figure out why this is blocking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077628073,Research option for returning all rows from arbitrary query, https://github.com/simonw/datasette/issues/1518#issuecomment-991819781,https://api.github.com/repos/simonw/datasette/issues/1518,991819781,IC_kwDOBm6k_c47HfgF,9599,simonw,2021-12-12T01:53:10Z,2021-12-12T01:53:10Z,OWNER,"I have a hunch that the conclusion of this experiment may end up being that the `asyncinject` trick is kinda neat but the code will be easier to maintain (while still executing in parallel) if it's written using `asyncio.gather` directly instead. It's possible `asyncinject` will end up being neat enough that I'll want to keep it though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991822853,https://api.github.com/repos/simonw/datasette/issues/1518,991822853,IC_kwDOBm6k_c47HgQF,9599,simonw,2021-12-12T02:24:00Z,2021-12-12T02:24:00Z,OWNER,Rebuilding `TableView` from the ground up is proving not to be much fun. I'm going to explore starting the refactor of the existing code by separating out the bit that generates the SQL query from the rest of it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991823001,https://api.github.com/repos/simonw/datasette/issues/1518,991823001,IC_kwDOBm6k_c47HgSZ,9599,simonw,2021-12-12T02:25:32Z,2021-12-12T02:25:32Z,OWNER,The tests for `TableView` are currently mixed in with everything else in `tests/test_api.py` and `tests/html.py` - might be good to split those out into `test_table_html.py` and `test_table_api.py` since they're such a key part of how Datasette works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991827468,https://api.github.com/repos/simonw/datasette/issues/1518,991827468,IC_kwDOBm6k_c47HhYM,9599,simonw,2021-12-12T03:15:00Z,2021-12-12T03:15:00Z,OWNER," I don't think this code is necessary any more: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/views/table.py#L396-L399 That dates back from when Datasette was built on top of Sanic and Sanic didn't preserve those query parameters the way I needed it to: https://github.com/simonw/datasette/blob/1f69269fe93e4cd42e56890126cc0dbcf719c6cb/datasette/views/table.py#L202-L206","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-991828014,https://api.github.com/repos/simonw/datasette/issues/1518,991828014,IC_kwDOBm6k_c47Hhgu,9599,simonw,2021-12-12T03:21:35Z,2021-12-12T03:21:35Z,OWNER,"No, removing that gave me the following test failure: ``` tests/test_table_api.py::test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] FAILED [100%] =============================================================================== FAILURES ================================================================================ ______________________________________ test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] ______________________________________ app_client = , path = '/fixtures/simple_primary_key.json?content__exact=', expected_rows = [['3', '']] @pytest.mark.parametrize( ""path,expected_rows"", [ (""/fixtures/simple_primary_key.json?content=hello"", [[""1"", ""hello""]]), ( ""/fixtures/simple_primary_key.json?content__contains=o"", [ [""1"", ""hello""], [""2"", ""world""], [""4"", ""RENDER_CELL_DEMO""], ], ), (""/fixtures/simple_primary_key.json?content__exact="", [[""3"", """"]]), ( ""/fixtures/simple_primary_key.json?content__not=world"", [ [""1"", ""hello""], [""3"", """"], [""4"", ""RENDER_CELL_DEMO""], [""5"", ""RENDER_CELL_ASYNC""], ], ), ], ) def test_table_filter_queries(app_client, path, expected_rows): response = app_client.get(path) > assert expected_rows == response.json[""rows""] E AssertionError: assert [['3', '']] == [['1', 'hello'],\n ['2', 'world'],\n ['3', ''],\n ['4', 'RENDER_CELL_DEMO'],\n ['5', 'RENDER_CELL_ASYNC']] E At index 0 diff: ['3', ''] != ['1', 'hello'] E Right contains 4 more items, first extra item: ['2', 'world'] E Full diff: E [ E - ['1', E - 'hello'], E - ['2', E - 'world'], E ['3', E ''], E - ['4', E - 'RENDER_CELL_DEMO'], E - ['5', E - 'RENDER_CELL_ASYNC'], E ] /Users/simon/Dropbox/Development/datasette/tests/test_table_api.py:511: AssertionError ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1551#issuecomment-991960416,https://api.github.com/repos/simonw/datasette/issues/1551,991960416,IC_kwDOBm6k_c47IB1g,9599,simonw,2021-12-12T19:56:12Z,2021-12-12T19:56:12Z,OWNER,"Python documentation for `parse_qs`: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.parse_qs > The optional argument *keep_blank_values* is a flag indicating whether blank values in percent-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`, https://github.com/simonw/datasette/issues/1551#issuecomment-991960719,https://api.github.com/repos/simonw/datasette/issues/1551,991960719,IC_kwDOBm6k_c47IB6P,9599,simonw,2021-12-12T19:58:17Z,2021-12-12T19:58:17Z,OWNER,"Here's an example of the difference that causes: ```pycon >>> import urllib.parse >>> urllib.parse.parse_qs(""foo=bar"") {'foo': ['bar']} >>> urllib.parse.parse_qs(""foo=bar&baz="") {'foo': ['bar']} >>> urllib.parse.parse_qs(""foo=bar&baz="", keep_blank_values=True) {'foo': ['bar'], 'baz': ['']} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`, https://github.com/simonw/datasette/issues/1551#issuecomment-991960179,https://api.github.com/repos/simonw/datasette/issues/1551,991960179,IC_kwDOBm6k_c47IBxz,9599,simonw,2021-12-12T19:54:45Z,2021-12-12T19:54:45Z,OWNER,This is technically a backwards-incompatible for any plugins that use `request.args` - but it's unlikely to break anything. At any rate this needs to happen before Datasette 1.0!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077893013,`keep_blank_values=True` when parsing `request.args`, https://github.com/simonw/datasette/issues/1518#issuecomment-991978789,https://api.github.com/repos/simonw/datasette/issues/1518,991978789,IC_kwDOBm6k_c47IGUl,9599,simonw,2021-12-12T22:04:19Z,2021-12-12T22:04:19Z,OWNER,Idea: in JSON output include a `warnings` block listing any _ parameters that were not recognized.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/526#issuecomment-992971072,https://api.github.com/repos/simonw/datasette/issues/526,992971072,IC_kwDOBm6k_c47L4lA,536941,fgregg,2021-12-13T22:29:34Z,2021-12-13T22:29:34Z,CONTRIBUTOR,just came by to open this issue. would make my data analysis in observable a lot better!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/1553#issuecomment-992986587,https://api.github.com/repos/simonw/datasette/issues/1553,992986587,IC_kwDOBm6k_c47L8Xb,536941,fgregg,2021-12-13T22:57:04Z,2021-12-13T22:57:04Z,CONTRIBUTOR,would also be good if the header said the what the max row limit was,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079111498,if csv export is truncated in non streaming mode set informative response header, https://github.com/simonw/datasette/issues/1518#issuecomment-992833868,https://api.github.com/repos/simonw/datasette/issues/1518,992833868,IC_kwDOBm6k_c47LXFM,9599,simonw,2021-12-13T19:59:17Z,2021-12-13T19:59:17Z,OWNER,"Built a new plugin to help with this work by improving the display of `?_trace=1` output: https://datasette.io/plugins/datasette-pretty-traces ![image](https://user-images.githubusercontent.com/9599/145879751-36621f43-ba68-4ccd-b14b-379ed8f2111a.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-993000787,https://api.github.com/repos/simonw/datasette/issues/1518,993000787,IC_kwDOBm6k_c47L_1T,9599,simonw,2021-12-13T23:19:20Z,2021-12-14T17:06:05Z,OWNER,"Useful old comment here: https://github.com/simonw/datasette/issues/617#issuecomment-552253893 > As noted in [#621 (comment)](https://github.com/simonw/datasette/issues/621#issuecomment-552253208) a common pattern in this method is blocks of code that append new items to the `where_clauses`, `params` and `extra_human_descriptions` arrays. This is a useful refactoring opportunity. > > Code that fits this pattern: > > * The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)` > * Code that handles `?_where=`: `where_clauses.extend(request.args[""_where""])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses > * The `_through=` code, see [Syntax for ?_through= that works as a form field #621](https://github.com/simonw/datasette/issues/621) for details > * The code that deals with `?_search=` FTS > > The keyset pagination code modifies `where_clauses` and `params` too, but I don't think it's quite going to work with the same abstraction that would cover the above examples. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/pull/1554#issuecomment-993002933,https://api.github.com/repos/simonw/datasette/issues/1554,993002933,IC_kwDOBm6k_c47MAW1,22429695,codecov[bot],2021-12-13T23:22:58Z,2021-12-13T23:22:58Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1554](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1d08b46) into [main](https://codecov.io/gh/simonw/datasette/commit/a6ff123de5464806441f6a6f95145c9a83b7f20b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a6ff123) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1554/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1554 +/- ## ======================================= Coverage 91.84% 91.84% ======================================= Files 34 34 Lines 4437 4437 ======================================= Hits 4075 4075 Misses 362 362 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.04% <ø> (ø)` | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a6ff123...1d08b46](https://codecov.io/gh/simonw/datasette/pull/1554?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079129258,TableView refactor, https://github.com/simonw/datasette/pull/1554#issuecomment-993006521,https://api.github.com/repos/simonw/datasette/issues/1554,993006521,IC_kwDOBm6k_c47MBO5,9599,simonw,2021-12-13T23:28:47Z,2021-12-13T23:28:47Z,OWNER,"That's frustrating: you can only attach comments to lines that were changed in the PR or are within about 3-4 lines of them: ![comments](https://user-images.githubusercontent.com/9599/145905357-5d8873f5-99c9-4b46-b4d5-35d38f5cb686.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079129258,TableView refactor, https://github.com/simonw/datasette/issues/1553#issuecomment-993014772,https://api.github.com/repos/simonw/datasette/issues/1553,993014772,IC_kwDOBm6k_c47MDP0,536941,fgregg,2021-12-13T23:46:18Z,2021-12-13T23:46:18Z,CONTRIBUTOR,these headers would also be relevant for json exports of custom queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079111498,if csv export is truncated in non streaming mode set informative response header, https://github.com/simonw/datasette/issues/526#issuecomment-993078038,https://api.github.com/repos/simonw/datasette/issues/526,993078038,IC_kwDOBm6k_c47MSsW,536941,fgregg,2021-12-14T01:46:52Z,2021-12-14T01:46:52Z,CONTRIBUTOR,"the nested query idea is very nice, and i stole if for [my client side paginator](https://observablehq.com/d/1d5da3a3c3f2f347#DatasetteClient). However, it won't do the right thing if the original query orders by random(). If you go the nested query route, maybe raise a 4XX status code if the query has such a clause?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/1518#issuecomment-994042389,https://api.github.com/repos/simonw/datasette/issues/1518,994042389,IC_kwDOBm6k_c47P-IV,9599,simonw,2021-12-14T21:35:53Z,2021-12-14T21:35:53Z,OWNER,"Maybe a better way to approach this would be to focus on the JSON side of things - try to get a basic JSON version with `?_extra=` support working, then eventually build that up to the point where it can power the HTML version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/621#issuecomment-994005634,https://api.github.com/repos/simonw/datasette/issues/621,994005634,IC_kwDOBm6k_c47P1KC,9599,simonw,2021-12-14T21:02:50Z,2021-12-14T21:02:50Z,OWNER,"This would also mean that an extra text input box could be easily shown on the page. https://latest-with-plugins.datasette.io/fixtures/roadside_attractions?_through={""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id"",""value"":""1""} but with the annotated box added (and made to look good): ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/1518#issuecomment-993794247,https://api.github.com/repos/simonw/datasette/issues/1518,993794247,IC_kwDOBm6k_c47PBjH,9599,simonw,2021-12-14T17:09:40Z,2021-12-14T17:09:40Z,OWNER,- `table_actions` should be an extra.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/621#issuecomment-993813210,https://api.github.com/repos/simonw/datasette/issues/621,993813210,IC_kwDOBm6k_c47PGLa,9599,simonw,2021-12-14T17:30:13Z,2021-12-14T20:23:57Z,OWNER,"Might be able to create a web form that's unambiguous using: `https://latest.datasette.io/fixtures/roadside_attractions?_through.[""roadside_attraction_characteristics"",""characteristic_id""]=1` So: ```html ``` I'm pretty confident this is allowed by the HTML specification. This works: ```html
    ``` ASGI parsing seems to work too: https://latest-with-plugins.datasette.io/-/asgi-scope?_through.[%22roadside_attraction_characteristics%22%2C%22characteristic_id%22]=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/1423#issuecomment-993876599,https://api.github.com/repos/simonw/datasette/issues/1423,993876599,IC_kwDOBm6k_c47PVp3,6165713,plpxsk,2021-12-14T18:48:09Z,2021-12-14T18:48:09Z,NONE,"Great feature. But what is the right way to enable this to show up? Currently, it seems I need to edit the URL to add, in the right place, `&_facet_size=max` Is there another (easier) way to enable this feature?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/621#issuecomment-993958242,https://api.github.com/repos/simonw/datasette/issues/621,993958242,IC_kwDOBm6k_c47Ppli,9599,simonw,2021-12-14T20:33:25Z,2021-12-14T20:33:56Z,OWNER,"Alternative idea: since current syntax is: `?_through={""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id"",""value"":""1""}` The form-encoding-friendly syntax could be: `?_through.{""table"":""roadside_attraction_characteristics"",""column"":""characteristic_id""}=1` Which is more consistent than the array proposal: `?_through.[""roadside_attraction_characteristics"",""characteristic_id""]=1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/1518#issuecomment-994085710,https://api.github.com/repos/simonw/datasette/issues/1518,994085710,IC_kwDOBm6k_c47QItO,9599,simonw,2021-12-14T22:03:16Z,2021-12-14T22:04:28Z,OWNER,"There are actually four forms of SQL query used by the table page: - `from_sql` - just the `from table_name where ...` - `sql_no_order_no_limit` - used for faceting, `""select {select_all_columns} from {table_name} {where}""` - `sql` - the above but with order and limit clauses: `""select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}""` - `count_sql` used for the count, built out of `from_sql`: `""select count(*) {from_sql}""` I'm tempted to encapsulate those in a `Query` class.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1542#issuecomment-995022217,https://api.github.com/repos/simonw/datasette/issues/1542,995022217,IC_kwDOBm6k_c47TtWJ,9599,simonw,2021-12-15T17:47:07Z,2021-12-15T17:47:07Z,OWNER,"This does make sense to me. I've been hoping to significantly improve the way JavaScript plugins work - there are some notes on that here: - #983 Encouraging plugins such as `datasette-cluster-map` to emit events that can then be listened to by other plugins is a really interesting idea that I hadn't considered.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072106103,feature request: order and dependency of plugins (that use js), https://github.com/simonw/datasette/issues/1423#issuecomment-995023410,https://api.github.com/repos/simonw/datasette/issues/1423,995023410,IC_kwDOBm6k_c47Ttoy,9599,simonw,2021-12-15T17:48:40Z,2021-12-15T17:48:40Z,OWNER,You've caused me to rethink this feature - I no longer think there's value in only showing these numbers if `?_facet_size=max` as opposed to all of the time. New issue coming up.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max, https://github.com/simonw/datasette/issues/1552#issuecomment-995034143,https://api.github.com/repos/simonw/datasette/issues/1552,995034143,IC_kwDOBm6k_c47TwQf,9599,simonw,2021-12-15T18:02:53Z,2021-12-15T18:02:53Z,OWNER,"This is definitely a missing feature. The ""different types of facet"" stuff feels incomplete to me generally - this is one issue, but this one as well: - #625","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/262#issuecomment-995034911,https://api.github.com/repos/simonw/datasette/issues/262,995034911,IC_kwDOBm6k_c47Twcf,9599,simonw,2021-12-15T18:03:46Z,2021-12-15T18:03:56Z,OWNER,"This is relevant to the big refactor in: - #1518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/issues/1552#issuecomment-995296725,https://api.github.com/repos/simonw/datasette/issues/1552,995296725,IC_kwDOBm6k_c47UwXV,3556,davidbgk,2021-12-15T23:29:32Z,2021-12-15T23:29:32Z,CONTRIBUTOR,"@simonw thank you for your fast answer and your guidance! While digging into the code, I found an undocumented way of doing it: ```yaml facets: [""Facet for a column"", {""array"": ""Facet for an array""}] ``` The only remaining problem with that solution is here: https://github.com/simonw/datasette/blob/250db8192cb8aba5eb8cd301ccc2a49525bc3d24/datasette/facets.py#L33 We have: ```python type, metadata_config = metadata_config.items()[0] ``` But it requires to cast the `dict_items` as a list prior to access the first element: ```python type, metadata_config = list(metadata_config.items())[0] ``` I guess it's an unspotted bug? (I mean, independently of the facets-with-arrays issue.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/625#issuecomment-996100774,https://api.github.com/repos/simonw/datasette/issues/625,996100774,IC_kwDOBm6k_c47X0qm,9599,simonw,2021-12-16T19:10:01Z,2021-12-16T19:10:48Z,OWNER,"I think the problem here may be in the design of the JSON returned by facets. It looks like this: ``` ""facet_results"": { ""tags"": { ""name"": ""tags"", ""type"": ""array"", ""results"": [...], ""hideable"": false, ""toggle_url"": ""/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1"", ""truncated"": false }, ""created"": { ""name"": ""created"", ""type"": ""date"", ""results"": [...] ``` The problem then is that the `tags` key is over-ridden by the second facet with a different type against the same column name! https://latest-with-plugins.datasette.io/fixtures/facetable?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags confirms that the SQL queries for those facets are being executed - but the final JSON doesn't show them on https://latest-with-plugins.datasette.io/fixtures/facetable.json?_trace=1&_facet=created&_facet_date=created&_facet_array=tags&_facet=tags They're not available in the template context either: https://latest-with-plugins.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet_array=tags&_facet=tags&_context=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1553#issuecomment-996103956,https://api.github.com/repos/simonw/datasette/issues/1553,996103956,IC_kwDOBm6k_c47X1cU,9599,simonw,2021-12-16T19:14:38Z,2021-12-16T19:14:38Z,OWNER,This is a really interesting idea - kind of similar to how many APIs include custom HTTP headers informing of rate-limits.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1079111498,if csv export is truncated in non streaming mode set informative response header, https://github.com/simonw/datasette/issues/1556#issuecomment-996104214,https://api.github.com/repos/simonw/datasette/issues/1556,996104214,IC_kwDOBm6k_c47X1gW,9599,simonw,2021-12-16T19:15:00Z,2021-12-16T19:15:28Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet=planet_int&_facet=_city_id&_facet=created#facet-created ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1081318247,"Show count of facet values always, not just for `?_facet_size=max`", https://github.com/simonw/datasette/issues/625#issuecomment-996118401,https://api.github.com/repos/simonw/datasette/issues/625,996118401,IC_kwDOBm6k_c47X4-B,9599,simonw,2021-12-16T19:34:28Z,2021-12-16T19:34:55Z,OWNER,"The big question here is do I break any existing clients of the `""facet_results""` JSON API? It's still pre-1.0 so I could break them, but I've also built my own code against this in the past so it's likely other people have too. If I don't break them, I will instead need to come up with a naming convention for those keys - something like `""tags__array""` for example. As well as a way to ensure that a column called `tags__array` doesn't end up conflicting with the `tags__array` key!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1557#issuecomment-996115949,https://api.github.com/repos/simonw/datasette/issues/1557,996115949,IC_kwDOBm6k_c47X4Xt,9599,simonw,2021-12-16T19:30:55Z,2021-12-16T19:30:55Z,OWNER,"Demo: compare https://latest.datasette.io/fixtures/facetable?_facet=_city_id&_nosuggest=1 to https://latest.datasette.io/fixtures/facetable?_facet=_city_id Documentation: bottom of https://docs.datasette.io/en/latest/json_api.html#special-table-arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082564912,`?_nosuggest=1` parameter for disabling facet suggestions on table view, https://github.com/simonw/datasette/issues/625#issuecomment-996119954,https://api.github.com/repos/simonw/datasette/issues/625,996119954,IC_kwDOBm6k_c47X5WS,9599,simonw,2021-12-16T19:36:01Z,2021-12-16T19:36:11Z,OWNER,"Datasette's own HTML rendering code doesn't actually use the keys in `facet_results` - it instead loops through `sorted_facet_results` which is defined like this: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L937-L941 And used like this: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/templates/table.html#L154-L156","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996121736,https://api.github.com/repos/simonw/datasette/issues/625,996121736,IC_kwDOBm6k_c47X5yI,9599,simonw,2021-12-16T19:37:08Z,2021-12-16T19:37:08Z,OWNER,"Really `facet_results` here should be an array of objects, not an object that maps poorly designed string keys to those objects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1558#issuecomment-996134716,https://api.github.com/repos/simonw/datasette/issues/1558,996134716,IC_kwDOBm6k_c47X888,9599,simonw,2021-12-16T19:46:21Z,2021-12-16T19:46:21Z,OWNER,"The flaw in the current design is illustrated by this example: ``` ""facet_results"": { ""tags"": { ""name"": ""tags"", ""type"": ""array"", ""results"": [...], ""hideable"": false, ""toggle_url"": ""/fixtures/facetable.json?_facet=tags&_trace=1&_nosuggest=1"", ""truncated"": false }, ""created"": { ""name"": ""created"", ""type"": ""date"", ""results"": [...] ``` This was the cause of the bug in #625 - the each of those objects is keyed by the name of the column, which left no room for faceting the same column once by date and once by column value.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082584499,Redesign `facet_results` JSON structure prior to Datasette 1.0, https://github.com/simonw/datasette/issues/625#issuecomment-996130862,https://api.github.com/repos/simonw/datasette/issues/625,996130862,IC_kwDOBm6k_c47X8Au,9599,simonw,2021-12-16T19:44:48Z,2021-12-16T19:44:48Z,OWNER,"Decision: as an initial fix I'm going to de-duplicate those keys by using `tags__array` etc - with a `_2` on the end if that key is already used. I'll open a separate issue to redesign this better for Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996146762,https://api.github.com/repos/simonw/datasette/issues/625,996146762,IC_kwDOBm6k_c47X_5K,9599,simonw,2021-12-16T19:51:44Z,2021-12-16T19:51:44Z,OWNER,"Here's where `facet_results` is built up: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/views/table.py#L752-L758 So the decision to key things based on column name is actually embedded deep in the existing facet classes here: https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L224-L226 https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L395-L397 https://github.com/simonw/datasette/blob/992496f2611a72bd51e94bfd0b17c1d84e732487/datasette/facets.py#L510-L512","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1552#issuecomment-996034408,https://api.github.com/repos/simonw/datasette/issues/1552,996034408,IC_kwDOBm6k_c47Xkdo,9599,simonw,2021-12-16T17:37:37Z,2021-12-16T17:37:37Z,OWNER,"I think you're right! I had completely forgotten that piece of code. This just turned into a bug fix and a documentation update. Thanks for the research!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996045776,https://api.github.com/repos/simonw/datasette/issues/1552,996045776,IC_kwDOBm6k_c47XnPQ,9599,simonw,2021-12-16T17:52:54Z,2021-12-16T17:52:54Z,OWNER,"I tried that fix you suggested and now this `metadata.json` does the right thing: ```json { ""databases"": { ""fixtures"": { ""tables"": { ""facetable"": { ""facets"": [ { ""array"": ""tags"" } ] } } } } } ``` It does further highlight the bug in #625 though - since then if you try to add `?_facet=tags` to facet by tags treating them NOT as an array your request to do so is ignored.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996046304,https://api.github.com/repos/simonw/datasette/issues/1552,996046304,IC_kwDOBm6k_c47XnXg,9599,simonw,2021-12-16T17:53:40Z,2021-12-16T18:16:12Z,OWNER,"I'm also not convinced that this configuration syntax is right. It's a bit weird having a `""facets""` list that can either by column-name-strings or `{""type-of-facet"": ""column-name""}` objects. Maybe there's a better design for this? Part of the problem here is that facets were designed to accept optional extra configuration - partly to support `m2m` facets in #495 - but I haven't actually shipped any facets that use that ability. Facet by delimiter would be a good one to exercise that ability: - #510","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996077053,https://api.github.com/repos/simonw/datasette/issues/1552,996077053,IC_kwDOBm6k_c47Xu39,9599,simonw,2021-12-16T18:36:41Z,2021-12-16T18:36:41Z,OWNER,"... actually no, I WILL document this, because not documenting this is what got us to this point in the first place!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996076373,https://api.github.com/repos/simonw/datasette/issues/1552,996076373,IC_kwDOBm6k_c47XutV,9599,simonw,2021-12-16T18:35:40Z,2021-12-16T18:35:40Z,OWNER,"I'm going to ship your fix now, but I'm not going to add this to the documentation yet because I hope to improve the design prior to Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1552#issuecomment-996084899,https://api.github.com/repos/simonw/datasette/issues/1552,996084899,IC_kwDOBm6k_c47Xwyj,9599,simonw,2021-12-16T18:48:14Z,2021-12-16T18:48:14Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/20a2ed6bec367d2f6759be4a879364a72780b59d/docs/facets.rst#facets-in-metadatajson,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/625#issuecomment-996093884,https://api.github.com/repos/simonw/datasette/issues/625,996093884,IC_kwDOBm6k_c47Xy-8,9599,simonw,2021-12-16T19:00:28Z,2021-12-16T19:00:28Z,OWNER,Implementing #1552 has made a fix for this bug even more important.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996150904,https://api.github.com/repos/simonw/datasette/issues/625,996150904,IC_kwDOBm6k_c47YA54,9599,simonw,2021-12-16T19:57:52Z,2021-12-16T19:57:52Z,OWNER,Good news - GitHub's new code search doesn't show ANYONE using that plugin hook - not surprising since it has that documentation warning plus it's just not a very clearly usable hook: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/830#issuecomment-996151246,https://api.github.com/repos/simonw/datasette/issues/830,996151246,IC_kwDOBm6k_c47YA_O,9599,simonw,2021-12-16T19:58:22Z,2021-12-16T19:58:22Z,OWNER,"As of today, 16 December 2021, I'm still not seeing any evidence that anyone is using this hook (yet) according to GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=register_facet_classes%20-repo%3Asimonw%2Fdatasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636511683,Redesign register_facet_classes plugin hook, https://github.com/simonw/datasette/issues/625#issuecomment-996149720,https://api.github.com/repos/simonw/datasette/issues/625,996149720,IC_kwDOBm6k_c47YAnY,9599,simonw,2021-12-16T19:56:14Z,2021-12-16T19:56:14Z,OWNER,"This bad design is even covered in the plugin hooks documentation: https://docs.datasette.io/en/0.59.4/plugin_hooks.html#register-facet-classes It does at least have the following warning: > **Warning** > > The design of this plugin hook is unstable and may change. See [issue 830](https://github.com/simonw/datasette/issues/830).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996152213,https://api.github.com/repos/simonw/datasette/issues/625,996152213,IC_kwDOBm6k_c47YBOV,9599,simonw,2021-12-16T19:59:46Z,2021-12-16T20:00:05Z,OWNER,"Since no-one is using that plugin hook I'm going to alter its contract slightly. I'll still keep the existing JSON format working though (until 1.0), since it's much more likely that people are using that JSON somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996161380,https://api.github.com/repos/simonw/datasette/issues/625,996161380,IC_kwDOBm6k_c47YDdk,9599,simonw,2021-12-16T20:13:05Z,2021-12-16T20:13:05Z,OWNER,I updated the example code in the facet plugin hook documentation: https://github.com/simonw/datasette/blob/95d0dd7a1cf6be6b7da41e1404184217eb93f64a/docs/plugin_hooks.rst#register_facet_classes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996165659,https://api.github.com/repos/simonw/datasette/issues/625,996165659,IC_kwDOBm6k_c47YEgb,9599,simonw,2021-12-16T20:19:53Z,2021-12-16T20:19:53Z,OWNER,Demo of the fix: https://latest.datasette.io/fixtures/facetable?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags#facet-tags,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/625#issuecomment-996170510,https://api.github.com/repos/simonw/datasette/issues/625,996170510,IC_kwDOBm6k_c47YFsO,9599,simonw,2021-12-16T20:27:41Z,2021-12-16T20:27:41Z,OWNER,"And here's the new JSON: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1 ``` { ""database"": ""fixtures"", ""table"": ""facetable"", ""is_view"": false, ""human_description_en"": """", ... ""facet_results"": { ""created"": { ""name"": ""created"", ""type"": ""column"", ... }, ""tags"": { ""name"": ""tags"", ""type"": ""column"", ... }, ""created_2"": { ""name"": ""created"", ""type"": ""date"", ... }, ""tags_2"": { ""name"": ""tags"", ""type"": ""array"", ... } } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520740741,If you apply ?_facet_array=tags then &_facet=tags does nothing, https://github.com/simonw/datasette/issues/1558#issuecomment-996204369,https://api.github.com/repos/simonw/datasette/issues/1558,996204369,IC_kwDOBm6k_c47YN9R,9599,simonw,2021-12-16T21:23:25Z,2021-12-16T21:23:25Z,OWNER,"Related: Following the fix for #625 I noticed that `facets_timed_out` gives you just the column name, but doesn't let you know which particular type of facet (`date` or `array` for example) suffered the timeout: https://github.com/simonw/datasette/blob/0d4145d0f4d8b2a7edc1ba4aac1be56cd536a10a/datasette/facets.py#L269-L270 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082584499,Redesign `facet_results` JSON structure prior to Datasette 1.0, https://github.com/simonw/datasette/issues/1518#issuecomment-996219117,https://api.github.com/repos/simonw/datasette/issues/1518,996219117,IC_kwDOBm6k_c47YRjt,9599,simonw,2021-12-16T21:47:51Z,2021-12-16T21:49:24Z,OWNER,"Should facets really not be displayed on pages past page one (where `?_next=` is set)? That made sense to me at the time, but I'm now having second thoughts about it. I guess it's a useful performance tweak for when crawlers keep hitting the `?_next=` link. Actually it looks like facets DO display on subsequent pages, e.g. on https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 - but facet suggestions do not, thanks to this code: https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/views/table.py#L777-L785 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1552#issuecomment-996229007,https://api.github.com/repos/simonw/datasette/issues/1552,996229007,IC_kwDOBm6k_c47YT-P,3556,davidbgk,2021-12-16T22:04:39Z,2021-12-16T22:04:39Z,CONTRIBUTOR,"Wow, that was fast, thank you so much @simonw ! > I'm also not convinced that this configuration syntax is right. It's a bit weird having a `""facets""` list that can either by column-name-strings or `{""type-of-facet"": ""column-name""}` objects. Maybe there's a better design for this? I agree that it's not ideal, my initial naive approach was to detect if it's an array, like what is done here: https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/facets.py#L312-L313 But it requires an extra query to determine the type, which is a bit problematic, especially for big tables I guess. Taking a look at #510, I wonder if a `facet_delimiter` should be defined for that kind of columns (that would help our team not to have an intermediary conversion step from `foo|bar` to `[""foo"",""bar""]` for instance). To be consistent with the `--extract-column` parameter, maybe an explicit casting/delimiter would be useful: `--set-column 'Foo:Array:|'`. Throwing a lot of ideas without knowing the big picture… but sometimes newcomers have superpowers :).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1078702875,Allow to set `facets_array` in metadata (like current `facets`), https://github.com/simonw/datasette/issues/1518#issuecomment-996225235,https://api.github.com/repos/simonw/datasette/issues/1518,996225235,IC_kwDOBm6k_c47YTDT,9599,simonw,2021-12-16T21:58:24Z,2021-12-16T21:58:41Z,OWNER,"A fundamental operation of this view is to construct the SQL query and accompanying human description based on the incoming query string parameters. The human description is the bit at the top of https://latest.datasette.io/fixtures/searchable?_search=dog&_sort=pk&_facet=text2&text2=sara+weasel that says: > 1 row where search matches ""dog"" and text2 = ""sara weasel"" sorted by pk (Also used in the page ``). The code actually gathers three things: - Fragments of the `where` clause, for example ` ""text2"" = :p0` - Parameters, e.g. `{""p0"": ""sara weasel""}` - Human description components, e.g. `text2 = ""sara weasel""` Some operations such as `?_where=` don't currently provide an extra human description component. `_where=` also doesn't populate a parameter, but maybe it could? Would be neat if in the future `?_where=foo+=+:bar` worked and added a `bar` input field to the screen, as seen with custom queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996225889,https://api.github.com/repos/simonw/datasette/issues/1518,996225889,IC_kwDOBm6k_c47YTNh,9599,simonw,2021-12-16T21:59:32Z,2021-12-16T22:00:42Z,OWNER,I added a ton of comments to the `data()` method which really helps get a better feel for how this all works: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L322,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996227713,https://api.github.com/repos/simonw/datasette/issues/1518,996227713,IC_kwDOBm6k_c47YTqB,9599,simonw,2021-12-16T22:02:35Z,2021-12-16T22:03:55Z,OWNER,"Is there an opportunity to refactor things using a new plugin hook here? Maybe the `register_filters` hook from #473, where the hook becomes responsible for building where clauses (and human descriptions of them) based on the incoming query string. That version dealt with `Filter` classes, but those might be a bit too low-level for this. `?_spatial_within=GEOJSON` was an interesting idea attached to that issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996240802,https://api.github.com/repos/simonw/datasette/issues/1518,996240802,IC_kwDOBm6k_c47YW2i,9599,simonw,2021-12-16T22:25:00Z,2021-12-16T22:36:04Z,OWNER,"I think that plugin hook would get given the `request` object (and `datasette` and the name of the database and table) and returns a list of SQL fragments, a dictionary of lookup arguments and a list of human-description fragments - or an awaitable. `filters_from_request(request, database, table, datasette)` perhaps? (Similar in name to `actor_from_request`). ```python @hookspec def filters_from_request(request, database, table, datasette): """"""Return (where_clauses, params_dict, human_descriptions) based on the request"""""" ``` Turns out that's pretty much exactly what I implemented in 5116c4ec8aed5091e1f75415424b80f613518dc6 for #473: ```python @hookspec def table_filter(): ""Custom filtering of the current table based on the request"" ``` ```python TableFilter = namedtuple(""TableFilter"", ( ""human_description_extras"", ""where_clauses"", ""params"") ) ``` ```python # filter_arguments plugin hook support for awaitable_fn in pm.hook.table_filter(): extras = await awaitable_fn( view=self, name=name, table=table, request=request ) human_description_extras.extend(extras.human_description_extras) where_clauses.extend(extras.where_clauses) params.update(extras.params) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996248713,https://api.github.com/repos/simonw/datasette/issues/1518,996248713,IC_kwDOBm6k_c47YYyJ,9599,simonw,2021-12-16T22:39:47Z,2021-12-16T22:39:47Z,OWNER,"The hook could return a named tuple like this one: ```python from typing import NamedTuple, List, Optional, Union, Dict class FilterArguments(NamedTuple): where_clauses: List[str] params: Dict[str, Union[str, int, float]] human_descriptions: List[str] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996250585,https://api.github.com/repos/simonw/datasette/issues/1518,996250585,IC_kwDOBm6k_c47YZPZ,9599,simonw,2021-12-16T22:43:37Z,2021-12-16T22:45:07Z,OWNER,"Ran into a problem prototyping that hook up for handling `?_where=` - that feature also adds a little bit of extra template context in order to show the interface for removing wheres - the `extra_wheres_for_ui` variable: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L457-L463 Maybe change to this? ```python class FilterArguments(NamedTuple): where_clauses: List[str] params: Dict[str, Union[str, int, float]] human_descriptions: List[str] extra_context: Dict[str, Any] ``` That might be necessary for `_search` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-996264617,https://api.github.com/repos/simonw/datasette/issues/1518,996264617,IC_kwDOBm6k_c47Ycqp,9599,simonw,2021-12-16T23:11:12Z,2021-12-16T23:11:12Z,OWNER,I managed to extract both `_search=` and `_where=` out using a prototype of that hook. I wonder if it could extract the complex code for `?_next` too?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/473#issuecomment-996267817,https://api.github.com/repos/simonw/datasette/issues/473,996267817,IC_kwDOBm6k_c47Ydcp,9599,simonw,2021-12-16T23:17:52Z,2021-12-16T23:19:00Z,OWNER,"I revisited this idea in #1518 and came up with a slightly different name and design for the hook: ```python @hookspec def filters_from_request(request, database, table, datasette): """""" Return FilterArguments( where_clauses=[str, str, str], params={}, human_descriptions=[str, str, str], extra_context={} ) based on the request"""""" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/1518#issuecomment-996272906,https://api.github.com/repos/simonw/datasette/issues/1518,996272906,IC_kwDOBm6k_c47YesK,9599,simonw,2021-12-16T23:27:42Z,2021-12-16T23:27:42Z,OWNER,Got a TIL out of this: https://til.simonwillison.net/pluggy/multiple-hooks-same-file,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/473#issuecomment-996275108,https://api.github.com/repos/simonw/datasette/issues/473,996275108,IC_kwDOBm6k_c47YfOk,9599,simonw,2021-12-16T23:32:22Z,2021-12-16T23:32:30Z,OWNER,This filter design can only influence the `where` component of the SQL clause - it's not able to modify the `SELECT` columns or adjust the `ORDER BY` or `OFFSET LIMIT` parts. I think that's OK.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/1518#issuecomment-996286104,https://api.github.com/repos/simonw/datasette/issues/1518,996286104,IC_kwDOBm6k_c47Yh6Y,9599,simonw,2021-12-17T00:00:07Z,2021-12-17T00:00:07Z,OWNER,Documentation of the new hook in the PR: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/473#issuecomment-996286199,https://api.github.com/repos/simonw/datasette/issues/473,996286199,IC_kwDOBm6k_c47Yh73,9599,simonw,2021-12-17T00:00:22Z,2021-12-17T00:00:22Z,OWNER,Documentation for that hook in the PR branch: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/pull/1559#issuecomment-996286808,https://api.github.com/repos/simonw/datasette/issues/1559,996286808,IC_kwDOBm6k_c47YiFY,9599,simonw,2021-12-17T00:01:43Z,2021-12-17T00:01:43Z,OWNER,"This already has tests and documentation, and I've used it to refactor out the logic for `?_where=` and `?_search=` and `?_through=`. Do I like this enough to land it on `main`? Also, I think I can still use it to refactor out the `Filters` code that implements `?col=x` and `?col__lt=5` and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/pull/1559#issuecomment-996289541,https://api.github.com/repos/simonw/datasette/issues/1559,996289541,IC_kwDOBm6k_c47YiwF,22429695,codecov[bot],2021-12-17T00:07:42Z,2021-12-17T17:28:54Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1559](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (71af58d) into [main](https://codecov.io/gh/simonw/datasette/commit/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0663d55) will **increase** coverage by `0.09%`. > The diff coverage is `97.97%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1559/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1559 +/- ## ========================================== + Coverage 91.96% 92.05% +0.09% ========================================== Files 34 34 Lines 4442 4493 +51 ========================================== + Hits 4085 4136 +51 Misses 357 357 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/1559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | | | [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/1559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `95.69% <97.67%> (+1.33%)` | :arrow_up: | | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.21% <100.00%> (+0.13%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0663d55...71af58d](https://codecov.io/gh/simonw/datasette/pull/1559?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/issues/473#issuecomment-996345233,https://api.github.com/repos/simonw/datasette/issues/473,996345233,IC_kwDOBm6k_c47YwWR,9599,simonw,2021-12-17T01:20:31Z,2021-12-17T18:13:01Z,OWNER,I could use this hook to add table filtering on a map to the existing `datasette-leaflet-freedraw` plugin.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/issues/473#issuecomment-996484551,https://api.github.com/repos/simonw/datasette/issues/473,996484551,IC_kwDOBm6k_c47ZSXH,9599,simonw,2021-12-17T07:02:21Z,2021-12-17T07:04:23Z,OWNER,"The one slightly weird thing about this hook is how it adds `extra_context` without an obvious way for plugins to add extra HTML to the templates based on that context. Maybe I need the proposed mechanism from - #1191 Which has an in-progress PR: - #1204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/pull/1204#issuecomment-996488925,https://api.github.com/repos/simonw/datasette/issues/1204,996488925,IC_kwDOBm6k_c47ZTbd,9599,simonw,2021-12-17T07:10:48Z,2021-12-17T07:10:48Z,OWNER,I think this is missing the `_macro.html` template file but I have that in my Dropbox.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793002853,WIP: Plugin includes, https://github.com/simonw/datasette/pull/1559#issuecomment-996895423,https://api.github.com/repos/simonw/datasette/issues/1559,996895423,IC_kwDOBm6k_c47a2q_,9599,simonw,2021-12-17T17:28:44Z,2021-12-17T17:28:44Z,OWNER,"Before I land this I'm going to build one prototype plugin against it to confirm that the new hook is useful in its current shape. I'll add support for filtering a table by drawing on a map to https://datasette.io/plugins/datasette-leaflet-freedraw","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/pull/1562#issuecomment-996716158,https://api.github.com/repos/simonw/datasette/issues/1562,996716158,IC_kwDOBm6k_c47aK5-,22429695,codecov[bot],2021-12-17T13:18:49Z,2021-12-17T13:18:49Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1562](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2f008e8) into [main](https://codecov.io/gh/simonw/datasette/commit/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0663d55) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1562/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1562 +/- ## ======================================= Coverage 91.96% 91.96% ======================================= Files 34 34 Lines 4442 4442 ======================================= Hits 4085 4085 Misses 357 357 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0663d55...2f008e8](https://codecov.io/gh/simonw/datasette/pull/1562?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/issues/1568#issuecomment-997153253,https://api.github.com/repos/simonw/datasette/issues/1568,997153253,IC_kwDOBm6k_c47b1nl,9599,simonw,2021-12-18T06:20:23Z,2021-12-18T06:20:23Z,OWNER,Now running at https://latest-with-plugins.datasette.io/github/commits?_trace=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083726550,Trace should show queries on the write connection too, https://github.com/simonw/datasette/issues/473#issuecomment-996958442,https://api.github.com/repos/simonw/datasette/issues/473,996958442,IC_kwDOBm6k_c47bGDq,9599,simonw,2021-12-17T18:59:27Z,2021-12-17T18:59:27Z,OWNER,I'm happy with how the prototype that used this plugin in `datasette-leaflet-freedraw` turned out: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",445850934,Plugin hook: filters_from_request, https://github.com/simonw/datasette/pull/1559#issuecomment-996959325,https://api.github.com/repos/simonw/datasette/issues/1559,996959325,IC_kwDOBm6k_c47bGRd,9599,simonw,2021-12-17T18:59:54Z,2021-12-17T18:59:54Z,OWNER,I've convinced myself that this plugin hook design is good through this `datasette-leaflet-freedraw` prototype: https://github.com/simonw/datasette-leaflet-freedraw/blob/e8a16a0fe90656b8d655c02881d23a2b9833281d/datasette_leaflet_freedraw/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/pull/1559#issuecomment-996961196,https://api.github.com/repos/simonw/datasette/issues/1559,996961196,IC_kwDOBm6k_c47bGus,9599,simonw,2021-12-17T19:00:53Z,2021-12-17T19:00:53Z,OWNER,"I'm going to merge this to `main` now. I can continue the refactoring there, but having it in `main` means I can put out an alpha release with the new hook which will unblock me from running tests against it in this repo: https://github.com/simonw/datasette-leaflet-freedraw/pull/8","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/issues/1565#issuecomment-997069128,https://api.github.com/repos/simonw/datasette/issues/1565,997069128,IC_kwDOBm6k_c47bhFI,9599,simonw,2021-12-17T22:31:18Z,2021-12-17T22:31:18Z,OWNER,This should aim to be as consistent as possible with the various arguments to hooks on https://docs.datasette.io/en/stable/plugin_hooks.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/pull/1562#issuecomment-997080352,https://api.github.com/repos/simonw/datasette/issues/1562,997080352,IC_kwDOBm6k_c47bj0g,9599,simonw,2021-12-17T23:03:08Z,2021-12-17T23:03:08Z,OWNER,"They say they've dropped 3.6 support, but Datasette's tests against 3.6 are still passing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/issues/1565#issuecomment-997077410,https://api.github.com/repos/simonw/datasette/issues/1565,997077410,IC_kwDOBm6k_c47bjGi,9599,simonw,2021-12-17T22:54:45Z,2021-12-17T22:54:45Z,OWNER,"The table page should expose the query both with and without the `limit` clause. The above gave me back: ```sql select id, ACCESS_TYP, UNIT_ID, UNIT_NAME, SUID_NMA, AGNCY_ID, AGNCY_NAME, AGNCY_LEV, AGNCY_TYP, AGNCY_WEB, LAYER, MNG_AG_ID, MNG_AGENCY, MNG_AG_LEV, MNG_AG_TYP, PARK_URL, COUNTY, ACRES, LABEL_NAME, YR_EST, DES_TP, GAP_STS, geometry from CPAD_2020a_Units where ""AGNCY_LEV"" = :p0 order by id limit 101 ``` But I actually wanted to run a `fetch()` against a version of that without the `order by id limit 101` bit (I wanted to figure out the `Extent()` of the `geometry` column) - so I need something like `datasette.table_sql_no_order_no_limit`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1566#issuecomment-997078812,https://api.github.com/repos/simonw/datasette/issues/1566,997078812,IC_kwDOBm6k_c47bjcc,9599,simonw,2021-12-17T22:58:55Z,2021-12-17T22:58:55Z,OWNER,The release notes for the 0.60a0 alpha will be useful here: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/pull/1562#issuecomment-997081673,https://api.github.com/repos/simonw/datasette/issues/1562,997081673,IC_kwDOBm6k_c47bkJJ,9599,simonw,2021-12-17T23:06:38Z,2021-12-17T23:06:38Z,OWNER,"From this diff between `0.7.0` and `1.0`: https://github.com/aio-libs/janus/compare/v0.7.0...v1.0.0 It looks like the only change relevant to compatibility is `loop = asyncio.get_running_loop()` directly instead of falling back to `asyncio.get_event_loop()` if `get_running_loop` isn't available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/pull/1562#issuecomment-997082189,https://api.github.com/repos/simonw/datasette/issues/1562,997082189,IC_kwDOBm6k_c47bkRN,9599,simonw,2021-12-17T23:08:14Z,2021-12-17T23:08:14Z,OWNER,"Oh that makes sense: In Python 3.6 this happens: ``` Collecting janus<1.1,>=0.6.2 Using cached janus-0.7.0-py3-none-any.whl (6.9 kB) ``` While in Python 3.7 or higher this happens: ``` Collecting janus<1.1,>=0.6.2 Downloading janus-1.0.0-py3-none-any.whl (6.9 kB) ``` So this is safe to apply because `pip` is smart enough to pick the version of Janus that works for that Python version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083246400,"Update janus requirement from <0.8,>=0.6.2 to >=0.6.2,<1.1", https://github.com/simonw/datasette/pull/1559#issuecomment-997082676,https://api.github.com/repos/simonw/datasette/issues/1559,997082676,IC_kwDOBm6k_c47bkY0,9599,simonw,2021-12-17T23:09:41Z,2021-12-17T23:09:41Z,OWNER,This is now available to try out in Datasette 0.60a0: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082743068,"filters_from_request plugin hook, now used in TableView", https://github.com/simonw/datasette/issues/1518#issuecomment-997082845,https://api.github.com/repos/simonw/datasette/issues/1518,997082845,IC_kwDOBm6k_c47bkbd,9599,simonw,2021-12-17T23:10:09Z,2021-12-17T23:10:17Z,OWNER,These changes so far are now in the 0.60a0 alpha: https://github.com/simonw/datasette/releases/tag/0.60a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/621#issuecomment-997120723,https://api.github.com/repos/simonw/datasette/issues/621,997120723,IC_kwDOBm6k_c47btrT,9599,simonw,2021-12-18T01:42:33Z,2021-12-18T01:42:33Z,OWNER,I refactored this code out into the `filters.py` module in aa7f0037a46eb76ae6fe9bf2a1f616c58738ecdf,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520681725,Syntax for ?_through= that works as a form field, https://github.com/simonw/datasette/issues/1565#issuecomment-997121215,https://api.github.com/repos/simonw/datasette/issues/1565,997121215,IC_kwDOBm6k_c47bty_,9599,simonw,2021-12-18T01:45:44Z,2021-12-18T01:45:44Z,OWNER,I want to get this into Datasette 0.60 - #1566 - it's a small change that can unlock a lot of potential.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1564#issuecomment-997122938,https://api.github.com/repos/simonw/datasette/issues/1564,997122938,IC_kwDOBm6k_c47buN6,9599,simonw,2021-12-18T01:55:25Z,2021-12-18T01:55:46Z,OWNER,"Made this change while working on this issue: - #1567 I'm going to write a test for this that uses that `sleep()` SQL function from c35b84a2aabe2f14aeacf6cda4110ae1e94d6059.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083581011,_prepare_connection not called on write connections, https://github.com/simonw/datasette/issues/1546#issuecomment-997124280,https://api.github.com/repos/simonw/datasette/issues/1546,997124280,IC_kwDOBm6k_c47bui4,9599,simonw,2021-12-18T02:05:16Z,2021-12-18T02:05:16Z,OWNER,"Sure - there are actually several levels to this. The code that creates connections to the database is this: https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/database.py#L72-L95 For files on disk, it does this: ```python # For read-only connections conn = sqlite3.connect( ""file:my.db?mode=ro"", uri=True, check_same_thread=False) # For connections that should be treated as immutable: conn = sqlite3.connect( ""file:my.db?immutable=1"", uri=True, check_same_thread=False) ``` For in-memory databases it runs this after the connection has been created: ```python conn.execute(""PRAGMA query_only=1"") ``` SQLite `PRAGMA` queries are treated as dangerous: someone could run `PRAGMA query_only=0` to turn that previous option off for example. So this function runs against any incoming SQL to verify that it looks like a `SELECT ...` and doesn't have anything like that in it. https://github.com/simonw/datasette/blob/83bacfa9452babe7bd66e3579e23af988d00f6ac/datasette/utils/__init__.py#L195-L204 You can see the tests for that here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/tests/test_utils.py#L136-L170","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1076057610,validating the sql, https://github.com/simonw/datasette/issues/1563#issuecomment-997125191,https://api.github.com/repos/simonw/datasette/issues/1563,997125191,IC_kwDOBm6k_c47buxH,9599,simonw,2021-12-18T02:10:20Z,2021-12-18T02:10:20Z,OWNER,I should document the usage of this constructor in https://docs.datasette.io/en/stable/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083573206,Datasette(... files=) should not be a required argument, https://github.com/simonw/datasette/issues/1563#issuecomment-997127084,https://api.github.com/repos/simonw/datasette/issues/1563,997127084,IC_kwDOBm6k_c47bvOs,9599,simonw,2021-12-18T02:22:30Z,2021-12-18T02:22:30Z,OWNER,Docs here: https://docs.datasette.io/en/latest/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083573206,Datasette(... files=) should not be a required argument, https://github.com/simonw/datasette/issues/1561#issuecomment-997127784,https://api.github.com/repos/simonw/datasette/issues/1561,997127784,IC_kwDOBm6k_c47bvZo,9599,simonw,2021-12-18T02:27:56Z,2021-12-18T02:27:56Z,OWNER,"Oh that's an interesting solution, combining the hashes of all of the individual databases. I'm actually not a big fan of `hashed_url` mode - I implemented it right at the start of the project because it felt like a clever hack, and then ended up making it not-the-default a few years ago: - #418 - #419 - #421 I've since not found myself wanting to use it at all for any of my projects - which makes me nervous, because it means there's a pretty complex feature that I'm not using at all, so it's only really protected by the existing unit tests for it. What I'd really like to do is figure out how to have hashed URL mode work entirely as a plugin - then I could extract it from Datasette core entirely (which would simplify a bunch of stuff) but people who find the optimization useful would be able to access it. I'm not sure that the existing plugin hooks are robust enough to do that yet though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082765654,"add hash id to ""_memory"" url if hashed url mode is turned on and crossdb is also turned on", https://github.com/simonw/datasette/issues/1555#issuecomment-997128080,https://api.github.com/repos/simonw/datasette/issues/1555,997128080,IC_kwDOBm6k_c47bveQ,9599,simonw,2021-12-18T02:30:19Z,2021-12-18T02:30:19Z,OWNER,"I think all of these queries happen in one place - in the `populate_schema_tables()` function - so optimizing them might be localized to just that area of the code, which would be nice: https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L97-L183","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997128251,https://api.github.com/repos/simonw/datasette/issues/1555,997128251,IC_kwDOBm6k_c47bvg7,9599,simonw,2021-12-18T02:31:51Z,2021-12-18T02:31:51Z,OWNER,"I was thinking it might even be possible to convert this into a `insert into tables select from ...` query: https://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L102-L112 But the `SELECT` runs against a separate database from the `INSERT INTO`, so I would have to setup a cross-database connection for this which feels a little too complicated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997128368,https://api.github.com/repos/simonw/datasette/issues/1555,997128368,IC_kwDOBm6k_c47bviw,9599,simonw,2021-12-18T02:32:43Z,2021-12-18T02:32:43Z,OWNER,I wonder why the `INSERT INTO` queries don't show up in that `?trace=1` view?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997128508,https://api.github.com/repos/simonw/datasette/issues/1555,997128508,IC_kwDOBm6k_c47bvk8,9599,simonw,2021-12-18T02:33:57Z,2021-12-18T02:33:57Z,OWNER,"Here's why - `trace` only applies to read, not write SQL operations: https://github.com/simonw/datasette/blob/7c8f8aa209e4ba7bf83976f8495d67c28fbfca24/datasette/database.py#L209-L211","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1561#issuecomment-997128712,https://api.github.com/repos/simonw/datasette/issues/1561,997128712,IC_kwDOBm6k_c47bvoI,536941,fgregg,2021-12-18T02:35:48Z,2021-12-18T02:35:48Z,CONTRIBUTOR,interesting! i love this feature. this + full caching with cloudflare is really super!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082765654,"add hash id to ""_memory"" url if hashed url mode is turned on and crossdb is also turned on", https://github.com/simonw/datasette/issues/1568#issuecomment-997128950,https://api.github.com/repos/simonw/datasette/issues/1568,997128950,IC_kwDOBm6k_c47bvr2,9599,simonw,2021-12-18T02:38:01Z,2021-12-18T02:38:01Z,OWNER,"Prototype: ```diff diff --git a/datasette/database.py b/datasette/database.py index 0a0c104..468e936 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -99,7 +99,9 @@ class Database: with conn: return conn.execute(sql, params or []) - return await self.execute_write_fn(_inner, block=block) + with trace(""sql"", database=self.name, sql=sql.strip(), params=params): + results = await self.execute_write_fn(_inner, block=block) + return results async def execute_write_fn(self, fn, block=False): task_id = uuid.uuid5(uuid.NAMESPACE_DNS, ""datasette.io"") ``` <img width=""1292"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/146626249-cc8609f6-590c-49e3-abab-0cbc132916d1.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083726550,Trace should show queries on the write connection too, https://github.com/simonw/datasette/issues/1555#issuecomment-997234858,https://api.github.com/repos/simonw/datasette/issues/1555,997234858,IC_kwDOBm6k_c47cJiq,9599,simonw,2021-12-18T17:28:44Z,2021-12-18T17:28:44Z,OWNER,"Maybe it would be worth exploring attaching each DB in turn to the _internal connection in order to perform these queries faster. I'm a bit worried about leaks though: the internal database isn't meant to be visible, even temporarily attaching another DB to it could cause SQL queries against that DB to be able to access the internal data. So maybe instead the _internal connection gets to connect to the other DBs? There's a maximum of ten there I think, which is good for most but not all cases. But the cases with the most connected databases will see the worst performance!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997235086,https://api.github.com/repos/simonw/datasette/issues/1555,997235086,IC_kwDOBm6k_c47cJmO,9599,simonw,2021-12-18T17:30:13Z,2021-12-18T17:30:13Z,OWNER,"Now that trace sees write queries (#1568) it's clear that there is a whole lot more DB activity then I had realized: <img width=""1292"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/146626249-cc8609f6-590c-49e3-abab-0cbc132916d1.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1566#issuecomment-997235388,https://api.github.com/repos/simonw/datasette/issues/1566,997235388,IC_kwDOBm6k_c47cJq8,9599,simonw,2021-12-18T17:32:07Z,2021-12-18T17:32:07Z,OWNER,I can release a new version of `datasette-leaflet-freedraw` as soon as this is out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1555#issuecomment-997241645,https://api.github.com/repos/simonw/datasette/issues/1555,997241645,IC_kwDOBm6k_c47cLMt,9599,simonw,2021-12-18T18:12:26Z,2021-12-18T18:12:26Z,OWNER,"A simpler optimization would be just to turn all of those column and index reads into a single efficient UNION query against each database, then figure out the most efficient pattern to send them all as writes in one go as opposed to calling `.execute_write()` in a loop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997241969,https://api.github.com/repos/simonw/datasette/issues/1555,997241969,IC_kwDOBm6k_c47cLRx,9599,simonw,2021-12-18T18:13:04Z,2021-12-18T18:13:04Z,OWNER,Also: running all of those `CREATE TABLE IF NOT EXISTS` in a single call to `conn.executescript()` rather than as separate queries may speed things up too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997248364,https://api.github.com/repos/simonw/datasette/issues/1555,997248364,IC_kwDOBm6k_c47cM1s,9599,simonw,2021-12-18T18:20:10Z,2021-12-18T18:20:10Z,OWNER,"Idea: teach `execute_write` to accept an optional `executescript=True` parameter, like this: ```diff diff --git a/datasette/database.py b/datasette/database.py index 468e936..1a424f5 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -94,10 +94,14 @@ class Database: f""file:{self.path}{qs}"", uri=True, check_same_thread=False ) - async def execute_write(self, sql, params=None, block=False): + async def execute_write(self, sql, params=None, executescript=False, block=False): + assert not executescript and params, ""Cannot use params with executescript=True"" def _inner(conn): with conn: - return conn.execute(sql, params or []) + if executescript: + return conn.executescript(sql) + else: + return conn.execute(sql, params or []) with trace(""sql"", database=self.name, sql=sql.strip(), params=params): results = await self.execute_write_fn(_inner, block=block) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997245301,https://api.github.com/repos/simonw/datasette/issues/1555,997245301,IC_kwDOBm6k_c47cMF1,9599,simonw,2021-12-18T18:17:04Z,2021-12-18T18:17:04Z,OWNER,"One downside of `conn.executescript()` is that it won't be picked up by the tracing mechanism - in fact nothing that uses `await db.execute_write_fn(fn, block=True)` or `await db.execute_fn(fn, block=True)` gets picked up by tracing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1569#issuecomment-997249563,https://api.github.com/repos/simonw/datasette/issues/1569,997249563,IC_kwDOBm6k_c47cNIb,9599,simonw,2021-12-18T18:21:23Z,2021-12-18T18:21:23Z,OWNER,"Goal here is to gain the ability to use `conn.executescript()` and still have it show up in the tracer. https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.executescript","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083895395,"db.execute_write(..., executescript=True) parameter", https://github.com/simonw/datasette/issues/1555#issuecomment-997262475,https://api.github.com/repos/simonw/datasette/issues/1555,997262475,IC_kwDOBm6k_c47cQSL,9599,simonw,2021-12-18T18:34:18Z,2021-12-18T18:34:18Z,OWNER,"<img width=""1055"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/146652142-1c0bc34e-4a18-407d-bd59-28d565b631a6.png""> Using `executescript=True` that call now takes 1.89ms to create all of those tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997266100,https://api.github.com/repos/simonw/datasette/issues/1555,997266100,IC_kwDOBm6k_c47cRK0,9599,simonw,2021-12-18T18:40:02Z,2021-12-18T18:40:02Z,OWNER,The implementation of `cursor.executemany()` looks very efficient - it turns into a call to this C function with `multiple` set to `1`: https://github.com/python/cpython/blob/e002bbc6cce637171fb2b1391ffeca8643a13843/Modules/_sqlite/cursor.c#L468-L469,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1569#issuecomment-997266687,https://api.github.com/repos/simonw/datasette/issues/1569,997266687,IC_kwDOBm6k_c47cRT_,9599,simonw,2021-12-18T18:41:40Z,2021-12-18T18:41:40Z,OWNER,Updated documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-executescript-false-block-false,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083895395,"db.execute_write(..., executescript=True) parameter", https://github.com/simonw/datasette/issues/1555#issuecomment-997267416,https://api.github.com/repos/simonw/datasette/issues/1555,997267416,IC_kwDOBm6k_c47cRfY,9599,simonw,2021-12-18T18:44:53Z,2021-12-18T18:45:28Z,OWNER,"Rather than adding a `executemany=True` parameter, I'm now thinking a better design might be to have three methods: - `db.execute_write(sql, params=None, block=False)` - `db.execute_writescript(sql, block=False)` - `db.execute_writemany(sql, params_seq, block=False)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1570#issuecomment-997267583,https://api.github.com/repos/simonw/datasette/issues/1570,997267583,IC_kwDOBm6k_c47cRh_,9599,simonw,2021-12-18T18:46:05Z,2021-12-18T18:46:12Z,OWNER,This will replace the work done in #1569.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083921371,Separate db.execute_write() into three methods, https://github.com/simonw/datasette/issues/1555#issuecomment-997272223,https://api.github.com/repos/simonw/datasette/issues/1555,997272223,IC_kwDOBm6k_c47cSqf,9599,simonw,2021-12-18T19:17:13Z,2021-12-18T19:17:13Z,OWNER,That's a good optimization. Still need to deal with the huge flurry of `PRAGMA` queries though before I can consider this done.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1566#issuecomment-997272328,https://api.github.com/repos/simonw/datasette/issues/1566,997272328,IC_kwDOBm6k_c47cSsI,9599,simonw,2021-12-18T19:18:01Z,2021-12-18T19:18:01Z,OWNER,"Added some useful new documented internal methods in: - #1570","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1566#issuecomment-997457790,https://api.github.com/repos/simonw/datasette/issues/1566,997457790,IC_kwDOBm6k_c47c_9-,9599,simonw,2021-12-19T20:40:50Z,2021-12-19T20:40:57Z,OWNER,"Also release new version of `datasette-pretty-traces` with this feature: - https://github.com/simonw/datasette-pretty-traces/issues/7","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1555#issuecomment-997320824,https://api.github.com/repos/simonw/datasette/issues/1555,997320824,IC_kwDOBm6k_c47ceh4,9599,simonw,2021-12-19T02:59:57Z,2021-12-19T03:00:44Z,OWNER,"To list all indexes: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++index_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_index_list%28sqlite_master.name%29+AS+index_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, index_list.* FROM sqlite_master, pragma_index_list(sqlite_master.name) AS index_list WHERE sqlite_master.type = 'table' ``` Foreign keys: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++foreign_key_list.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_foreign_key_list%28sqlite_master.name%29+AS+foreign_key_list%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, foreign_key_list.* FROM sqlite_master, pragma_foreign_key_list(sqlite_master.name) AS foreign_key_list WHERE sqlite_master.type = 'table' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321115,https://api.github.com/repos/simonw/datasette/issues/1555,997321115,IC_kwDOBm6k_c47cemb,9599,simonw,2021-12-19T03:03:12Z,2021-12-19T03:03:12Z,OWNER,"Table columns is a bit harder, because `table_xinfo` is only in SQLite 3.26.0 or higher: https://github.com/simonw/datasette/blob/d637ed46762fdbbd8e32b86f258cd9a53c1cfdc7/datasette/utils/__init__.py#L565-L581 So if that function is available: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_xinfo.*%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_xinfo%28sqlite_master.name%29+AS+table_xinfo%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, table_xinfo.* FROM sqlite_master, pragma_table_xinfo(sqlite_master.name) AS table_xinfo WHERE sqlite_master.type = 'table' ``` And otherwise, using `table_info`: https://latest.datasette.io/fixtures?sql=SELECT%0D%0A++sqlite_master.name%2C%0D%0A++table_info.*%2C%0D%0A++0+as+hidden%0D%0AFROM%0D%0A++sqlite_master%2C%0D%0A++pragma_table_info%28sqlite_master.name%29+AS+table_info%0D%0AWHERE%0D%0A++sqlite_master.type+%3D+%27table%27 ```sql SELECT sqlite_master.name, table_info.*, 0 as hidden FROM sqlite_master, pragma_table_info(sqlite_master.name) AS table_info WHERE sqlite_master.type = 'table' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321217,https://api.github.com/repos/simonw/datasette/issues/1555,997321217,IC_kwDOBm6k_c47ceoB,9599,simonw,2021-12-19T03:04:16Z,2021-12-19T03:04:16Z,OWNER,"One thing to watch out for though, from https://sqlite.org/pragma.html#pragfunc > The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321327,https://api.github.com/repos/simonw/datasette/issues/1555,997321327,IC_kwDOBm6k_c47cepv,9599,simonw,2021-12-19T03:05:39Z,2021-12-19T03:05:44Z,OWNER,"This caught me out once before in: - https://github.com/simonw/datasette/issues/1276 Turns out Glitch was running SQLite 3.11.0 from 2016-02-15.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321477,https://api.github.com/repos/simonw/datasette/issues/1555,997321477,IC_kwDOBm6k_c47cesF,9599,simonw,2021-12-19T03:07:33Z,2021-12-19T03:07:33Z,OWNER,"If I want to continue supporting SQLite prior to 3.16.0 (2017-01-02) I'll need this optimization to only kick in with versions that support table-valued PRAGMA functions, while keeping the old `PRAGMA foreign_key_list(table)` stuff working for those older versions. That's feasible, but it's a bit more work - and I need to make sure I have robust testing in place for SQLite 3.15.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321653,https://api.github.com/repos/simonw/datasette/issues/1555,997321653,IC_kwDOBm6k_c47ceu1,9599,simonw,2021-12-19T03:09:43Z,2021-12-19T03:09:43Z,OWNER,"On that same documentation page I just spotted this: > This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported. This makes me nervous to rely on pragma function optimizations in Datasette itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997321767,https://api.github.com/repos/simonw/datasette/issues/1555,997321767,IC_kwDOBm6k_c47cewn,9599,simonw,2021-12-19T03:10:58Z,2021-12-19T03:10:58Z,OWNER,"I wonder how much overhead there is switching between the `async` event loop main code and the thread that runs the SQL queries. Would there be a performance boost if I gathered all of the column/index information in a single function run on the thread using `db.execute_fn()` I wonder? It would eliminate a bunch of switching between threads. Would be great to understand how much of an impact that would have.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997324156,https://api.github.com/repos/simonw/datasette/issues/1555,997324156,IC_kwDOBm6k_c47cfV8,9599,simonw,2021-12-19T03:40:05Z,2021-12-19T03:40:05Z,OWNER,"Using the prototype of this: - https://github.com/simonw/datasette-pretty-traces/issues/5 I'm seeing about 180ms spent running all of these queries on startup! ![CleanShot 2021-12-18 at 19 38 37@2x](https://user-images.githubusercontent.com/9599/146663045-46bda669-90de-474f-8870-345182725dc1.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997324666,https://api.github.com/repos/simonw/datasette/issues/1555,997324666,IC_kwDOBm6k_c47cfd6,9599,simonw,2021-12-19T03:47:51Z,2021-12-19T03:48:09Z,OWNER,"Here's a hacked together prototype of running all of that stuff inside a single function passed to `.execute_fn()`: ```diff diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py index 95055d8..58f9982 100644 --- a/datasette/utils/internal_db.py +++ b/datasette/utils/internal_db.py @@ -1,4 +1,5 @@ import textwrap +from datasette.utils import table_column_details async def init_internal_db(db): @@ -70,49 +71,70 @@ async def populate_schema_tables(internal_db, db): ""DELETE FROM tables WHERE database_name = ?"", [database_name], block=True ) tables = (await db.execute(""select * from sqlite_master WHERE type = 'table'"")).rows - tables_to_insert = [] - columns_to_delete = [] - columns_to_insert = [] - foreign_keys_to_delete = [] - foreign_keys_to_insert = [] - indexes_to_delete = [] - indexes_to_insert = [] - for table in tables: - table_name = table[""name""] - tables_to_insert.append( - (database_name, table_name, table[""rootpage""], table[""sql""]) - ) - columns_to_delete.append((database_name, table_name)) - columns = await db.table_column_details(table_name) - columns_to_insert.extend( - { - **{""database_name"": database_name, ""table_name"": table_name}, - **column._asdict(), - } - for column in columns - ) - foreign_keys_to_delete.append((database_name, table_name)) - foreign_keys = ( - await db.execute(f""PRAGMA foreign_key_list([{table_name}])"") - ).rows - foreign_keys_to_insert.extend( - { - **{""database_name"": database_name, ""table_name"": table_name}, - **dict(foreign_key), - } - for foreign_key in foreign_keys - ) - indexes_to_delete.append((database_name, table_name)) - indexes = (await db.execute(f""PRAGMA index_list([{table_name}])"")).rows - indexes_to_insert.extend( - { - **{""database_name"": database_name, ""table_name"": table_name}, - **dict(index), - } - for index in indexes + def collect_info(conn): + tables_to_insert = [] + columns_to_delete = [] + columns_to_insert = [] + foreign_keys_to_delete = [] + foreign_keys_to_insert = [] + indexes_to_delete = [] + indexes_to_insert = [] + + for table in tables: + table_name = table[""name""] + tables_to_insert.append( + (database_name, table_name, table[""rootpage""], table[""sql""]) + ) + columns_to_delete.append((database_name, table_name)) + columns = table_column_details(conn, table_name) + columns_to_insert.extend( + { + **{""database_name"": database_name, ""table_name"": table_name}, + **column._asdict(), + } + for column in columns + ) + foreign_keys_to_delete.append((database_name, table_name)) + foreign_keys = conn.execute( + f""PRAGMA foreign_key_list([{table_name}])"" + ).fetchall() + foreign_keys_to_insert.extend( + { + **{""database_name"": database_name, ""table_name"": table_name}, + **dict(foreign_key), + } + for foreign_key in foreign_keys + ) + indexes_to_delete.append((database_name, table_name)) + indexes = conn.execute(f""PRAGMA index_list([{table_name}])"").fetchall() + indexes_to_insert.extend( + { + **{""database_name"": database_name, ""table_name"": table_name}, + **dict(index), + } + for index in indexes + ) + return ( + tables_to_insert, + columns_to_delete, + columns_to_insert, + foreign_keys_to_delete, + foreign_keys_to_insert, + indexes_to_delete, + indexes_to_insert, ) + ( + tables_to_insert, + columns_to_delete, + columns_to_insert, + foreign_keys_to_delete, + foreign_keys_to_insert, + indexes_to_delete, + indexes_to_insert, + ) = await db.execute_fn(collect_info) + await internal_db.execute_write_many( """""" INSERT INTO tables (database_name, table_name, rootpage, sql) ``` First impressions: it looks like this helps **a lot** - as far as I can tell this is now taking around 21ms to get to the point at which all of those internal databases have been populated, where previously it took more than 180ms. ![CleanShot 2021-12-18 at 19 47 22@2x](https://user-images.githubusercontent.com/9599/146663192-bba098d5-e7bd-4e2e-b525-2270867888a0.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997325189,https://api.github.com/repos/simonw/datasette/issues/1555,997325189,IC_kwDOBm6k_c47cfmF,9599,simonw,2021-12-19T03:55:01Z,2021-12-19T20:54:51Z,OWNER,"It's a bit annoying that the queries no longer show up in the trace at all now, thanks to running in `.execute_fn()`. I wonder if there's something smart I can do about that - maybe have `trace()` record that function with a traceback even though it doesn't have the executed SQL string? 5fac26aa221a111d7633f2dd92014641f7c0ade9 has the same problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997342494,https://api.github.com/repos/simonw/datasette/issues/1555,997342494,IC_kwDOBm6k_c47cj0e,9599,simonw,2021-12-19T07:22:04Z,2021-12-19T07:22:04Z,OWNER,"Another option would be to provide an abstraction that makes it easier to run a group of SQL queries in the same thread at the same time, and have them traced correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997459637,https://api.github.com/repos/simonw/datasette/issues/1555,997459637,IC_kwDOBm6k_c47dAa1,9599,simonw,2021-12-19T20:53:46Z,2021-12-19T20:53:46Z,OWNER,Using #1571 showed me that the `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ? and table_name = ?` queries were running way more times than I expected. I came up with a new optimization that just does `DELETE FROM columns/foreign_keys/indexes WHERE database_name = ?` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1555#issuecomment-997459958,https://api.github.com/repos/simonw/datasette/issues/1555,997459958,IC_kwDOBm6k_c47dAf2,9599,simonw,2021-12-19T20:55:59Z,2021-12-19T20:55:59Z,OWNER,"Closing this issue because I've optimized this a whole bunch, and it's definitely good enough for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079149656,Optimize all those calls to index_list and foreign_key_list, https://github.com/simonw/datasette/issues/1570#issuecomment-997460061,https://api.github.com/repos/simonw/datasette/issues/1570,997460061,IC_kwDOBm6k_c47dAhd,9599,simonw,2021-12-19T20:56:54Z,2021-12-19T20:56:54Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#await-db-execute-write-sql-params-none-block-false,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083921371,Separate db.execute_write() into three methods, https://github.com/simonw/datasette/issues/1547#issuecomment-997460731,https://api.github.com/repos/simonw/datasette/issues/1547,997460731,IC_kwDOBm6k_c47dAr7,9599,simonw,2021-12-19T21:02:15Z,2021-12-19T21:02:15Z,OWNER,"Yes, this is a bug. It looks like the problem is with the `if write:` branch in this code here: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L252-L327 Is missing this bit of code: https://github.com/simonw/datasette/blob/5fac26aa221a111d7633f2dd92014641f7c0ade9/datasette/views/database.py#L343-L347","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1573#issuecomment-997462117,https://api.github.com/repos/simonw/datasette/issues/1573,997462117,IC_kwDOBm6k_c47dBBl,9599,simonw,2021-12-19T21:13:13Z,2021-12-19T21:13:13Z,OWNER,This might also be the impetus I need to bring the https://datasette.io/plugins/datasette-pretty-traces plugin into Datasette core itself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084185188,Make trace() a documented internal API, https://github.com/simonw/datasette/issues/1545#issuecomment-997462604,https://api.github.com/repos/simonw/datasette/issues/1545,997462604,IC_kwDOBm6k_c47dBJM,9599,simonw,2021-12-19T21:17:08Z,2021-12-19T21:17:08Z,OWNER,"Here's the relevant code: https://github.com/simonw/datasette/blob/4094741c2881c2ada3f3f878b532fdaec7914953/datasette/app.py#L1204-L1219 It's using `route_path.split(""/"")` which should be OK because that's the incoming `request.path` path - which I would expect to use `/` even on Windows. Then it uses `os.path.join` which should do the right thing. I need to get myself a proper Windows development environment setup to investigate this one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249,Custom pages don't work on windows, https://github.com/simonw/datasette/issues/1565#issuecomment-997473856,https://api.github.com/repos/simonw/datasette/issues/1565,997473856,IC_kwDOBm6k_c47dD5A,9599,simonw,2021-12-19T22:35:20Z,2021-12-19T22:35:20Z,OWNER,"Quick prototype of that tagged template `query` function: ```javascript function query(pieces, ...parameters) { var qs = new URLSearchParams(); var sql = pieces[0]; parameters.forEach((param, i) => { sql += `:p${i}${pieces[i + 1]}`; qs.append(`p${i}`, param); }); qs.append(""sql"", sql); return qs.toString(); } var id = 4; console.log(query`select * from ids where id > ${id}`); ``` Outputs: ``` p0=4&sql=select+*+from+ids+where+id+%3E+%3Ap0 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997474022,https://api.github.com/repos/simonw/datasette/issues/1565,997474022,IC_kwDOBm6k_c47dD7m,9599,simonw,2021-12-19T22:36:49Z,2021-12-19T22:37:29Z,OWNER,"No way with a tagged template literal to pass an extra database name argument, so instead I need a method that returns a callable that can be used for the tagged template literal for a specific database - or the default database. This could work (bit weird looking though): ```javascript var rows = await datasette.query(""fixtures"")`select * from foo`; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1566#issuecomment-997470633,https://api.github.com/repos/simonw/datasette/issues/1566,997470633,IC_kwDOBm6k_c47dDGp,9599,simonw,2021-12-19T22:12:00Z,2021-12-19T22:12:00Z,OWNER,"Released another alpha, 0.60a1: https://github.com/simonw/datasette/releases/tag/0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1547#issuecomment-997471672,https://api.github.com/repos/simonw/datasette/issues/1547,997471672,IC_kwDOBm6k_c47dDW4,9599,simonw,2021-12-19T22:18:26Z,2021-12-19T22:18:26Z,OWNER,"I released this [in an alpha](https://github.com/simonw/datasette/releases/tag/0.60a1), so you can try out this fix using: pip install datasette==0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1518#issuecomment-997472214,https://api.github.com/repos/simonw/datasette/issues/1518,997472214,IC_kwDOBm6k_c47dDfW,9599,simonw,2021-12-19T22:22:08Z,2021-12-19T22:22:08Z,OWNER,"I sketched out a chained SQL builder pattern that might be useful for further tidying up this code - though with the new plugin hook I'm less excited about it than I was: ```python class TableQuery: def __init__(self, table, columns, pks, is_view=False, prev=None): self.table = table self.columns = columns self.pks = pks self.is_view = is_view self.prev = prev # These can be changed for different instances in the chain: self._where_clauses = None self._order_by = None self._page_size = None self._offset = None self._select_columns = None self.select_all_columns = '*' self.select_specified_columns = '*' @property def where_clauses(self): wheres = [] current = self while current: if current._where_clauses is not None: wheres.extend(current._where_clauses) current = current.prev return list(reversed(wheres)) def where(self, where): new_cls = TableQuery(self.table, self.columns, self.pks, self.is_view, self) new_cls._where_clauses = [where] return new_cls @classmethod async def introspect(cls, db, table): return cls( table, columns = await db.table_columns(table), pks = await db.primary_keys(table), is_view = bool(await db.get_view_definition(table)) ) @property def sql_from(self): return f""from {self.table}{self.sql_where}"" @property def sql_where(self): if not self.where_clauses: return """" else: return f"" where {' and '.join(self.where_clauses)}"" @property def sql_no_order_no_limit(self): return f""select {self.select_all_columns} from {self.table}{self.sql_where}"" @property def sql(self): return f""select {self.select_specified_columns} from {self.table} {self.sql_where}{self._order_by} limit {self._page_size}{self._offset}"" @property def sql_count(self): return f""select count(*) {self.sql_from}"" def __repr__(self): return f""<TableQuery sql={self.sql}>"" ``` Usage: ```python from datasette.app import Datasette ds = Datasette(memory=True, files=[""/Users/simon/Dropbox/Development/datasette/fixtures.db""]) db = ds.get_database(""fixtures"") query = await TableQuery.introspect(db, ""facetable"") print(query.where(""foo = bar"").where(""baz = 1"").sql_count) # 'select count(*) from facetable where foo = bar and baz = 1' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1565#issuecomment-997472370,https://api.github.com/repos/simonw/datasette/issues/1565,997472370,IC_kwDOBm6k_c47dDhy,9599,simonw,2021-12-19T22:23:36Z,2021-12-19T22:23:36Z,OWNER,This should also expose the JSON API endpoints used to execute SQL against this database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997472509,https://api.github.com/repos/simonw/datasette/issues/1565,997472509,IC_kwDOBm6k_c47dDj9,9599,simonw,2021-12-19T22:24:50Z,2021-12-19T22:24:50Z,OWNER,"... huh, it could even expose a JavaScript function that can be called to execute a SQL query. ```javascript datasette.query(""select * from blah"").then(...) ``` Maybe it takes an optional second argument that specifies the database - defaulting to the one for the current page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1565#issuecomment-997472639,https://api.github.com/repos/simonw/datasette/issues/1565,997472639,IC_kwDOBm6k_c47dDl_,9599,simonw,2021-12-19T22:25:50Z,2021-12-19T22:25:50Z,OWNER,"Or... ```javascript rows = await datasette.query`select * from searchable where id > ${id}`; ``` And it knows how to turn that into a parameterized call using tagged template literals.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083657868,Documented JavaScript variables on different templates made available for plugins, https://github.com/simonw/datasette/issues/1547#issuecomment-997511968,https://api.github.com/repos/simonw/datasette/issues/1547,997511968,IC_kwDOBm6k_c47dNMg,127565,wragge,2021-12-20T01:21:59Z,2021-12-20T01:21:59Z,CONTRIBUTOR,"I've installed the alpha version but get an error when starting up Datasette: ``` Traceback (most recent call last): File ""/Users/tim/.pyenv/versions/stock-exchange/bin/datasette"", line 5, in <module> from datasette.cli import cli File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/cli.py"", line 15, in <module> from .app import Datasette, DEFAULT_SETTINGS, SETTINGS, SQLITE_LIMIT_ATTACHED, pm File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/app.py"", line 31, in <module> from .views.database import DatabaseDownload, DatabaseView File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/views/database.py"", line 25, in <module> from datasette.plugins import pm File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/plugins.py"", line 29, in <module> mod = importlib.import_module(plugin) File ""/Users/tim/.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py"", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File ""/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/filters.py"", line 9, in <module> @hookimpl(specname=""filters_from_request"") TypeError: __call__() got an unexpected keyword argument 'specname' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1575#issuecomment-997513177,https://api.github.com/repos/simonw/datasette/issues/1575,997513177,IC_kwDOBm6k_c47dNfZ,9599,simonw,2021-12-20T01:24:25Z,2021-12-20T01:24:25Z,OWNER,Looks like `specname` is new in Pluggy 1.0: https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst#pluggy-100-2021-08-25,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084257842,__call__() got an unexpected keyword argument 'specname', https://github.com/simonw/datasette/issues/1547#issuecomment-997513369,https://api.github.com/repos/simonw/datasette/issues/1547,997513369,IC_kwDOBm6k_c47dNiZ,9599,simonw,2021-12-20T01:24:43Z,2021-12-20T01:24:43Z,OWNER,"@wragge thanks, that's a bug! Working on that in #1575.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1547#issuecomment-997514220,https://api.github.com/repos/simonw/datasette/issues/1547,997514220,IC_kwDOBm6k_c47dNvs,9599,simonw,2021-12-20T01:26:25Z,2021-12-20T01:26:25Z,OWNER,"OK, this should hopefully fix that for you: pip install https://github.com/simonw/datasette/archive/f36e010b3b69ada104b79d83c7685caf9359049e.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/issues/1547#issuecomment-997519202,https://api.github.com/repos/simonw/datasette/issues/1547,997519202,IC_kwDOBm6k_c47dO9i,127565,wragge,2021-12-20T01:36:58Z,2021-12-20T01:36:58Z,CONTRIBUTOR,"Yep, that works -- thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1076388044,Writable canned queries fail to load custom templates, https://github.com/simonw/datasette/pull/1554#issuecomment-998354538,https://api.github.com/repos/simonw/datasette/issues/1554,998354538,IC_kwDOBm6k_c47ga5q,9599,simonw,2021-12-20T23:52:04Z,2021-12-20T23:52:04Z,OWNER,Abandoning this since it didn't work how I wanted.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079129258,TableView refactor, https://github.com/simonw/datasette/issues/1181#issuecomment-998999230,https://api.github.com/repos/simonw/datasette/issues/1181,998999230,IC_kwDOBm6k_c47i4S-,9308268,rayvoelker,2021-12-21T18:25:15Z,2021-12-21T18:25:15Z,NONE,"I wonder if I'm encountering the same bug (or something related). I had previously been using the .csv feature to run queries and then fetch results for the pandas `read_csv()` function, but it seems to have stopped working recently. https://ilsweb.cincinnatilibrary.org/collection-analysis/collection-analysis/current_collection-3d56dbf.csv?sql=select%0D%0A++*%0D%0Afrom%0D%0A++bib%0D%0Alimit%0D%0A++100&_size=max Datasette v0.59.4 ![image](https://user-images.githubusercontent.com/9308268/146979957-66911877-2cd9-4022-bc76-fd54e4a3a6f7.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",781262510,"Certain database names results in 404: ""Database not found: None""", https://github.com/simonw/datasette/issues/1518#issuecomment-999831967,https://api.github.com/repos/simonw/datasette/issues/1518,999831967,IC_kwDOBm6k_c47mDmf,9599,simonw,2021-12-22T20:04:47Z,2021-12-22T20:10:11Z,OWNER,"I think I might be able to clean up a lot of the stuff in here using the `render_cell` plugin hook: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L87-L89 The catch with that hook - https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette - is that it gets called for every single cell. I don't want the overhead of looking up the foreign key relationships etc once for every value in a specific column. But maybe I could extend the hook to include a shared cache that gets used for all of the cells in a specific table? Something like this: ```python render_cell(value, column, table, database, datasette, cache) ``` `cache` is a dictionary - and the same dictionary is passed to every call to that hook while rendering a specific page. It's a bit of a gross hack though, and would it ever be useful for plugins outside of the default plugin in Datasette which does the foreign key stuff? If I can think of one other potential application for this `cache` then I might implement it. No, this optimization doesn't make sense: the most complex cell enrichment logic is the stuff that does a `select * from categories where id in (2, 5, 6)` query, using just the distinct set of IDs that are rendered on the current page. That's not going to fit in the `render_cell` hook no matter how hard I try to warp it into the right shape, because it needs full visibility of all of the results that are being rendered in order to collect those unique ID values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999837569,https://api.github.com/repos/simonw/datasette/issues/1518,999837569,IC_kwDOBm6k_c47mE-B,9599,simonw,2021-12-22T20:15:45Z,2021-12-22T20:15:45Z,OWNER,"Also the whole `special_args` v.s. `request.args` thing is pretty confusing, I think that might be an older code pattern back from when I was using Sanic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999837220,https://api.github.com/repos/simonw/datasette/issues/1518,999837220,IC_kwDOBm6k_c47mE4k,9599,simonw,2021-12-22T20:15:04Z,2021-12-22T20:15:04Z,OWNER,"I think I can move this much higher up in the method, it's a bit confusing having it half way through: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L414-L436","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999850191,https://api.github.com/repos/simonw/datasette/issues/1518,999850191,IC_kwDOBm6k_c47mIDP,9599,simonw,2021-12-22T20:29:38Z,2021-12-22T20:29:38Z,OWNER,New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999863269,https://api.github.com/repos/simonw/datasette/issues/1518,999863269,IC_kwDOBm6k_c47mLPl,9599,simonw,2021-12-22T20:35:41Z,2021-12-22T20:37:13Z,OWNER,"It looks like the count has to be executed before facets can be, because the facet_class constructor needs that total count figure: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L660-L671 It's used in facet suggestion logic here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/facets.py#L172-L178","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1518#issuecomment-999870282,https://api.github.com/repos/simonw/datasette/issues/1518,999870282,IC_kwDOBm6k_c47mM9K,9599,simonw,2021-12-22T20:45:56Z,2021-12-22T20:46:08Z,OWNER,"> New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`. I wrote code to execute those in parallel using `asyncio.gather()` - which seems to work but causes the SQL run inside the parallel `async def` functions not to show up in the trace graph at all. ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 9808fd2..ec9db64 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import urllib import itertools import json @@ -615,44 +616,37 @@ class TableView(RowTableShared): if request.args.get(""_timelimit""): extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit"")) - # Execute the main query! - results = await db.execute(sql, params, truncate=True, **extra_args) - - # Calculate the total count for this query - filtered_table_rows_count = None - if ( - not db.is_mutable - and self.ds.inspect_data - and count_sql == f""select count(*) from {table} "" - ): - # We can use a previously cached table row count - try: - filtered_table_rows_count = self.ds.inspect_data[database][""tables""][ - table - ][""count""] - except KeyError: - pass - - # Otherwise run a select count(*) ... - if count_sql and filtered_table_rows_count is None and not nocount: - try: - count_rows = list(await db.execute(count_sql, from_sql_params)) - filtered_table_rows_count = count_rows[0][0] - except QueryInterrupted: - pass - - # Faceting - if not self.ds.setting(""allow_facet"") and any( - arg.startswith(""_facet"") for arg in request.args - ): - raise BadRequest(""_facet= is not allowed"") + async def execute_count(): + # Calculate the total count for this query + filtered_table_rows_count = None + if ( + not db.is_mutable + and self.ds.inspect_data + and count_sql == f""select count(*) from {table} "" + ): + # We can use a previously cached table row count + try: + filtered_table_rows_count = self.ds.inspect_data[database][ + ""tables"" + ][table][""count""] + except KeyError: + pass + + if count_sql and filtered_table_rows_count is None and not nocount: + try: + count_rows = list(await db.execute(count_sql, from_sql_params)) + filtered_table_rows_count = count_rows[0][0] + except QueryInterrupted: + pass + + return filtered_table_rows_count + + filtered_table_rows_count = await execute_count() # pylint: disable=no-member facet_classes = list( itertools.chain.from_iterable(pm.hook.register_facet_classes()) ) - facet_results = {} - facets_timed_out = [] facet_instances = [] for klass in facet_classes: facet_instances.append( @@ -668,33 +662,58 @@ class TableView(RowTableShared): ) ) - if not nofacet: - for facet in facet_instances: - ( - instance_facet_results, - instance_facets_timed_out, - ) = await facet.facet_results() - for facet_info in instance_facet_results: - base_key = facet_info[""name""] - key = base_key - i = 1 - while key in facet_results: - i += 1 - key = f""{base_key}_{i}"" - facet_results[key] = facet_info - facets_timed_out.extend(instance_facets_timed_out) - - # Calculate suggested facets - suggested_facets = [] - if ( - self.ds.setting(""suggest_facets"") - and self.ds.setting(""allow_facet"") - and not _next - and not nofacet - and not nosuggest - ): - for facet in facet_instances: - suggested_facets.extend(await facet.suggest()) + async def execute_suggested_facets(): + # Calculate suggested facets + suggested_facets = [] + if ( + self.ds.setting(""suggest_facets"") + and self.ds.setting(""allow_facet"") + and not _next + and not nofacet + and not nosuggest + ): + for facet in facet_instances: + suggested_facets.extend(await facet.suggest()) + return suggested_facets + + async def execute_facets(): + facet_results = {} + facets_timed_out = [] + if not self.ds.setting(""allow_facet"") and any( + arg.startswith(""_facet"") for arg in request.args + ): + raise BadRequest(""_facet= is not allowed"") + + if not nofacet: + for facet in facet_instances: + ( + instance_facet_results, + instance_facets_timed_out, + ) = await facet.facet_results() + for facet_info in instance_facet_results: + base_key = facet_info[""name""] + key = base_key + i = 1 + while key in facet_results: + i += 1 + key = f""{base_key}_{i}"" + facet_results[key] = facet_info + facets_timed_out.extend(instance_facets_timed_out) + + return facet_results, facets_timed_out + + # Execute the main query, facets and facet suggestions in parallel: + ( + results, + suggested_facets, + (facet_results, facets_timed_out), + ) = await asyncio.gather( + db.execute(sql, params, truncate=True, **extra_args), + execute_suggested_facets(), + execute_facets(), + ) + + results = await db.execute(sql, params, truncate=True, **extra_args) # Figure out columns and rows for the query columns = [r[0] for r in results.description] ``` Here's the trace for `http://127.0.0.1:4422/fixtures/compound_three_primary_keys?_trace=1&_facet=pk1&_facet=pk2` with the missing facet and facet suggestion queries: <img width=""1447"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/147153051-62cdb9a5-de5e-4cc3-9215-b779f92a81c8.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1576#issuecomment-999874484,https://api.github.com/repos/simonw/datasette/issues/1576,999874484,IC_kwDOBm6k_c47mN-0,9599,simonw,2021-12-22T20:54:52Z,2021-12-22T20:54:52Z,OWNER,"Here's the full current relevant code from `tracer.py`: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L8-L64 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1518#issuecomment-999870993,https://api.github.com/repos/simonw/datasette/issues/1518,999870993,IC_kwDOBm6k_c47mNIR,9599,simonw,2021-12-22T20:47:18Z,2021-12-22T20:50:24Z,OWNER,"The reason they aren't showing up in the traces is that traces are stored just for the currently executing `asyncio` task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25 This is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively ""child tasks"" of the current request, and hence should be tracked the same. https://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using `asyncio.gather()` to work around this kind of problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template, https://github.com/simonw/datasette/issues/1576#issuecomment-999874886,https://api.github.com/repos/simonw/datasette/issues/1576,999874886,IC_kwDOBm6k_c47mOFG,9599,simonw,2021-12-22T20:55:42Z,2021-12-22T20:57:28Z,OWNER,"One way to solve this would be to introduce a `set_task_id()` method, which sets an ID which will be returned by `get_task_id()` instead of using `id(current_task(loop=loop))`. It would be really nice if I could solve this using `with` syntax somehow. Something like: ```python with trace_child_tasks(): ( suggested_facets, (facet_results, facets_timed_out), ) = await asyncio.gather( execute_suggested_facets(), execute_facets(), ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999876666,https://api.github.com/repos/simonw/datasette/issues/1576,999876666,IC_kwDOBm6k_c47mOg6,9599,simonw,2021-12-22T20:59:22Z,2021-12-22T21:18:09Z,OWNER,"This article is relevant: [Context information storage for asyncio](https://blog.sqreen.com/asyncio/) - in particular the section https://blog.sqreen.com/asyncio/#context-inheritance-between-tasks which describes exactly the problem I have and their solution, which involves this trickery: ```python def request_task_factory(loop, coro): child_task = asyncio.tasks.Task(coro, loop=loop) parent_task = asyncio.Task.current_task(loop=loop) current_request = getattr(parent_task, 'current_request', None) setattr(child_task, 'current_request', current_request) return child_task loop = asyncio.get_event_loop() loop.set_task_factory(request_task_factory) ``` They released their solution as a library: https://pypi.org/project/aiocontext/ and https://github.com/sqreen/AioContext - but that company was acquired by Datadog back in April and doesn't seem to be actively maintaining their open source stuff any more: https://twitter.com/SqreenIO/status/1384906075506364417","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999878907,https://api.github.com/repos/simonw/datasette/issues/1576,999878907,IC_kwDOBm6k_c47mPD7,9599,simonw,2021-12-22T21:03:49Z,2021-12-22T21:10:46Z,OWNER,"`context_vars` can solve this but they were introduced in Python 3.7: https://www.python.org/dev/peps/pep-0567/ Python 3.6 support ends in a few days time, and it looks like Glitch has updated to 3.7 now - so maybe I can get away with Datasette needing 3.7 these days? Tweeted about that here: https://twitter.com/simonw/status/1473761478155010048","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999987418,https://api.github.com/repos/simonw/datasette/issues/1576,999987418,IC_kwDOBm6k_c47mpja,9599,simonw,2021-12-23T01:59:58Z,2021-12-23T02:02:12Z,OWNER,"Another option: https://github.com/Skyscanner/aiotask-context - looks like it might be better as it's been updated for Python 3.7 in this commit https://github.com/Skyscanner/aiotask-context/commit/67108c91d2abb445655cc2af446fdb52ca7890c4 The Skyscanner one doesn't attempt to wrap any existing factories, but that's OK for my purposes since I don't need to handle arbitrary `asyncio` code written by other people.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-999990414,https://api.github.com/repos/simonw/datasette/issues/1576,999990414,IC_kwDOBm6k_c47mqSO,9599,simonw,2021-12-23T02:08:39Z,2021-12-23T18:16:35Z,OWNER,"It's tiny: I'm tempted to vendor it. https://github.com/Skyscanner/aiotask-context/blob/master/aiotask_context/__init__.py No, I'll add it as a pinned dependency, which I can then drop when I drop 3.6 support.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1577#issuecomment-1000461275,https://api.github.com/repos/simonw/datasette/issues/1577,1000461275,IC_kwDOBm6k_c47odPb,9599,simonw,2021-12-23T18:18:11Z,2021-12-23T18:18:11Z,OWNER,"From the Twitter thread, there are still a decent amount of LTS Linux releases out there that are stuck on pre-3.7 Python. Though many of those are 3.5 and Datasette dropped support for 3.5 in November 2019: cf7776d36fbacefa874cbd6e5fcdc9fff7661203","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1577#issuecomment-1000461900,https://api.github.com/repos/simonw/datasette/issues/1577,1000461900,IC_kwDOBm6k_c47odZM,9599,simonw,2021-12-23T18:19:44Z,2021-12-23T18:19:44Z,OWNER,"The 3.7 feature I want to use today is [contextvars](https://docs.python.org/3/library/contextvars.html) - but I have a workaround for the moment, see https://github.com/simonw/datasette/issues/1576#issuecomment-999987418 So I'm going to hold off on dropping 3.6 for a little bit longer. I imagine I'll drop it before Datasette 1.0 though. Leaving this issue open to gather thoughts and feedback on this issue from Datasette users and potential users.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1577#issuecomment-1000462309,https://api.github.com/repos/simonw/datasette/issues/1577,1000462309,IC_kwDOBm6k_c47odfl,9599,simonw,2021-12-23T18:20:46Z,2021-12-23T18:20:46Z,OWNER,There are a lot of improvements to `asyncio` in 3.7: https://docs.python.org/3/whatsnew/3.7.html#whatsnew37-asyncio,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1578#issuecomment-1000469107,https://api.github.com/repos/simonw/datasette/issues/1578,1000469107,IC_kwDOBm6k_c47ofJz,9599,simonw,2021-12-23T18:36:38Z,2021-12-23T18:36:38Z,OWNER,"This problem doesn't occur on my `localhost` running Uvicorn directly - but I'm seeing it in my production environment that runs Datasette behind an nginx proxy: ``` location / { proxy_pass http://127.0.0.1:8000/; proxy_set_header Host $host; } ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1578#issuecomment-1000470652,https://api.github.com/repos/simonw/datasette/issues/1578,1000470652,IC_kwDOBm6k_c47ofh8,9599,simonw,2021-12-23T18:40:46Z,2021-12-23T18:40:46Z,OWNER,"[This StackOverflow answer](https://serverfault.com/a/463932) suggests that the fix is to change this: proxy_pass http://127.0.0.1:8000/; To this: proxy_pass http://127.0.0.1:8000; Quoting the nginx documentation: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass > A request URI is passed to the server as follows: > > - If the `proxy_pass` directive is specified with a URI, then when a request is passed to the server, the part of a [normalized](http://nginx.org/en/docs/http/ngx_http_core_module.html#location) request URI matching the location is replaced by a URI specified in the directive: > > location /name/ { > proxy_pass http://127.0.0.1/remote/; > } > > - If `proxy_pass` is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI: > > location /some/path/ { > proxy_pass http://127.0.0.1; > }","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1578#issuecomment-1000471371,https://api.github.com/repos/simonw/datasette/issues/1578,1000471371,IC_kwDOBm6k_c47oftL,9599,simonw,2021-12-23T18:42:50Z,2021-12-23T18:42:50Z,OWNER,"Confirmed, that fixed the bug for me on my server.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1578#issuecomment-1000471782,https://api.github.com/repos/simonw/datasette/issues/1578,1000471782,IC_kwDOBm6k_c47ofzm,9599,simonw,2021-12-23T18:44:01Z,2021-12-23T18:44:01Z,OWNER,"The example nginx config on https://docs.datasette.io/en/stable/deploying.html#nginx-proxy-configuration is currently: ``` daemon off; events { worker_connections 1024; } http { server { listen 80; location /my-datasette { proxy_pass http://127.0.0.1:8009/my-datasette; proxy_set_header Host $host; } } } ``` This looks to me like it might exhibit the bug. Need to confirm that and figure out an alternative.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087919372,Confirm if documented nginx proxy config works for row pages with escaped characters in their primary key, https://github.com/simonw/datasette/issues/1579#issuecomment-1000476413,https://api.github.com/repos/simonw/datasette/issues/1579,1000476413,IC_kwDOBm6k_c47og79,9599,simonw,2021-12-23T18:56:06Z,2021-12-23T18:56:06Z,OWNER,"This is technically a breaking change, but a GitHub code search at https://cs.github.com/?scopeName=All+repos&scope=&q=execute_write%20datasette%20-owner%3Asimonw shows only one repo not-owned-by-me using this, and they're using `block=True`: https://github.com/mfa/datasette-webhook-write/blob/e82440f372a2f2e3ed27d1bd34c9fa3a53b49b94/datasette_webhook_write/__init__.py#L88-L89","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000477621,https://api.github.com/repos/simonw/datasette/issues/1579,1000477621,IC_kwDOBm6k_c47ohO1,9599,simonw,2021-12-23T18:59:12Z,2021-12-23T18:59:12Z,OWNER,"The easiest way to change this would be to default to `block=True` such that you need to pass `block=False` to the APIs to have them do fire-and-forget. An alternative would be to add new, separately named methods which do the fire-and-forget thing. If I hadn't recently added `execute_write_script` and `execute_write_many` in #1570 I'd be more into this idea, but I don't want to end up with eight methods - `execute_write`, `execute_write_queue`, `execute_write_many`, `execute_write_many_queue`, `execute_write_script`, `execute_write_scrript_queue`, `execute_write_fn`, `execute_write_fn_queue`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000477813,https://api.github.com/repos/simonw/datasette/issues/1579,1000477813,IC_kwDOBm6k_c47ohR1,9599,simonw,2021-12-23T18:59:41Z,2021-12-23T18:59:41Z,OWNER,"I'm going to go with `execute_write(..., block=False)` as the mechanism for fire-and-forget write queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000479737,https://api.github.com/repos/simonw/datasette/issues/1579,1000479737,IC_kwDOBm6k_c47ohv5,9599,simonw,2021-12-23T19:04:23Z,2021-12-23T19:04:23Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/docs/internals.rst#await-dbexecute_writesql-paramsnone-blocktrue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000481686,https://api.github.com/repos/simonw/datasette/issues/1579,1000481686,IC_kwDOBm6k_c47oiOW,9599,simonw,2021-12-23T19:09:23Z,2021-12-23T19:09:23Z,OWNER,"Re-opening this because I missed updating some of the docs, and I also need to update Datasette's own code to not use `block=True` in a bunch of places.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000485505,https://api.github.com/repos/simonw/datasette/issues/1579,1000485505,IC_kwDOBm6k_c47ojKB,9599,simonw,2021-12-23T19:19:13Z,2021-12-23T19:19:13Z,OWNER,Updated docs for `execute_write_fn()`: https://github.com/simonw/datasette/blob/75153ea9b94d09ec3d61f7c6ebdf378e0c0c7a0b/docs/internals.rst#await-dbexecute_write_fnfn-blocktrue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1579#issuecomment-1000485719,https://api.github.com/repos/simonw/datasette/issues/1579,1000485719,IC_kwDOBm6k_c47ojNX,9599,simonw,2021-12-23T19:19:45Z,2021-12-23T19:19:45Z,OWNER,All of those removed `block=True` lines in 8c401ee0f054de2f568c3a8302c9223555146407 really help confirm to me that this was a good decision.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087931918,`.execute_write(... block=True)` should be the default behaviour, https://github.com/simonw/datasette/issues/1534#issuecomment-1000535904,https://api.github.com/repos/simonw/datasette/issues/1534,1000535904,IC_kwDOBm6k_c47ovdg,9599,simonw,2021-12-23T21:44:31Z,2021-12-23T21:44:31Z,OWNER,A big downside to this is that I would need to use `Vary: Accept` for when Datasette is running behind a cache such as Cloudflare - would that greatly reduce overall cache efficiency due to subtle variations in the accept headers sent by common browsers?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/issues/1577#issuecomment-1000673444,https://api.github.com/repos/simonw/datasette/issues/1577,1000673444,IC_kwDOBm6k_c47pRCk,9599,simonw,2021-12-24T06:08:58Z,2021-12-24T06:08:58Z,OWNER,"https://pypistats.org/packages/datasette shows a breakdown of downloads by Python version: <img width=""986"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/147323253-1ee22d93-3be2-472b-8ead-495d925958e5.png""> It looks like on a recent day I had 4,071 downloads from Python 3.7... and just 2 downloads from Python 3.6!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1576#issuecomment-1000935523,https://api.github.com/repos/simonw/datasette/issues/1576,1000935523,IC_kwDOBm6k_c47qRBj,9599,simonw,2021-12-24T21:33:05Z,2021-12-24T21:33:05Z,OWNER,"Another option would be to attempt to import `contextvars` and, if the import fails (for Python 3.6) continue using the current mechanism - then let Python 3.6 users know in the documentation that under Python 3.6 they will miss out on nested traces.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/878#issuecomment-1001699559,https://api.github.com/repos/simonw/datasette/issues/878,1001699559,IC_kwDOBm6k_c47tLjn,9599,simonw,2021-12-27T18:53:04Z,2021-12-27T18:53:04Z,OWNER,"I'm going to see if I can come up with the simplest possible version of this pattern for the `/-/metadata` and `/-/metadata.json` page, then try it for the database query page, before tackling the much more complex table page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1152#issuecomment-1001791592,https://api.github.com/repos/simonw/datasette/issues/1152,1001791592,IC_kwDOBm6k_c47tiBo,9599,simonw,2021-12-27T23:04:31Z,2021-12-27T23:04:31Z,OWNER,Another option: rethink permissions to always work in terms of where clauses users as part of a SQL query that returns the overall allowed set of databases or tables. This would require rethinking existing permissions but it might be worthwhile prior to 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view, https://github.com/simonw/datasette/issues/1583#issuecomment-1002825217,https://api.github.com/repos/simonw/datasette/issues/1583,1002825217,IC_kwDOBm6k_c47xeYB,536941,fgregg,2021-12-30T00:34:16Z,2021-12-30T00:34:16Z,CONTRIBUTOR,"if that is not desirable, it might be good to document that users might want to set up a lifecycle rule to automatically delete these build artifacts. something like https://stackoverflow.com/questions/59937542/can-i-delete-container-images-from-google-cloud-storage-artifacts-bucket","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1090810196,consider adding deletion step of cloudbuild artifacts to gcloud publish, https://github.com/simonw/datasette/issues/1609#issuecomment-1020456608,https://api.github.com/repos/simonw/datasette/issues/1609,1020456608,IC_kwDOBm6k_c480u6g,9599,simonw,2022-01-24T19:20:09Z,2022-01-24T19:20:09Z,OWNER,Uvicorn have a release out now that would have fixed this issue if I hadn't shipped my own fix: https://github.com/encode/uvicorn/releases/tag/0.17.0.post1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1612#issuecomment-1021413700,https://api.github.com/repos/simonw/datasette/issues/1612,1021413700,IC_kwDOBm6k_c484YlE,9599,simonw,2022-01-25T17:07:29Z,2022-01-25T17:07:29Z,OWNER,"That's a much better place for them, I like this idea. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1612#issuecomment-1021472918,https://api.github.com/repos/simonw/datasette/issues/1612,1021472918,IC_kwDOBm6k_c484nCW,9599,simonw,2022-01-25T18:14:27Z,2022-01-25T18:15:54Z,OWNER,"They're currently shown at the very bottom of the page, under the list of tables and far away from the SQL query box: https://latest.datasette.io/fixtures <img width=""353"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/151034640-c3b79214-7fec-4137-b3ed-45541167feff.png""> I'm also questioning if ""Queries"" is the best header for this. Other options: - **Canned queries** (what the feature is called in the documentation, but I don't think it's a great user-facing term) - **Saved queries** - overlaps with a mechanism by which queries can be saved by the user using a plugin such as [datasette-saved-queries](https://github.com/simonw/datasette-saved-queries) - though that plugin does itself use the canned queries plugin hook so not completely unrelated - **Sample or Example queries** - I don't like these much because they're more than just examples - they are often the core functionality of the specific customized Datasette instance - **Prepared queries** - overlaps with terminology used in other databases, so not great either - **Pre-configured queries** - urgh, don't like that language, feels clumsy - **Query recipes** - bit out of left-field this one, only really makes sense for queries that include named parameters for specific use-cases Maybe ""Queries"" is right after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1612#issuecomment-1021477220,https://api.github.com/repos/simonw/datasette/issues/1612,1021477220,IC_kwDOBm6k_c484oFk,9599,simonw,2022-01-25T18:19:31Z,2022-01-25T18:19:31Z,OWNER,"Here's something I like: I also added a ""Tables"" `<h2>` heading and bumped the tables themselves down from a `<h2>` to a `<h3>`: <img width=""909"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/151035671-9920db99-a943-46a6-8cd8-7a0ad7524474.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1612#issuecomment-1021489826,https://api.github.com/repos/simonw/datasette/issues/1612,1021489826,IC_kwDOBm6k_c484rKi,9599,simonw,2022-01-25T18:34:21Z,2022-01-25T18:34:21Z,OWNER,"OK, that's live on https://latest.datasette.io/fixtures now - I really like it. Thanks for the suggestion!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1612#issuecomment-1021497165,https://api.github.com/repos/simonw/datasette/issues/1612,1021497165,IC_kwDOBm6k_c484s9N,639012,jsfenfen,2022-01-25T18:44:23Z,2022-01-25T18:44:23Z,CONTRIBUTOR,"OMG, this might be the fastest OS ticket I've ever filed, thanks so much @simonw ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905,Move canned queries closer to the SQL input area, https://github.com/simonw/datasette/issues/1613#issuecomment-1021860694,https://api.github.com/repos/simonw/datasette/issues/1613,1021860694,IC_kwDOBm6k_c486FtW,9599,simonw,2022-01-26T04:57:53Z,2022-01-26T04:57:53Z,OWNER,"The existing flow where you can apply filters to a table and then click ""View and edit SQL"" to see the query is a good starting point. Group by queries are both crucially important and difficult to assemble for beginners. Providing a way to see the query that was used by a facet (since facets are really just group-by-counts) would be very useful, which could come out of this: - #1080","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/issues/1613#issuecomment-1022254258,https://api.github.com/repos/simonw/datasette/issues/1613,1022254258,IC_kwDOBm6k_c487lyy,9599,simonw,2022-01-26T14:33:46Z,2022-01-26T14:37:31Z,OWNER,"Tool for setting up foreign key relationships. It could even verify the relationship before you apply it - checking that every value in the column does indeed correspond to a value in the other table. Could also detect and suggest possible ones.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/issues/1613#issuecomment-1022255862,https://api.github.com/repos/simonw/datasette/issues/1613,1022255862,IC_kwDOBm6k_c487mL2,9599,simonw,2022-01-26T14:35:31Z,2022-01-26T14:37:44Z,OWNER,"Joins are really hard. A mechanism for constructing them in the table view would help a lot: - https://github.com/simonw/datasette/issues/613","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/issues/1613#issuecomment-1022257496,https://api.github.com/repos/simonw/datasette/issues/1613,1022257496,IC_kwDOBm6k_c487mlY,9599,simonw,2022-01-26T14:37:14Z,2022-01-26T14:37:14Z,OWNER,"Better contextual help on the SQL editor - like in Django SQL Dashboard which shows all available tables and columns. Fancy inline autocomplete would be great too, but that's pretty hard for SQL based on past research.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/issues/1613#issuecomment-1022381732,https://api.github.com/repos/simonw/datasette/issues/1613,1022381732,IC_kwDOBm6k_c488E6k,9599,simonw,2022-01-26T16:41:45Z,2022-01-26T16:41:45Z,OWNER,A better interface for modifying the columns used in the SELECT clause would be useful too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238,Improvements to help make Datasette a better tool for learning SQL, https://github.com/simonw/datasette/pull/1574#issuecomment-1007844190,https://api.github.com/repos/simonw/datasette/issues/1574,1007844190,IC_kwDOBm6k_c48Ente,536941,fgregg,2022-01-08T00:42:12Z,2022-01-08T00:42:12Z,CONTRIBUTOR,is there a reason to not always use the slim option?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084193403,introduce new option for datasette package to use a slim base image, https://github.com/simonw/datasette/issues/1587#issuecomment-1008157908,https://api.github.com/repos/simonw/datasette/issues/1587,1008157908,IC_kwDOBm6k_c48F0TU,9599,simonw,2022-01-08T21:29:06Z,2022-01-08T21:29:06Z,OWNER,"Depending on the SQLite version (and compile options) that ran `ANALYZE` these can be called: - `sqlite_stat1` - `sqlite_stat2` - `sqlite_stat3` - `sqlite_stat4`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097040427,Add `sqlite_stat1`(-4) tables to hidden table list, https://github.com/simonw/datasette/issues/1587#issuecomment-1008157998,https://api.github.com/repos/simonw/datasette/issues/1587,1008157998,IC_kwDOBm6k_c48F0Uu,9599,simonw,2022-01-08T21:29:54Z,2022-01-08T21:29:54Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/datasette/database.py#L339-L354,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097040427,Add `sqlite_stat1`(-4) tables to hidden table list, https://github.com/simonw/datasette/issues/1588#issuecomment-1008227436,https://api.github.com/repos/simonw/datasette/issues/1588,1008227436,IC_kwDOBm6k_c48GFRs,9599,simonw,2022-01-09T04:23:37Z,2022-01-09T04:25:04Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L163-L170 https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L195-L204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097101917,`explain query plan select` is too strict about whitespace, https://github.com/simonw/datasette/issues/1588#issuecomment-1008227491,https://api.github.com/repos/simonw/datasette/issues/1588,1008227491,IC_kwDOBm6k_c48GFSj,9599,simonw,2022-01-09T04:24:09Z,2022-01-09T04:24:09Z,OWNER,"I think this is the fix: ```python re.compile(r""^explain\s+query\s+plan\s+select\b""), ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097101917,`explain query plan select` is too strict about whitespace, https://github.com/simonw/datasette/pull/1574#issuecomment-1008279307,https://api.github.com/repos/simonw/datasette/issues/1574,1008279307,IC_kwDOBm6k_c48GR8L,33631,fs111,2022-01-09T11:26:06Z,2022-01-09T11:26:06Z,NONE,"@fgregg my thinking was backwards compatibility. I don't know what people do to their builds, I just wanted a smaller image for my use case. @simonw any chance to take a look at this? If there is no interest, feel free to close the PR","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084193403,introduce new option for datasette package to use a slim base image, https://github.com/simonw/datasette/issues/1590#issuecomment-1010559681,https://api.github.com/repos/simonw/datasette/issues/1590,1010559681,IC_kwDOBm6k_c48O-rB,1001306,eelkevdbos,2022-01-12T02:10:20Z,2022-01-12T02:10:20Z,NONE,"In my example, path matching happens at the application layer (being the Django channels URLRouter). That might be a somewhat exotic solution that would normally be solved by a proxy like Apache or Nginx. However, in my specific use case, this is a ""feature"" enabling me to do simple management of databases and metadata from within a Django admin app instance mapped in that same router.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010533133,https://api.github.com/repos/simonw/datasette/issues/1590,1010533133,IC_kwDOBm6k_c48O4MN,9599,simonw,2022-01-12T01:19:19Z,2022-01-12T01:19:19Z,OWNER,"Thanks for the steps to reproduce - I have your bug running on my laptop now. I've been mostly testing this stuff using the hosted copy of Datasette here, which doesn't exhibit the bug: https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures?sql=select+sqlite_version%28%29 Something interesting definitely going on here!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010537058,https://api.github.com/repos/simonw/datasette/issues/1590,1010537058,IC_kwDOBm6k_c48O5Ji,9599,simonw,2022-01-12T01:26:34Z,2022-01-12T01:26:34Z,OWNER,"I'm using the https://datasette.io/plugins/datasette-debug-asgi plugin to investigate. On my laptop using Daphne I get this: http://127.0.0.1:8032/datasettes/-/asgi-scope ``` {'actor': None, 'asgi': {'version': '3.0'}, 'client': ['127.0.0.1', 53767], 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x1122aeef0>, 'headers': [(b'host', b'127.0.0.1:8032'), (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko' b'/20100101 Firefox/95.0'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/' b'avif,image/webp,*/*;q=0.8'), (b'accept-language', b'en-US,en;q=0.5'), (b'accept-encoding', b'gzip, deflate'), (b'dnt', b'1'), (b'connection', b'keep-alive'), (b'cookie', b'_ga=GA1.1.742283954.1628542653'), (b'upgrade-insecure-requests', b'1'), (b'sec-fetch-dest', b'document'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-site', b'none'), (b'sec-fetch-user', b'?1')], 'http_version': '1.1', 'method': 'GET', 'path': '/datasettes/-/asgi-scope', 'path_remaining': '', 'query_string': b'', 'raw_path': b'/datasettes/-/asgi-scope', 'root_path': '', 'route_path': '/-/asgi-scope', 'scheme': 'http', 'server': ['127.0.0.1', 8032], 'type': 'http', 'url_route': {'kwargs': {}}} ``` On the demo running on Fly (which I just redeployed with that plugin) I get this: https://datasette-apache-proxy-demo.fly.dev/prefix/-/asgi-scope ``` {'actor': None, 'asgi': {'spec_version': '2.1', 'version': '3.0'}, 'client': ('86.109.12.167', 0), 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x7f4c0413bca0>, 'headers': [(b'host', b'datasette-apache-proxy-demo.fly.dev'), (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko' b'/20100101 Firefox/95.0'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/' b'avif,image/webp,*/*;q=0.8'), (b'accept-language', b'en-US,en;q=0.5'), (b'accept-encoding', b'gzip, deflate, br'), (b'dnt', b'1'), (b'x-request-start', b't=1641950740651658'), (b'sec-fetch-dest', b'document'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-site', b'none'), (b'sec-fetch-user', b'?1'), (b'fly-client-ip', b'24.5.172.176'), (b'x-forwarded-for', b'24.5.172.176, 213.188.193.173, 86.109.12.167'), (b'fly-forwarded-proto', b'https'), (b'x-forwarded-proto', b'https'), (b'fly-forwarded-ssl', b'on'), (b'x-forwarded-ssl', b'on'), (b'fly-forwarded-port', b'443'), (b'x-forwarded-port', b'443'), (b'fly-region', b'sjc'), (b'fly-request-id', b'01FS5Y805BX43HM94T8XW610KG'), (b'via', b'2 fly.io'), (b'fly-dispatch-start', b't=1641950740683198;instance=87f188a2'), (b'x-forwarded-host', b'datasette-apache-proxy-demo.fly.dev'), (b'x-forwarded-server', b'localhost'), (b'connection', b'Keep-Alive')], 'http_version': '1.1', 'method': 'GET', 'path': '/-/asgi-scope', 'query_string': b'', 'raw_path': b'/-/asgi-scope', 'root_path': '', 'scheme': 'https', 'server': ('127.0.0.1', 8001), 'type': 'http', 'url_route': {'kwargs': {}}} ``` The version that works as ` 'raw_path': b'/-/asgi-scope'` - the version that fails has `'raw_path': b'/datasettes/-/asgi-scope'`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010538016,https://api.github.com/repos/simonw/datasette/issues/1590,1010538016,IC_kwDOBm6k_c48O5Yg,9599,simonw,2022-01-12T01:28:19Z,2022-01-12T01:28:19Z,OWNER,"The Daphne one has this key: `'route_path': '/-/asgi-scope',` Maybe Datasette's routing code needs to look out for that, if it's available, and use it to reconstruct the requested path? The code in question is here: https://github.com/simonw/datasette/blob/8c401ee0f054de2f568c3a8302c9223555146407/datasette/app.py#L1143-L1149","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010538188,https://api.github.com/repos/simonw/datasette/issues/1590,1010538188,IC_kwDOBm6k_c48O5bM,9599,simonw,2022-01-12T01:28:41Z,2022-01-12T01:30:43Z,OWNER,"Oh wait! It looks like `route_path` is something I invented there. Yup, I added it in https://github.com/simonw/datasette/commit/a63412152518581c6a3d4e142b937e27dabdbfdb - commit message says: > - new `route_path` key in `request.scope` storing the path that was used for routing with the `base_url` prefix stripped So actually part of the mystery here is: why does the Fly hosted one NOT have that key?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010540923,https://api.github.com/repos/simonw/datasette/issues/1590,1010540923,IC_kwDOBm6k_c48O6F7,9599,simonw,2022-01-12T01:33:49Z,2022-01-12T01:33:49Z,OWNER,"Looking closer at the code quoted above, it doesn't modify `path` or `raw_path` at all - ALL it does is add the `route_path` to the scope.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1590#issuecomment-1010556333,https://api.github.com/repos/simonw/datasette/issues/1590,1010556333,IC_kwDOBm6k_c48O92t,1001306,eelkevdbos,2022-01-12T02:03:59Z,2022-01-12T02:03:59Z,NONE,"Thank you for the quick reply! Just a quick observation, I am running this locally without a proxy, whereas your fly example seems to be running behind an apache proxy (if the name is accurate). Can it be that the apache proxy strips the prefix before it passes on the request to the daphne backend?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1591#issuecomment-1010764036,https://api.github.com/repos/simonw/datasette/issues/1591,1010764036,IC_kwDOBm6k_c48PwkE,9599,simonw,2022-01-12T08:22:16Z,2022-01-12T08:22:32Z,OWNER,"The challenge here is avoiding clashes. What if a plugin adds an option that I later want to use for a new Datasette core feature? Or what if two plugins define the same option? Maybe the solution is to make them use namespaces defined by the plugin name. How about this: datasette t.db --plugin-setting datasette-tiddlywiki db t.db It's a bit verbose having an option that itself then takes THREE strings: plugin name, setting name, setting value - but it would work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1591#issuecomment-1010947634,https://api.github.com/repos/simonw/datasette/issues/1591,1010947634,IC_kwDOBm6k_c48QdYy,82988,psychemedia,2022-01-12T11:32:17Z,2022-01-12T11:32:17Z,CONTRIBUTOR,Is it possible to parse things like `--ext-{plugin}-{arg} VALUE` ?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1592#issuecomment-1011185061,https://api.github.com/repos/simonw/datasette/issues/1592,1011185061,IC_kwDOBm6k_c48RXWl,9599,simonw,2022-01-12T15:50:41Z,2022-01-12T15:50:41Z,OWNER,Twitter: https://twitter.com/dracos/status/1481290103519592459,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100499619,Row pages should show links to foreign keys, https://github.com/simonw/datasette/pull/1593#issuecomment-1012128696,https://api.github.com/repos/simonw/datasette/issues/1593,1012128696,IC_kwDOBm6k_c48U9u4,22429695,codecov[bot],2022-01-13T13:18:35Z,2022-01-13T13:18:35Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1593](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (df73ebb) into [main](https://codecov.io/gh/simonw/datasette/commit/8c401ee0f054de2f568c3a8302c9223555146407?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8c401ee) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1593/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1593 +/- ## ======================================= Coverage 92.09% 92.09% ======================================= Files 34 34 Lines 4516 4516 ======================================= Hits 4159 4159 Misses 357 357 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [8c401ee...df73ebb](https://codecov.io/gh/simonw/datasette/pull/1593?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1101705012,"Update pytest-asyncio requirement from <0.17,>=0.10 to >=0.10,<0.18", https://github.com/simonw/datasette/issues/1590#issuecomment-1012656790,https://api.github.com/repos/simonw/datasette/issues/1590,1012656790,IC_kwDOBm6k_c48W-qW,9599,simonw,2022-01-14T01:05:34Z,2022-01-14T01:05:34Z,OWNER,"I think this prefixed string mechanism is supposed to prevent the `base_url` prefix from being applied twice: https://github.com/simonw/datasette/blob/3664ddd400062123e99500d28b160c7944408c1a/datasette/url_builder.py#L9-L16 But with a bit of extra logging all of the inputs to that are NOT prefixed strings: ``` Urls.path called with: /datasettes/fixtures/compound_three_primary_keys?_sort=content (PrefixedUrlString = False) returning /datasettes/datasettes/fixtures/compound_three_primary_keys?_sort=content ``` So it looks like `urls.path(...)` is indeed the code responsible for doubling up that `/datasettes/` prefix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1595#issuecomment-1012626243,https://api.github.com/repos/simonw/datasette/issues/1595,1012626243,IC_kwDOBm6k_c48W3ND,9599,simonw,2022-01-14T00:00:33Z,2022-01-14T00:00:33Z,OWNER,"Copying those in here: - New plugin hook: [filters_from_request(request, database, table, datasette)](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-filters-from-request), which runs on the table page and can be used to support new custom query string parameters that modify the SQL query. ([#473](https://github.com/simonw/datasette/issues/473)) - The number of unique values in a facet is now always displayed. Previously it was only displayed if the user specified `?_facet_size=max`. ([#1556](https://github.com/simonw/datasette/issues/1556)) - Fixed bug where `?_facet_array=tags&_facet=tags` would only display one of the two selected facets. ([#625](https://github.com/simonw/datasette/issues/625)) - Facets of type `date` or `array` can now be configured in `metadata.json`, see [Facets in metadata.json](https://docs.datasette.io/en/latest/facets.html#facets-metadata). Thanks, David Larlet. ([#1552](https://github.com/simonw/datasette/issues/1552)) - New `?_nosuggest=1` parameter for table views, which disables facet suggestion. ([#1557](https://github.com/simonw/datasette/issues/1557)) - Label columns detected for foreign keys are now case-insensitive, so `Name` or `TITLE` will be detected in the same way as `name` or `title`. ([#1544](https://github.com/simonw/datasette/issues/1544)) - The query string variables exposed by `request.args` will now include blank strings for arguments such as `foo` in `?foo=&bar=1` rather than ignoring those parameters entirely. ([#1551](https://github.com/simonw/datasette/issues/1551)) - Database write connections now execute the [prepare_connection(conn, database, datasette)](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-prepare-connection) plugin hook. ([#1564](https://github.com/simonw/datasette/issues/1564)) - The `Datasette()` constructor no longer requires the `files=` argument, and is now documented at [Datasette class](https://docs.datasette.io/en/latest/internals.html#internals-datasette). ([#1563](https://github.com/simonw/datasette/issues/1563)) - The tracing feature now traces write queries, not just read queries. ([#1568](https://github.com/simonw/datasette/issues/1568)) - Added two methods for writing to the database: [await db.execute_write_script(sql, block=False)](https://docs.datasette.io/en/latest/internals.html#database-execute-write-script) and [await db.execute_write_many(sql, params_seq, block=False)](https://docs.datasette.io/en/latest/internals.html#database-execute-write-many). ([#1570](https://github.com/simonw/datasette/issues/1570)) - Made several performance improvements to the database schema introspection code that runs when Datasette first starts up. ([#1555](https://github.com/simonw/datasette/issues/1555)) - Fixed bug where writable canned queries could not be used with custom templates. ([#1547](https://github.com/simonw/datasette/issues/1547))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102484126,Release notes for 0.60, https://github.com/simonw/datasette/issues/1595#issuecomment-1012626410,https://api.github.com/repos/simonw/datasette/issues/1595,1012626410,IC_kwDOBm6k_c48W3Pq,9599,simonw,2022-01-14T00:00:56Z,2022-01-14T01:17:47Z,OWNER,Commits since 0.60a1: https://github.com/simonw/datasette/compare/0.60a1...3664ddd40,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102484126,Release notes for 0.60, https://github.com/simonw/datasette/issues/1597#issuecomment-1012629825,https://api.github.com/repos/simonw/datasette/issues/1597,1012629825,IC_kwDOBm6k_c48W4FB,9599,simonw,2022-01-14T00:07:36Z,2022-01-14T00:07:36Z,OWNER,"Fixed: ``` % datasette --help Usage: datasette [OPTIONS] COMMAND [ARGS]... Datasette is an open source multi-tool for exploring and publishing data About Datasette: https://datasette.io/ Full documentation: https://docs.datasette.io/ Options: --version Show the version and exit. --help Show this message and exit. Commands: serve* Serve up specified SQLite database files with a web UI inspect Generate JSON summary of provided database files ... % datasette inspect --help Usage: datasette inspect [OPTIONS] [FILES]... Generate JSON summary of provided database files This can then be passed to ""datasette --inspect-file"" to speed up count operations against immutable database files. Options: --inspect-file TEXT --load-extension TEXT Path to a SQLite extension to load --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102612922,"""datasette inspect"" has no help summary", https://github.com/simonw/datasette/issues/1527#issuecomment-1012634659,https://api.github.com/repos/simonw/datasette/issues/1527,1012634659,IC_kwDOBm6k_c48W5Qj,9599,simonw,2022-01-14T00:17:00Z,2022-01-14T00:18:11Z,OWNER,"That's because that page has this unnecessary hidden form field: ```html <input type=""hidden"" name=""_city_id__gt"" value=""1""> ``` That field is added by this bit in the template: https://github.com/simonw/datasette/blob/515f8d38ebae203efc15ca79a8b42848276b35e5/datasette/templates/table.html#L119-L122 Which is populated here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/views/table.py#L813-L821 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1527#issuecomment-1012635696,https://api.github.com/repos/simonw/datasette/issues/1527,1012635696,IC_kwDOBm6k_c48W5gw,9599,simonw,2022-01-14T00:19:10Z,2022-01-14T00:20:36Z,OWNER,"Oh! This is because `_city_id` has a leading underscore (for testing purposes). I think I need to filter out any keys that contain `__` in that case. What happens to columns that contain a `__`? They shouldn't be reflected in the hidden arguments either - this code is really only supposed to catch things like `_where` and `_m2m_through=` and `_col` - basically most of the list on https://docs.datasette.io/en/stable/json_api.html#special-table-arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1598#issuecomment-1012643882,https://api.github.com/repos/simonw/datasette/issues/1598,1012643882,IC_kwDOBm6k_c48W7gq,9599,simonw,2022-01-14T00:34:49Z,2022-01-14T00:34:49Z,OWNER,There are four places in the documentation that use `.. literalinclude::` at the moment which I can replace - I can actually just link directly to the new https://docs.datasette.io/en/latest/cli-reference.html page instead of embedding the help directly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102637351,Replace update-docs-help.py script with cog, https://github.com/simonw/datasette/issues/1590#issuecomment-1012653966,https://api.github.com/repos/simonw/datasette/issues/1590,1012653966,IC_kwDOBm6k_c48W9-O,9599,simonw,2022-01-14T00:59:07Z,2022-01-14T00:59:07Z,OWNER,"Since this is a special case bug for when using Datasette as a library I wonder if a good fix here would be to support something like this: ```python application = URLRouter([ re_path(r""^datasettes/.*"", asgi_cors(datasette_.app(remove_path_prefix=""datasettes/""), allow_all=True)), ]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1527#issuecomment-1012653109,https://api.github.com/repos/simonw/datasette/issues/1527,1012653109,IC_kwDOBm6k_c48W9w1,9599,simonw,2022-01-14T00:57:08Z,2022-01-14T00:57:08Z,OWNER,Bug is fixed on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1059555791,Columns starting with an underscore behave poorly in filters, https://github.com/simonw/datasette/issues/1591#issuecomment-1012506595,https://api.github.com/repos/simonw/datasette/issues/1591,1012506595,IC_kwDOBm6k_c48WZ_j,9599,simonw,2022-01-13T20:52:56Z,2022-01-13T20:52:56Z,OWNER,"You can already run `datasette --help-settings` to see detailed help on available settings. Maybe `datasette --help-plugin-settings` could do the same thing for plugin settings? Or I could even have available plugin settings show up as a list at the bottom of the `datasette --help-settings` output - which currently looks like this: ``` % datasette --help-settings Settings: default_page_size Default page size for the table view (default=100) max_returned_rows Maximum rows that can be returned from a table or custom query (default=1000) num_sql_threads Number of threads in the thread pool for executing SQLite queries (default=3) sql_time_limit_ms Time limit for a SQL query in milliseconds (default=1000) default_facet_size Number of values to return for requested facets (default=30) facet_time_limit_ms Time limit for calculating a requested facet (default=200) facet_suggest_time_limit_ms Time limit for calculating a suggested facet (default=50) hash_urls Include DB file contents hash in URLs, for far- future caching (default=False) allow_facet Allow users to specify columns to facet using ?_facet= parameter (default=True) allow_download Allow users to download the original SQLite database files (default=True) suggest_facets Calculate and display suggested facets (default=True) default_cache_ttl Default HTTP cache TTL (used in Cache-Control: max-age= header) (default=5) default_cache_ttl_hashed Default HTTP cache TTL for hashed URL pages (default=31536000) cache_size_kb SQLite cache size in KB (0 == use SQLite default) (default=0) allow_csv_stream Allow .csv?_stream=1 to download all rows (ignoring max_returned_rows) (default=True) max_csv_mb Maximum size allowed for CSV export in MB - set 0 to disable this limit (default=100) truncate_cells_html Truncate cells longer than this in HTML table view - set 0 to disable (default=2048) force_https_urls Force URLs in API output to always use https:// protocol (default=False) template_debug Allow display of template debug information with ?_context=1 (default=False) trace_debug Allow display of SQL trace debug information with ?_trace=1 (default=False) base_url Datasette URLs should use this base path (default=/) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1591#issuecomment-1012504251,https://api.github.com/repos/simonw/datasette/issues/1591,1012504251,IC_kwDOBm6k_c48WZa7,9599,simonw,2022-01-13T20:49:19Z,2022-01-13T20:49:59Z,OWNER,"I try to stick pretty closely to what [Click](https://click.palletsprojects.com/en/8.0.x/) supports, and Click likes you to define options explicitly so that it can display them in the output of `--help`. But... that makes me think that actually showing these options in `--help` is likely a better idea. My `--plugin-setting` concept would have help that looks something like this: ``` % datasette serve --help ... --plugin-setting <TEXT TEXT TEXT>... Setting for a specified plugin. ``` That's not great help! The alternative would be to allow plugins to register their extra options with the command - which would mean the help output could look like this instead: ``` % datasette serve --help ... --tiddlywiki-db <TEXT> Name of database to use for datasette-tiddlywiki ``` This feels like a good argument to me for plugins to explicitly register their settings. I'm not sure if I should enforce the `name-of-plugin-` prefix, or if I should allow plugins to define any setting they like. I'm still nervous about plugins over-riding existing or future default options to that command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1591#issuecomment-1012505706,https://api.github.com/repos/simonw/datasette/issues/1591,1012505706,IC_kwDOBm6k_c48WZxq,9599,simonw,2022-01-13T20:51:30Z,2022-01-13T20:51:30Z,OWNER,"Another option: if I make plugin settings a higher level concept in Datasette than they are at the moment, I could allow them to be set either using `--options` OR using the existing `metadata.yml/json` mechanism. https://docs.datasette.io/en/stable/plugins.html#plugin-configuration I want to make changes to that anyway, because I'm increasingly uncomfortable with plugin settings ending up in the ""metadata"" mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1590#issuecomment-1012661522,https://api.github.com/repos/simonw/datasette/issues/1590,1012661522,IC_kwDOBm6k_c48W_0S,9599,simonw,2022-01-14T01:16:08Z,2022-01-14T01:16:34Z,OWNER,"OK, I'm going to recommend a workaround for this instead. Here's `asgi.py` updated to strip the prefix before passing the request on to Datasette: ```python import pathlib from asgi_cors import asgi_cors from channels.routing import URLRouter from django.urls import re_path from datasette.app import Datasette def rewrite_path(app, prefix_to_strip): async def rewrite_path_app(scope, receive, send): if ( scope[""type""] == ""http"" and ""path"" in scope and scope[""path""].startswith(prefix_to_strip) ): scope[""path""] = scope[""path""][len(prefix_to_strip) :] if ""raw_path"" in scope: scope[""raw_path""] = scope[""raw_path""][len(prefix_to_strip) :] await app(scope, receive, send) return rewrite_path_app datasette_ = Datasette( files=[""fixtures.db""], settings={""base_url"": ""/datasettes/"", ""plugins"": {}}, ) application = URLRouter( [ re_path( r""^datasettes/.*"", asgi_cors(rewrite_path(datasette_.app(), ""/datasettes""), allow_all=True), ), ] ) ``` This works on my laptop - please re-open the ticket if it doesn't work for you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1594#issuecomment-1012508787,https://api.github.com/repos/simonw/datasette/issues/1594,1012508787,IC_kwDOBm6k_c48Wahz,9599,simonw,2022-01-13T20:56:14Z,2022-01-13T20:56:34Z,OWNER,"The implementation can be _almost_ exactly the same as this: https://github.com/simonw/sqlite-utils/blame/74586d3cb26fa3cc3412721985ecdc1864c2a31d/docs/cli-reference.rst#L11-L76 I need to do something extra to ensure the output of `datasette --help-settings` is shown too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102359726,"Add a CLI reference page to the docs, inspired by sqlite-utils", https://github.com/simonw/datasette/issues/1594#issuecomment-1012535024,https://api.github.com/repos/simonw/datasette/issues/1594,1012535024,IC_kwDOBm6k_c48Wg7w,9599,simonw,2022-01-13T21:36:53Z,2022-01-13T21:36:53Z,OWNER,I went with a simpler pattern that `sqlite-utils` because Datasette has a lot less commands: https://github.com/simonw/datasette/blob/4b23f01f3e668c8f2a2f1a294be49f49b4073969/docs/cli-reference.rst#L9-L35,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102359726,"Add a CLI reference page to the docs, inspired by sqlite-utils", https://github.com/simonw/datasette/issues/1594#issuecomment-1012536257,https://api.github.com/repos/simonw/datasette/issues/1594,1012536257,IC_kwDOBm6k_c48WhPB,9599,simonw,2022-01-13T21:38:48Z,2022-01-13T21:38:48Z,OWNER,https://docs.datasette.io/en/latest/cli-reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102359726,"Add a CLI reference page to the docs, inspired by sqlite-utils", https://github.com/simonw/datasette/issues/1336#issuecomment-1012546924,https://api.github.com/repos/simonw/datasette/issues/1336,1012546924,IC_kwDOBm6k_c48Wj1s,9599,simonw,2022-01-13T21:55:58Z,2022-01-13T21:55:58Z,OWNER,"See also: - #1412","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",895686039,Document turning on WAL for live served SQLite databases, https://github.com/simonw/datasette/issues/1466#issuecomment-1012546223,https://api.github.com/repos/simonw/datasette/issues/1466,1012546223,IC_kwDOBm6k_c48Wjqv,9599,simonw,2022-01-13T21:54:51Z,2022-01-13T21:54:51Z,OWNER,"Going with this for the copy: > [Datasette Desktop](https://datasette.io/desktop) is a packaged Mac application which bundles Datasette together with Python and allows you to install and run Datasette directly on your laptop. This is the best option for local installation if you are not comfortable using the command line.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991467558,Add Datasette Desktop to installation documentation, https://github.com/simonw/datasette/issues/650#issuecomment-1012552760,https://api.github.com/repos/simonw/datasette/issues/650,1012552760,IC_kwDOBm6k_c48WlQ4,9599,simonw,2022-01-13T22:04:56Z,2022-01-13T22:04:56Z,OWNER,"Challenge: explain the difference between view as in SQL view, and view as in the code that serves `TableView` / `DatabaseView` etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534629631,Add a glossary to the documentation, https://github.com/simonw/datasette/issues/1588#issuecomment-1012554208,https://api.github.com/repos/simonw/datasette/issues/1588,1012554208,IC_kwDOBm6k_c48Wlng,9599,simonw,2022-01-13T22:07:15Z,2022-01-13T22:07:15Z,OWNER,This works now: https://latest.datasette.io/fixtures?sql=explain+query+plan++select+*+from+facetable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097101917,`explain query plan select` is too strict about whitespace, https://github.com/simonw/datasette/issues/1595#issuecomment-1012575013,https://api.github.com/repos/simonw/datasette/issues/1595,1012575013,IC_kwDOBm6k_c48Wqsl,9599,simonw,2022-01-13T22:29:22Z,2022-01-13T22:29:22Z,OWNER,"Most of these are already written for these two alpha releases: - https://github.com/simonw/datasette/releases/tag/0.60a0 - https://github.com/simonw/datasette/releases/tag/0.60a1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102484126,Release notes for 0.60, https://github.com/simonw/datasette/issues/1590#issuecomment-1012583091,https://api.github.com/repos/simonw/datasette/issues/1590,1012583091,IC_kwDOBm6k_c48Wsqz,9599,simonw,2022-01-13T22:41:15Z,2022-01-13T22:41:15Z,OWNER,"Seeing as this area of the code has produced so many bugs in the past, I think part of the fix may be to write comprehensive documentation about how routing works for the internals documentation. Doing so might help me figure this bug out!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099723916,Table+query JSON and CSV links broken when using `base_url` setting, https://github.com/simonw/datasette/issues/1595#issuecomment-1012664607,https://api.github.com/repos/simonw/datasette/issues/1595,1012664607,IC_kwDOBm6k_c48XAkf,9599,simonw,2022-01-14T01:22:58Z,2022-01-14T01:22:58Z,OWNER,"- Upgraded Pluggy dependency to 1.0. #1575 - Now using [Plausible](https://plausible.io/) analytics for the Datasette documentation. - The `db.execute_write()` internals method now defaults to blocking until the write operation has completed. Previously it defaulted to queuing the write and then continuing to run code while the write was in the queue. #1579 - `explain query plan` is now allowed with varying amounts of white space in the query. #1588 - New CLI reference page showing the output of `--help` for each of the `datasette` sub-commands. This lead to several small improvements to the help copy. #1594 - Fixed bug where columns with a underscore prefix could result in unnecessary hidden form fields. #1527","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102484126,Release notes for 0.60, https://github.com/simonw/datasette/issues/1566#issuecomment-1012680228,https://api.github.com/repos/simonw/datasette/issues/1566,1012680228,IC_kwDOBm6k_c48XEYk,9599,simonw,2022-01-14T01:59:54Z,2022-01-14T01:59:54Z,OWNER,This is now shipped!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1083669410,Release Datasette 0.60, https://github.com/simonw/datasette/issues/1591#issuecomment-1013669543,https://api.github.com/repos/simonw/datasette/issues/1591,1013669543,IC_kwDOBm6k_c48a16n,9599,simonw,2022-01-15T11:56:59Z,2022-01-15T11:56:59Z,OWNER,"There's actually already a way to move regular Datasette `--setting` options to a `settings.json` file thanks to configuration directory mode: https://docs.datasette.io/en/stable/settings.html Maybe extending that mechanism to handle plugins would be a neat path forward here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1591#issuecomment-1013668967,https://api.github.com/repos/simonw/datasette/issues/1591,1013668967,IC_kwDOBm6k_c48a1xn,9599,simonw,2022-01-15T11:53:21Z,2022-01-15T11:53:21Z,OWNER,"The `datasette publish --plugin-secret name setting value` option already implements something a bit like this. https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values It's a bit of a messy hack to compensate for metadata being visible. Maybe I could replace that mechanism with the proposed plugin configuration rethink from this issue. I still like the debug benefits of making plugin settings public - perhaps add a rule that if a plugin setting has a `secret:` prefix it gets redacted on a new `/-/plugin-settings` page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1100015398,Maybe let plugins define custom serve options?, https://github.com/simonw/datasette/issues/1603#issuecomment-1016588326,https://api.github.com/repos/simonw/datasette/issues/1603,1016588326,IC_kwDOBm6k_c48l-gm,9599,simonw,2022-01-19T15:35:33Z,2022-01-19T15:35:33Z,OWNER,"I don't think abusing the template loader mechanism for this will work: Jinja provides an API for loading text templates, but I don't think it can be sensibly abused to open binary image files instead. Loaded code is here: https://github.com/pallets/jinja/blob/main/src/jinja2/loaders.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1016589140,https://api.github.com/repos/simonw/datasette/issues/1603,1016589140,IC_kwDOBm6k_c48l-tU,9599,simonw,2022-01-19T15:36:16Z,2022-01-19T15:36:16Z,OWNER,For the moment then I will hard-code a new favicon and leave it to ASGI plugins if people want to define their own.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1016589519,https://api.github.com/repos/simonw/datasette/issues/1603,1016589519,IC_kwDOBm6k_c48l-zP,9599,simonw,2022-01-19T15:36:38Z,2022-01-19T15:36:38Z,OWNER,Also people can use a custom base template and link to a custom favicon if they want to.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1016579661,https://api.github.com/repos/simonw/datasette/issues/1603,1016579661,IC_kwDOBm6k_c48l8ZN,9599,simonw,2022-01-19T15:27:05Z,2022-01-19T15:27:05Z,OWNER,I'd like this to be customizable. I'm going to load it from the template loading system such that a custom favicon for a specific instance can be dropped in using a file in `templates/`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/pull/1602#issuecomment-1016456784,https://api.github.com/repos/simonw/datasette/issues/1602,1016456784,IC_kwDOBm6k_c48leZQ,22429695,codecov[bot],2022-01-19T13:17:24Z,2022-03-06T01:30:46Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1602](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9eb0bdf) into [main](https://codecov.io/gh/simonw/datasette/commit/5010d1359b9e9db90a5a69a3ca22d12862893e00?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (5010d13) will **increase** coverage by `0.13%`. > The diff coverage is `n/a`. > :exclamation: Current head 9eb0bdf differs from pull request most recent head a9c69dc. Consider uploading reports for the commit a9c69dc to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1602/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1602 +/- ## ========================================== + Coverage 92.03% 92.16% +0.13% ========================================== Files 34 34 Lines 4557 4531 -26 ========================================== - Hits 4194 4176 -18 + Misses 363 355 -8 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/tracer.py](https://codecov.io/gh/simonw/datasette/pull/1602/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3RyYWNlci5weQ==) | `82.95% <0.00%> (-1.09%)` | :arrow_down: | | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1602/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `77.85% <0.00%> (-0.09%)` | :arrow_down: | | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1602/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.79% <0.00%> (-0.07%)` | :arrow_down: | | [datasette/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1602/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL19faW5pdF9fLnB5) | `100.00% <0.00%> (ø)` | | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1602/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.37% <0.00%> (+1.05%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [73f2d25...a9c69dc](https://codecov.io/gh/simonw/datasette/pull/1602?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108084641,"Update pytest-timeout requirement from <2.1,>=1.4.2 to >=1.4.2,<2.2", https://github.com/simonw/datasette/issues/1601#issuecomment-1016639294,https://api.github.com/repos/simonw/datasette/issues/1601,1016639294,IC_kwDOBm6k_c48mK8-,9599,simonw,2022-01-19T16:26:44Z,2022-01-20T03:57:17Z,OWNER,"I need to add `sqlite_stat1` to the hidden tables too, see: - https://github.com/simonw/datasette/issues/1587 - https://github.com/simonw/sqlite-utils/issues/366","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1604#issuecomment-1016636561,https://api.github.com/repos/simonw/datasette/issues/1604,1016636561,IC_kwDOBm6k_c48mKSR,9599,simonw,2022-01-19T16:23:54Z,2022-01-19T16:23:54Z,OWNER,"Potential design: datasette publish cloudrun data.db \ --service my-service \ --domain demo.datasette.io I think I'm OK with calling this `--domain` even when it is being used with a subdomain. This will require `datasette.io` to already have been verified with Google. Not sure how best to handle the DNS part - maybe print out instructions for the necessary CNAME?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108300685,Option to assign a domain/subdomain using `datasette publish cloudrun`, https://github.com/simonw/datasette/issues/1601#issuecomment-1016637722,https://api.github.com/repos/simonw/datasette/issues/1601,1016637722,IC_kwDOBm6k_c48mKka,9599,simonw,2022-01-19T16:25:07Z,2022-01-19T16:25:07Z,OWNER,Good idea - though I'm nervous about accidentally hiding a `data_licenses` table outside of the SpatiaLite case. I'll only hide that one if SpatiaLite is loaded.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1601#issuecomment-1016651485,https://api.github.com/repos/simonw/datasette/issues/1601,1016651485,IC_kwDOBm6k_c48mN7d,25778,eyeseast,2022-01-19T16:39:03Z,2022-01-19T16:39:03Z,CONTRIBUTOR,"I think both of these are Spatialite specific. They get generated when you first initialize the extension. KNN is actually deprecated in favor of [KNN2](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN2), as I understand it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1577#issuecomment-1017112130,https://api.github.com/repos/simonw/datasette/issues/1577,1017112130,IC_kwDOBm6k_c48n-ZC,9599,simonw,2022-01-20T04:33:57Z,2022-01-20T04:33:57Z,OWNER,"Dropped support for Python 3.6: updated `setup.py`, changed all references in the documentation to say 3.7 is the minimum required version, got rid of the GitHub Actions tests against 3.6.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1576#issuecomment-1017112543,https://api.github.com/repos/simonw/datasette/issues/1576,1017112543,IC_kwDOBm6k_c48n-ff,9599,simonw,2022-01-20T04:35:00Z,2022-02-05T04:33:46Z,OWNER,I dropped support for Python 3.6 in fae3983c51f4a3aca8335f3e01ff85ef27076fbf so now free to use `contextvars` for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1601#issuecomment-1017113831,https://api.github.com/repos/simonw/datasette/issues/1601,1017113831,IC_kwDOBm6k_c48n-zn,9599,simonw,2022-01-20T04:38:14Z,2022-01-20T04:38:14Z,OWNER,"I don't have solid tests in place for exercising SpatiaLite, but this change feels safe enough that I'm not going to add tests for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1601#issuecomment-1017095821,https://api.github.com/repos/simonw/datasette/issues/1601,1017095821,IC_kwDOBm6k_c48n6aN,9599,simonw,2022-01-20T03:56:14Z,2022-01-20T03:56:28Z,OWNER,"Oh interesting, I hadn't heard about KNN2. It looks like it was added to SpatiaLite on 21st June 2021 in https://www.gaia-gis.it/fossil/libspatialite/info/03786a62cdb4ab17 but the most recent release of SpatiaLite is 5.0.1 from 7th February 2021 - so it's not yet in a release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1105916061,Add KNN and data_licenses to hidden tables list, https://github.com/simonw/datasette/issues/1606#issuecomment-1017108291,https://api.github.com/repos/simonw/datasette/issues/1606,1017108291,IC_kwDOBm6k_c48n9dD,9599,simonw,2022-01-20T04:24:54Z,2022-01-20T04:24:54Z,OWNER,"https://github.com/simonw/latest-datasette-with-all-plugins/commit/1e12ffe70be791e3281b41810e837515314c1317 shows that 5 days ago Datasette upgraded from Uvicorn 0.16 to 0.17 Sure enough, in the changelog for 0.17: https://github.com/encode/uvicorn/blob/0.17.0/CHANGELOG.md > - Drop Python 3.6 support ([#1261](https://github.com/encode/uvicorn/pull/1261)) 06/01/22","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108846067,Tests failing against Python 3.6, https://github.com/simonw/datasette/issues/1606#issuecomment-1017108960,https://api.github.com/repos/simonw/datasette/issues/1606,1017108960,IC_kwDOBm6k_c48n9ng,9599,simonw,2022-01-20T04:26:36Z,2022-01-20T04:26:36Z,OWNER,"https://pypistats.org/packages/datasette shows Python 3.6 is 0.24% of Datasette downloads. <img width=""941"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150273301-18ec2ec8-f9ce-49a3-b57d-e74826a5c7ae.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108846067,Tests failing against Python 3.6, https://github.com/simonw/datasette/issues/1606#issuecomment-1017109194,https://api.github.com/repos/simonw/datasette/issues/1606,1017109194,IC_kwDOBm6k_c48n9rK,9599,simonw,2022-01-20T04:27:07Z,2022-01-20T04:27:07Z,OWNER,"Relevant: - https://github.com/simonw/datasette/issues/1577","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108846067,Tests failing against Python 3.6, https://github.com/simonw/datasette/issues/1577#issuecomment-1017109401,https://api.github.com/repos/simonw/datasette/issues/1577,1017109401,IC_kwDOBm6k_c48n9uZ,9599,simonw,2022-01-20T04:27:34Z,2022-01-20T04:27:34Z,OWNER,"OK, now that Uvicorn has dropped 3.6 support - see #1606 - I think this decision is easy to make. I'm dropping 3.6.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087913724,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1605#issuecomment-1016977725,https://api.github.com/repos/simonw/datasette/issues/1605,1016977725,IC_kwDOBm6k_c48ndk9,9599,simonw,2022-01-19T23:55:08Z,2022-01-19T23:55:08Z,OWNER,"Oh that's interesting. I was thinking about this from a slightly different angle recently - pondering what a static site generator built on top of Datasette might look like. Just a sketch at the moment, but I was imagining a YAML configuration file with a SQL query that returns a list of paths - then a tool that runs that query and uses the equivalent of `datasette --get` to create a static copy of each of those paths. I think these two ideas can probably be merged. I'd love to know more about how you are solving this right now!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1605#issuecomment-1016994329,https://api.github.com/repos/simonw/datasette/issues/1605,1016994329,IC_kwDOBm6k_c48nhoZ,25778,eyeseast,2022-01-20T00:27:17Z,2022-01-20T00:27:17Z,CONTRIBUTOR,"Right now, I usually have a line in a Makefile like this: ```make combined.geojson: project.db pipenv run datasette project.db --get /project/combined.geojson \ --load-extension spatialite \ --setting sql_time_limit_ms 5000 \ --setting max_returned_rows 20000 \ -m metadata.yml > $@ ``` That all assumes I've loaded whatever I need into `project.db` and created a canned query called `combined` (and then uses `datasette-geojson` for geojson output). It works, but as you can see, it's a lot to manage, a lot of boilerplate, and it wasn't obvious how to get there. If there's an error in the canned query, I get an HTML error page, so that's hard to debug. And it's only one query, so each output needs a line like this. Make isn't ideal, either, for that reason. The thing I really liked with `datafreeze` was doing templated filenames. I have a project now where I need to export a bunch of litttle geojson files, based on queries, and it would be awesome to be able to do something like this: ```yml databases: project: queries: boundaries: sql: ""SELECT * FROM boundaries"" filename: ""boundaries/{id}.geojson"" mode: ""item"" format: geojson ``` And then do: ```sh datasette freeze -m metadata.yml project.db ``` For HTML export, maybe there's a `template` argument, or `format: template` or something. And that gets you a static site generator, kinda for free. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1356#issuecomment-1017016553,https://api.github.com/repos/simonw/datasette/issues/1356,1017016553,IC_kwDOBm6k_c48nnDp,9599,simonw,2022-01-20T01:06:37Z,2022-01-20T01:06:37Z,OWNER,"> A problem with this is that if you're using `--query` you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is `.csv` and plugin formats can't handle full streams, see #1062 and #1177. I figured out a neat pattern for streaming JSON arrays in this TIL: https://til.simonwillison.net/python/output-json-array-streaming","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""", https://github.com/simonw/datasette/issues/1600#issuecomment-1017124310,https://api.github.com/repos/simonw/datasette/issues/1600,1017124310,IC_kwDOBm6k_c48oBXW,9599,simonw,2022-01-20T05:06:09Z,2022-01-20T05:06:09Z,OWNER,Fixed: https://docs.datasette.io/en/latest/plugins.html#seeing-what-plugins-are-installed,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1104691662,plugins --all example should use cog, https://github.com/simonw/datasette/issues/1603#issuecomment-1017131209,https://api.github.com/repos/simonw/datasette/issues/1603,1017131209,IC_kwDOBm6k_c48oDDJ,9599,simonw,2022-01-20T05:22:40Z,2022-01-20T05:22:40Z,OWNER,This one is 101KB and 1536*1536 which is a bit big! https://github.com/simonw/datasette-app/blob/main/build/icon.png,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017131801,https://api.github.com/repos/simonw/datasette/issues/1603,1017131801,IC_kwDOBm6k_c48oDMZ,9599,simonw,2022-01-20T05:23:57Z,2022-01-20T05:23:57Z,OWNER,"https://adamj.eu/tech/2022/01/18/how-to-add-a-favicon-to-your-django-site/ suggests 64x64, I'm going with 128x128 just in case anyone invents a retina-retina screen.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017132927,https://api.github.com/repos/simonw/datasette/issues/1603,1017132927,IC_kwDOBm6k_c48oDd_,9599,simonw,2022-01-20T05:26:29Z,2022-01-20T05:26:29Z,OWNER,"Here's the 128x128 one - 11kb, I resized it using Preview: ![icon](https://user-images.githubusercontent.com/9599/150278798-48cc2da2-3640-414d-a440-20c9d93c09f4.png) Now running it through [Squoosh](https://squoosh.app/editor) using OxiPNG effort=3 colours=24 - brought it down to 1.36KB. ![favicon](https://user-images.githubusercontent.com/9599/150278906-b312940e-0cdb-43ef-a325-45d2b1c240de.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017133357,https://api.github.com/repos/simonw/datasette/issues/1603,1017133357,IC_kwDOBm6k_c48oDkt,9599,simonw,2022-01-20T05:27:34Z,2022-01-20T05:27:34Z,OWNER,"I'm going to drop it in `datasette/static/favicon.png` and rewrite the `async def favicon()` function to return it, with caching headers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017136897,https://api.github.com/repos/simonw/datasette/issues/1603,1017136897,IC_kwDOBm6k_c48oEcB,9599,simonw,2022-01-20T05:36:29Z,2022-01-20T05:36:29Z,OWNER,"Here's what it looks like in Firefox, Chrome and Safari: ![CleanShot 2022-01-19 at 21 35 06@2x](https://user-images.githubusercontent.com/9599/150279832-6c233115-6540-4746-8cd1-dce25321ebbf.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017137117,https://api.github.com/repos/simonw/datasette/issues/1603,1017137117,IC_kwDOBm6k_c48oEfd,9599,simonw,2022-01-20T05:37:05Z,2022-01-20T05:37:34Z,OWNER,"I'm not crazy about the look - I think this version of the grid may have too many lines for this particular display size. I'm going to try reducing the number of lines in Figma to see if I like that better. https://www.figma.com/file/LKjceTFNtKm6wCbScDqm1Y/Datasette-Logo","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017139321,https://api.github.com/repos/simonw/datasette/issues/1603,1017139321,IC_kwDOBm6k_c48oFB5,9599,simonw,2022-01-20T05:43:07Z,2022-01-20T05:46:42Z,OWNER,"My attempt with one less grid line looked awful when shrunk down to the browser size: <img width=""350"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150280445-a482de8e-8b5f-4145-8567-36ac7105c303.png""> File here: ![favicon](https://user-images.githubusercontent.com/9599/150280463-817a8065-8393-4c10-aec4-5a627cf46fae.png) I'm going with the first attempt for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017142395,https://api.github.com/repos/simonw/datasette/issues/1603,1017142395,IC_kwDOBm6k_c48oFx7,9599,simonw,2022-01-20T05:50:55Z,2022-01-20T05:50:55Z,OWNER,The new `/favicon.ico` view: https://github.com/simonw/datasette/blob/0467723ee57c2cbc0f02daa47cef632dd4651df0/datasette/app.py#L182-L192,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017143437,https://api.github.com/repos/simonw/datasette/issues/1603,1017143437,IC_kwDOBm6k_c48oGCN,9599,simonw,2022-01-20T05:53:17Z,2022-01-20T05:53:17Z,OWNER,New favicon now live on https://latest.datasette.io/ - see also https://latest.datasette.io/favicon.ico,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017143817,https://api.github.com/repos/simonw/datasette/issues/1603,1017143817,IC_kwDOBm6k_c48oGIJ,9599,simonw,2022-01-20T05:54:09Z,2022-01-20T05:54:09Z,OWNER,"Oops, I pushed the one I liked least out of the two options! Fixing now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017149828,https://api.github.com/repos/simonw/datasette/issues/1603,1017149828,IC_kwDOBm6k_c48oHmE,9599,simonw,2022-01-20T06:07:31Z,2022-01-20T06:07:31Z,OWNER,"Now live on https://latest.datastte.io/ - I'm happy enough with this for the moment: <img width=""364"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150283069-dd23e688-ec77-4200-a892-38e0f8892f30.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017800506,https://api.github.com/repos/simonw/datasette/issues/1603,1017800506,IC_kwDOBm6k_c48qmc6,9599,simonw,2022-01-20T18:31:18Z,2022-01-20T18:31:18Z,OWNER,"One last go at tidying this up. I decided to do a 32x32 pixel version in Pixelmator, using this trick to access a pixel brush: https://osxdaily.com/2016/11/17/enable-pixel-brush-pixelmator-mac/ <img width=""916"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150399724-10b17a97-3ecf-4d82-868c-f8a642f16838.png""> Frustrating how the white boxes are all exactly four pixels high and ALMOST all four pixels wide, but one of them has to be three pixels wide to fit the space.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017806497,https://api.github.com/repos/simonw/datasette/issues/1603,1017806497,IC_kwDOBm6k_c48qn6h,9599,simonw,2022-01-20T18:39:27Z,2022-01-20T18:39:57Z,OWNER,"Here's a comparison between my hand-edited version and the one I have now: ![CleanShot 2022-01-20 at 10 38 00@2x](https://user-images.githubusercontent.com/9599/150401244-7e78ee93-1973-4c95-8f91-01e51e0d5366.png) The new 32x32 image: <img src=""https://user-images.githubusercontent.com/9599/150401295-8873fa40-8859-40e3-99d7-e53276319a15.png"" width=""32"" height=""32"">","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017808898,https://api.github.com/repos/simonw/datasette/issues/1603,1017808898,IC_kwDOBm6k_c48qogC,9599,simonw,2022-01-20T18:42:35Z,2022-01-20T18:42:35Z,OWNER,"Resized it down to 208 bytes with https://squoosh.app <img width=""883"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150401693-d19c7fdf-7f5d-49b4-8f1e-5c9e62cbdbed.png""> ![favicon](https://user-images.githubusercontent.com/9599/150401709-11a50492-a8c8-4eee-848c-c813f8cd3e4e.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1603#issuecomment-1017842366,https://api.github.com/repos/simonw/datasette/issues/1603,1017842366,IC_kwDOBm6k_c48qwq-,9599,simonw,2022-01-20T19:19:54Z,2022-01-20T19:19:54Z,OWNER,Wrote up a TIL: https://til.simonwillison.net/pixelmator/pixel-editing-favicon,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,A proper favicon, https://github.com/simonw/datasette/issues/1608#issuecomment-1017970132,https://api.github.com/repos/simonw/datasette/issues/1608,1017970132,IC_kwDOBm6k_c48rP3U,9599,simonw,2022-01-20T22:08:55Z,2022-01-20T22:08:55Z,OWNER,"Might want to consider SEO here too - I want people from search engines to land on `/stable/`, I wonder if I should noindex or `rel=canonical` the other documentation versions? Not sure what best practices for that is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017969452,https://api.github.com/repos/simonw/datasette/issues/1608,1017969452,IC_kwDOBm6k_c48rPss,9599,simonw,2022-01-20T22:07:55Z,2022-01-20T22:07:55Z,OWNER,"I think I want a banner at the top of the page making it obvious which version the documentation is talking about. This can be pretty low key for the current stable release, but should be visually more prominent for the `/latest/` branch and for older releases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017971905,https://api.github.com/repos/simonw/datasette/issues/1608,1017971905,IC_kwDOBm6k_c48rQTB,9599,simonw,2022-01-20T22:11:40Z,2022-01-20T22:11:40Z,OWNER,"Huh, I had forgotten I already have a banner on older versions: ![D1A65C68-9A37-4FA2-80C4-534739A9D292](https://user-images.githubusercontent.com/9599/150430410-1e22e23f-ed27-4271-89ff-63467eb5f466.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017975322,https://api.github.com/repos/simonw/datasette/issues/1608,1017975322,IC_kwDOBm6k_c48rRIa,9599,simonw,2022-01-20T22:17:01Z,2022-01-20T22:27:07Z,OWNER,"Turns out that banner is something that ReadTheDocs implemented - I found it using GitHub code search, it's produced by this piece of JavaScript: https://github.com/readthedocs/readthedocs.org/blob/0852d7c10d725d954d3e9a93513171baa1116d9f/readthedocs/core/static-src/core/js/doc-embed/version-compare.js#L13-L21 ```javascript function init(data) { var rtd = rtddata.get(); /// Out of date message if (data.is_highest) { return; } var currentURL = window.location.pathname.replace(rtd['version'], data.slug); var warning = $( '<div class=""admonition warning""> ' + '<p class=""first admonition-title"">Note</p> ' + '<p class=""last""> ' + 'You are not reading the most recent version of this documentation. ' + '<a href=""#""></a> is the latest version available.' + '</p>' + '</div>'); warning .find('a') .attr('href', currentURL) .text(data.slug); var body = $(""div.body""); if (!body.length) { body = $(""div.document""); } body.prepend(warning); } ``` And here's where that module is called from the rest of their code: https://github.com/readthedocs/readthedocs.org/blob/bc3e147770e5740314a8e8c33fec5d111c850498/readthedocs/core/static-src/core/js/doc-embed/footer.js#L66-L86","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017981599,https://api.github.com/repos/simonw/datasette/issues/1608,1017981599,IC_kwDOBm6k_c48rSqf,9599,simonw,2022-01-20T22:26:32Z,2022-01-20T22:26:32Z,OWNER,I'm tempted to imitate their JavaScript but check for `/latest/` in the URL and use that to append a similar message warning about this being the documentation for the in-development version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017988556,https://api.github.com/repos/simonw/datasette/issues/1608,1017988556,IC_kwDOBm6k_c48rUXM,9599,simonw,2022-01-20T22:37:51Z,2022-01-20T22:37:51Z,OWNER,"Here's a jQuery recipe that seems to do the right thing: ```javascript jQuery(function ($) { // If this is a /latest/ doc page, show banner linking to /stable/ if (!/\/latest\//.test(location.pathname)) { return; } var stableUrl = location.pathname.replace(""/latest/"", ""/stable/""); // Check it's not a 404 fetch(stableUrl, { method: ""HEAD"" }).then((response) => { if (response.status == 200) { var warning = $( `<div class=""admonition warning""> <p class=""first admonition-title"">Note</p> <p class=""last""> This documentation covers the <strong>development version</strong> of Datasette.</p> <p>See <a href=""${stableUrl}"">this page</a> for the current stable release. </p> </div>` ); warning.find(""a"").attr(""href"", stableUrl); var body = $(""div.body""); if (!body.length) { body = $(""div.document""); } body.prepend(warning); } }); }); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017994925,https://api.github.com/repos/simonw/datasette/issues/1608,1017994925,IC_kwDOBm6k_c48rV6t,9599,simonw,2022-01-20T22:48:43Z,2022-01-20T22:54:02Z,OWNER,"https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for ""You are looking at v0.36 but you should be looking at 0.40"" but doesn't hand the case I need here which is ""you are looking at /latest/ but you should be looking at /stable/"". Just shipped my fix here: https://docs.datasette.io/en/latest/ <img width=""994"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/150434433-e2f0d472-77a8-430b-88e5-f22e2c6e8ef0.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017993482,https://api.github.com/repos/simonw/datasette/issues/1608,1017993482,IC_kwDOBm6k_c48rVkK,316517,astrojuanlu,2022-01-20T22:46:16Z,2022-01-20T22:46:16Z,NONE,Or you can use https://sphinx-version-warning.readthedocs.io/! 😄 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1017998993,https://api.github.com/repos/simonw/datasette/issues/1608,1017998993,IC_kwDOBm6k_c48rW6R,9599,simonw,2022-01-20T22:56:00Z,2022-01-20T22:56:00Z,OWNER,"> https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for ""You are looking at v0.36 but you should be looking at 0.40"" but doesn't hand the case I need here which is ""you are looking at /latest/ but you should be looking at /stable/"". Correction! That tool DOES support that, as can be seen in their example configuration for their own documentation: https://github.com/humitos/sphinx-version-warning/blob/a82156c2ea08e5feab406514d0ccd9d48a345f48/docs/conf.py#L32-L38 ```python versionwarning_messages = { 'latest': 'This is a custom message only for version ""latest"" of this documentation.', } versionwarning_admonition_type = 'tip' versionwarning_banner_title = 'Tip' versionwarning_body_selector = 'div[itemprop=""articleBody""]' ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1018017637,https://api.github.com/repos/simonw/datasette/issues/1608,1018017637,IC_kwDOBm6k_c48rbdl,9599,simonw,2022-01-20T23:27:59Z,2022-01-20T23:27:59Z,OWNER,"Got a couple of TILs out of this: - [Promoting the stable version of the documentation using rel=canonical](https://til.simonwillison.net/readthedocs/documentation-seo-canonical) - [Linking from /latest/ to /stable/ on Read The Docs](https://til.simonwillison.net/readthedocs/link-from-latest-to-stable)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1609#issuecomment-1018064620,https://api.github.com/repos/simonw/datasette/issues/1609,1018064620,IC_kwDOBm6k_c48rm7s,9599,simonw,2022-01-21T01:00:12Z,2022-01-21T01:00:12Z,OWNER,"I think there are two possible solutions then: 1. Convince Uvicorn to publish one last `0.16.1` version which includes that `python_requires=` line, such that there's a version of Uvicorn on PyPI that Python 3.6 can still install. 2. Release a `0.60.1` version of Datasette which pins that Uvicorn version, and hence can be installed. I've made the request for 1) in Uvicorn Gitter here: https://gitter.im/encode/community?at=61ea044a6d9ba23328d0fa28 I'm going to investigate option 2) myself now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018063681,https://api.github.com/repos/simonw/datasette/issues/1609,1018063681,IC_kwDOBm6k_c48rmtB,9599,simonw,2022-01-21T00:58:25Z,2022-01-21T00:58:32Z,OWNER,"On Twitter: https://twitter.com/simonw/status/1484317711672877065 Here's the problem: Uvicorn only added `python_requires` to their `setup.py` a few days ago, which means the releases they have out on PyPI at the moment don't specify the Python version they need, which is why this mechanism doesn't work as expected: - https://github.com/encode/uvicorn/pull/1328","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018077009,https://api.github.com/repos/simonw/datasette/issues/1609,1018077009,IC_kwDOBm6k_c48rp9R,9599,simonw,2022-01-21T01:24:15Z,2022-01-21T01:24:43Z,OWNER,"Problem: if I ship this, it will be the most recent release of Datasette - but unlike other previous releases it has exactly pinned versions of all of the dependencies. Which is bad for people who run `pip install datasette` but want to not be stuck to those exact library versions. So maybe I ship this as 0.60.1, then ship a 0.60.2 release directly afterwards which unpins the dependencies again and requires Python 3.7?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018075357,https://api.github.com/repos/simonw/datasette/issues/1609,1018075357,IC_kwDOBm6k_c48rpjd,9599,simonw,2022-01-21T01:20:56Z,2022-01-21T01:20:56Z,OWNER,"I used the combo of `pyenv` and `pipenv` to run tests and figure out what the most recent versions of each dependency were that worked on Python 3.6. I also clicked around in the latest releases on pages such as https://pypi.org/project/aiofiles ``` cd /tmp git clone git@github.com:simonw/datasette cd /tmp/datasette pipenv shell --python 3.6.10 pip install -e '.[test]' pytest ``` I also used `pip freeze | grep black` to see which version was installed, since packages with `python_requires=` in them would automatically install the highest compatible version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018082792,https://api.github.com/repos/simonw/datasette/issues/1609,1018082792,IC_kwDOBm6k_c48rrXo,9599,simonw,2022-01-21T01:37:11Z,2022-01-21T01:37:11Z,OWNER,"Another option from https://twitter.com/samuel_hames/status/1484327636860293121 - environment markers, described in https://www.python.org/dev/peps/pep-0508/#environment-markers Found some examples of those in use using GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=%22%3Bpython_version%22+path%3Asetup.py - in particular https://github.com/xmendez/wfuzz/blob/1b695ee9a87d66a7d7bf6cae70d60a33fae51541/setup.py#L31-L38 ```python install_requires = [ 'pycurl', 'pyparsing<2.4.2;python_version<=""3.4""', 'pyparsing>=2.4*;python_version>=""3.5""', 'six', 'configparser;python_version<""3.5""', 'chardet', ] ``` So maybe I can ship 0.60.1 with loose dependencies _except_ for the `uvicorn` one on Python 3.6, using an environment marker. Here's my `setup.py` at the moment: https://github.com/simonw/datasette/blob/ffca55dfd7cc9b53522c2e5a2fa1ff67c9beadf2/setup.py#L44-L61 One other problem: there might be packages in that list right now which don't specify their 3.6 Python version but which will, at some point in the future, release a new version that doesn't work with 3.6 (like Uvicorn did) - in which case Python 3.6 installs would break in the future. I think what I'll do then is ship the `0.60.1` Python 3.6 version with strict upper limits on each version which are the current, tested-with-Datasette-on-Python3.6 versions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018086273,https://api.github.com/repos/simonw/datasette/issues/1609,1018086273,IC_kwDOBm6k_c48rsOB,9599,simonw,2022-01-21T01:45:46Z,2022-01-21T01:45:46Z,OWNER,"This whole thing reminds me of my ongoing internal debate about version pinning: should the Datasette package released to PyPI pin to the exact versions of the dependencies that are known to work, or should it allow a range of dependencies so users can pick other versions of the dependencies to use in their environment? As I understand it, the general rule is to use exact pinning for applications but use ranges for libraries. Datasette is almost entirely an application... but it can also be used as a library - and in fact I'm hoping to encourage that usage more in the future, see: - #1398 I'd also like to release a packaged version of Datasette that doesn't require Uvicorn, for running on AWS Lambda and other function-as-a-service platforms. Those platforms have their own HTTP layer and hence don't need the Uvicorn dependency. Maybe the answer is to have a `datasette-core` package which provides the core of Datasette with unpinned dependencies and no Uvicorn, and then have the existing `datasette` package provide the Datasette CLI tool with Uvicorn and pinned dependencies?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018086697,https://api.github.com/repos/simonw/datasette/issues/1609,1018086697,IC_kwDOBm6k_c48rsUp,9599,simonw,2022-01-21T01:46:43Z,2022-01-21T01:46:43Z,OWNER,https://github.com/simonw/datasette/runs/4890775227?check_suite_focus=true - the tests passed on Python 3.6 for this commit with the pinned dependencies: https://github.com/simonw/datasette/commit/41060e7e7cb838328c879de6a98ae794dc1886d0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018092984,https://api.github.com/repos/simonw/datasette/issues/1609,1018092984,IC_kwDOBm6k_c48rt24,9599,simonw,2022-01-21T02:00:38Z,2022-01-21T02:00:38Z,OWNER,"Out of curiosity, I installed this latest `setup.py` file using both Python 3.6 and Python 3.10, ran `pip freeze` on both of them and created a Gist to compare the difference. The result is here: https://gist.github.com/simonw/2e7d5b1beba675ef9a5bcd310cadc372/revisions From that, it looks like the Python packages in my dependencies which have released new versions that don't work with Python 3.6 are: - https://pypi.org/project/janus/#history - https://pypi.org/project/Pint/#history - https://pypi.org/project/platformdirs/#history - https://pypi.org/project/uvicorn/#history (already discussed) Sure enough, for the first three of those browsing through their recent versions on PyPI confirms that they switched from e.g. ""Requires: Python >=3.6"" on https://pypi.org/project/janus/0.7.0/ to ""Requires: Python >=3.7"" on https://pypi.org/project/janus/1.0.0/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018091322,https://api.github.com/repos/simonw/datasette/issues/1609,1018091322,IC_kwDOBm6k_c48rtc6,9599,simonw,2022-01-21T01:56:42Z,2022-01-21T01:56:42Z,OWNER,"OK, the environment markers approach seems to work!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018094767,https://api.github.com/repos/simonw/datasette/issues/1609,1018094767,IC_kwDOBm6k_c48ruSv,9599,simonw,2022-01-21T02:04:14Z,2022-01-21T02:04:14Z,OWNER,So I don't need to release 0.60.1 AND 0.60.2 after all - I can just release 0.60.1 with a bug fix that it no longer breaks installation for Python 3.6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1609#issuecomment-1018104868,https://api.github.com/repos/simonw/datasette/issues/1609,1018104868,IC_kwDOBm6k_c48rwwk,9599,simonw,2022-01-21T02:24:13Z,2022-01-21T02:24:13Z,OWNER,Just shipped 0.60.1 with the fix - and tested that `pip install datasette` does indeed work correctly on Python 3.6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720,"Ensure ""pip install datasette"" still works with Python 3.6", https://github.com/simonw/datasette/issues/1605#issuecomment-1018741262,https://api.github.com/repos/simonw/datasette/issues/1605,1018741262,IC_kwDOBm6k_c48uMIO,25778,eyeseast,2022-01-21T18:05:09Z,2022-01-21T18:05:09Z,CONTRIBUTOR,"Thinking about this more, as well as #1356 and various other tickets related to output formats, I think there's a missing plugin hook for formatting results, separate from `register_output_renderer` (or maybe part of it, depending on #1101). Right now, as I understand it, getting output in any format goes through the normal view stack -- a table, a row or a query -- and so by the time `register_output_renderer` gets it, the results have already been truncated or paginated. What I'd want, I think, is to be able to register ways to format results independent of where those results are sent. It's possible this could be done using [`conn.row_factory`](https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.row_factory) (maybe in the `prepare_connection` hook), but I'm not sure that's where it belongs. Another option is some kind of registry of serializers, which `register_output_renderer` and other plugin hooks could use. What I'm trying to avoid here is writing a plugin that also needs plugins for formats I haven't thought of yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1605#issuecomment-1018766727,https://api.github.com/repos/simonw/datasette/issues/1605,1018766727,IC_kwDOBm6k_c48uSWH,9599,simonw,2022-01-21T18:41:21Z,2022-01-21T18:42:03Z,OWNER,"Yeah I think this all hinges on: - #1101 Also this comment about streaming full JSON arrays (not just newline-delimited) using [this trick](https://til.simonwillison.net/python/output-json-array-streaming): - https://github.com/simonw/datasette/issues/1356#issuecomment-1017016553 I'm about ready to figure these out, as with so much it's still a little bit blocked on the refactor stuff from: - #1518 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1605#issuecomment-1018778667,https://api.github.com/repos/simonw/datasette/issues/1605,1018778667,IC_kwDOBm6k_c48uVQr,25778,eyeseast,2022-01-21T19:00:01Z,2022-01-21T19:00:01Z,CONTRIBUTOR,"Let me know if you want help prototyping any of this, because I'm thinking about it and trying stuff out. Happy to be a sounding board, if it helps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1143#issuecomment-1038289584,https://api.github.com/repos/simonw/datasette/issues/1143,1038289584,IC_kwDOBm6k_c494wqw,9599,simonw,2022-02-13T17:40:50Z,2022-02-13T17:41:17Z,OWNER,"The way Drupal does this is interesting; https://www.drupal.org/node/2715637 - it supports the following YAML: ```yaml # Configure Cross-Site HTTP requests (CORS). # Read https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS # for more information about the topic in general. # Note: By default the configuration is disabled. cors.config: enabled: false # Specify allowed headers, like 'x-allowed-header'. allowedHeaders: [] # Specify allowed request methods, specify ['*'] to allow all possible ones. allowedMethods: [] # Configure requests allowed from specific origins. allowedOrigins: ['*'] # Sets the Access-Control-Expose-Headers header. exposedHeaders: false # Sets the Access-Control-Max-Age header. maxAge: false # Sets the Access-Control-Allow-Credentials header. supportsCredentials: false ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices", https://github.com/simonw/datasette/issues/1615#issuecomment-1023997327,https://api.github.com/repos/simonw/datasette/issues/1615,1023997327,IC_kwDOBm6k_c49CPWP,369053,aidansteele,2022-01-28T08:37:36Z,2022-01-28T08:37:36Z,NONE,"Oops, it feels like this should perhaps be migrated to GitHub Discussions - sorry! I don't think I have the ability to do that 😅","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1117132741,Potential simplified publishing mechanism, https://github.com/simonw/datasette/pull/1616#issuecomment-1025732071,https://api.github.com/repos/simonw/datasette/issues/1616,1025732071,IC_kwDOBm6k_c49I23n,22429695,codecov[bot],2022-01-31T13:20:18Z,2022-01-31T13:20:18Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1616](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4ebe94b) into [main](https://codecov.io/gh/simonw/datasette/commit/2aa686c6554bf6b8230eb5b3019574df6cc99225?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2aa686c) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1616/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1616 +/- ## ======================================= Coverage 92.09% 92.09% ======================================= Files 34 34 Lines 4518 4518 ======================================= Hits 4161 4161 Misses 357 357 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2aa686c...4ebe94b](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1119413338,Bump black from 21.12b0 to 22.1.0, https://github.com/simonw/datasette/issues/1533#issuecomment-1027633686,https://api.github.com/repos/simonw/datasette/issues/1533,1027633686,IC_kwDOBm6k_c49QHIW,9599,simonw,2022-02-02T06:42:53Z,2022-02-02T06:42:53Z,OWNER,"I'm going to apply the hack, then fix it again in: - #1518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065431383,"Add `Link: rel=""alternate""` header pointing to JSON for a table/query", https://github.com/simonw/datasette/issues/1607#issuecomment-1027634490,https://api.github.com/repos/simonw/datasette/issues/1607,1027634490,IC_kwDOBm6k_c49QHU6,9599,simonw,2022-02-02T06:44:30Z,2022-02-02T06:44:30Z,OWNER,"Prototype: ```diff diff --git a/datasette/app.py b/datasette/app.py index 09d7d03..e2a5aea 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -724,6 +724,47 @@ class Datasette: sqlite_extensions[extension] = None except Exception: pass + # More details on SpatiaLite + if ""spatialite"" in sqlite_extensions: + spatialite_details = {} + fns = ( + ""spatialite_version"", + ""spatialite_target_cpu"", + ""rcheck_strict_sql_quoting"", + ""freexl_version"", + ""proj_version"", + ""geos_version"", + ""rttopo_version"", + ""libxml2_version"", + ""HasIconv"", + ""HasMathSQL"", + ""HasGeoCallbacks"", + ""HasProj"", + ""HasProj6"", + ""HasGeos"", + ""HasGeosAdvanced"", + ""HasGeosTrunk"", + ""HasGeosReentrant"", + ""HasGeosOnlyReentrant"", + ""HasMiniZip"", + ""HasRtTopo"", + ""HasLibXML2"", + ""HasEpsg"", + ""HasFreeXL"", + ""HasGeoPackage"", + ""HasGCP"", + ""HasTopology"", + ""HasKNN"", + ""HasRouting"", + ) + for fn in fns: + try: + result = conn.execute(""select {}()"".format(fn)) + spatialite_details[fn] = result.fetchone()[0] + except Exception: + pass + sqlite_extensions[""spatialite""] = spatialite_details + # Figure out supported FTS versions ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109783030,More detailed information about installed SpatiaLite version, https://github.com/simonw/datasette/issues/1611#issuecomment-1027635175,https://api.github.com/repos/simonw/datasette/issues/1611,1027635175,IC_kwDOBm6k_c49QHfn,9599,simonw,2022-02-02T06:45:47Z,2022-02-02T06:45:47Z,OWNER,"Prototype, not sure that this actually works yet: ```diff diff --git a/datasette/database.py b/datasette/database.py index 6ce8721..0c4aec7 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -256,18 +256,26 @@ class Database: # Try to get counts for each table, $limit timeout for each count counts = {} for table in await self.table_names(): - try: - table_count = ( - await self.execute( - f""select count(*) from [{table}]"", - custom_time_limit=limit, - ) - ).rows[0][0] - counts[table] = table_count - # In some cases I saw ""SQL Logic Error"" here in addition to - # QueryInterrupted - so we catch that too: - except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError): - counts[table] = None + print(table.lower()) + if table.lower() == ""knn"": + counts[table] = 0 + else: + try: + table_count = ( + await self.execute( + f""select count(*) from [{table}]"", + custom_time_limit=limit, + ) + ).rows[0][0] + counts[table] = table_count + # In some cases I saw ""SQL Logic Error"" here in addition to + # QueryInterrupted - so we catch that too: + except ( + QueryInterrupted, + sqlite3.OperationalError, + sqlite3.DatabaseError, + ): + counts[table] = None if not self.is_mutable: self._cached_table_counts = counts return counts ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1113384383,Avoid ever running count(*) against SpatiaLite KNN table, https://github.com/simonw/datasette/issues/1576#issuecomment-1027635925,https://api.github.com/repos/simonw/datasette/issues/1576,1027635925,IC_kwDOBm6k_c49QHrV,9599,simonw,2022-02-02T06:47:20Z,2022-02-02T06:47:20Z,OWNER,"Here's what I was hacking around with when I uncovered this problem: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 77fb285..8c57d08 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import urllib import itertools import json @@ -615,44 +616,37 @@ class TableView(RowTableShared): if request.args.get(""_timelimit""): extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit"")) - # Execute the main query! - results = await db.execute(sql, params, truncate=True, **extra_args) - - # Calculate the total count for this query - filtered_table_rows_count = None - if ( - not db.is_mutable - and self.ds.inspect_data - and count_sql == f""select count(*) from {table} "" - ): - # We can use a previously cached table row count - try: - filtered_table_rows_count = self.ds.inspect_data[database][""tables""][ - table - ][""count""] - except KeyError: - pass - - # Otherwise run a select count(*) ... - if count_sql and filtered_table_rows_count is None and not nocount: - try: - count_rows = list(await db.execute(count_sql, from_sql_params)) - filtered_table_rows_count = count_rows[0][0] - except QueryInterrupted: - pass - - # Faceting - if not self.ds.setting(""allow_facet"") and any( - arg.startswith(""_facet"") for arg in request.args - ): - raise BadRequest(""_facet= is not allowed"") + async def execute_count(): + # Calculate the total count for this query + filtered_table_rows_count = None + if ( + not db.is_mutable + and self.ds.inspect_data + and count_sql == f""select count(*) from {table} "" + ): + # We can use a previously cached table row count + try: + filtered_table_rows_count = self.ds.inspect_data[database][ + ""tables"" + ][table][""count""] + except KeyError: + pass + + if count_sql and filtered_table_rows_count is None and not nocount: + try: + count_rows = list(await db.execute(count_sql, from_sql_params)) + filtered_table_rows_count = count_rows[0][0] + except QueryInterrupted: + pass + + return filtered_table_rows_count + + filtered_table_rows_count = await execute_count() # pylint: disable=no-member facet_classes = list( itertools.chain.from_iterable(pm.hook.register_facet_classes()) ) - facet_results = {} - facets_timed_out = [] facet_instances = [] for klass in facet_classes: facet_instances.append( @@ -668,33 +662,58 @@ class TableView(RowTableShared): ) ) - if not nofacet: - for facet in facet_instances: - ( - instance_facet_results, - instance_facets_timed_out, - ) = await facet.facet_results() - for facet_info in instance_facet_results: - base_key = facet_info[""name""] - key = base_key - i = 1 - while key in facet_results: - i += 1 - key = f""{base_key}_{i}"" - facet_results[key] = facet_info - facets_timed_out.extend(instance_facets_timed_out) - - # Calculate suggested facets - suggested_facets = [] - if ( - self.ds.setting(""suggest_facets"") - and self.ds.setting(""allow_facet"") - and not _next - and not nofacet - and not nosuggest - ): - for facet in facet_instances: - suggested_facets.extend(await facet.suggest()) + async def execute_suggested_facets(): + # Calculate suggested facets + suggested_facets = [] + if ( + self.ds.setting(""suggest_facets"") + and self.ds.setting(""allow_facet"") + and not _next + and not nofacet + and not nosuggest + ): + for facet in facet_instances: + suggested_facets.extend(await facet.suggest()) + return suggested_facets + + async def execute_facets(): + facet_results = {} + facets_timed_out = [] + if not self.ds.setting(""allow_facet"") and any( + arg.startswith(""_facet"") for arg in request.args + ): + raise BadRequest(""_facet= is not allowed"") + + if not nofacet: + for facet in facet_instances: + ( + instance_facet_results, + instance_facets_timed_out, + ) = await facet.facet_results() + for facet_info in instance_facet_results: + base_key = facet_info[""name""] + key = base_key + i = 1 + while key in facet_results: + i += 1 + key = f""{base_key}_{i}"" + facet_results[key] = facet_info + facets_timed_out.extend(instance_facets_timed_out) + + return facet_results, facets_timed_out + + # Execute the main query, facets and facet suggestions in parallel: + ( + results, + suggested_facets, + (facet_results, facets_timed_out), + ) = await asyncio.gather( + db.execute(sql, params, truncate=True, **extra_args), + execute_suggested_facets(), + execute_facets(), + ) + + results = await db.execute(sql, params, truncate=True, **extra_args) # Figure out columns and rows for the query columns = [r[0] for r in results.description] ``` It's a hacky attempt at running some of the table page queries in parallel to see what happens.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1619#issuecomment-1027646659,https://api.github.com/repos/simonw/datasette/issues/1619,1027646659,IC_kwDOBm6k_c49QKTD,9599,simonw,2022-02-02T07:10:37Z,2022-02-02T07:10:37Z,OWNER,It's not just the table with slashes in the name. Same thing on http://127.0.0.1:3344/foo/bar/fixtures/attraction_characteristic/1 - the `json` link goes to a JSON-rendered 404 on http://127.0.0.1:3344/foo/bar/foo/bar/fixtures/attraction_characteristic/1.json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414,JSON link on row page is 404 if base_url setting is used, https://github.com/simonw/datasette/issues/1619#issuecomment-1027647257,https://api.github.com/repos/simonw/datasette/issues/1619,1027647257,IC_kwDOBm6k_c49QKcZ,9599,simonw,2022-02-02T07:11:43Z,2022-02-02T07:11:43Z,OWNER,Weirdly the bug does NOT exhibit itself on this demo: https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1 - which correctly links to https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1.json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414,JSON link on row page is 404 if base_url setting is used, https://github.com/simonw/datasette/issues/1586#issuecomment-1027648180,https://api.github.com/repos/simonw/datasette/issues/1586,1027648180,IC_kwDOBm6k_c49QKq0,9599,simonw,2022-02-02T07:13:31Z,2022-02-02T07:13:31Z,OWNER,"Running it as part of `datasette publish` is a smart idea - I'm slightly nervous about modifying the database file that has been published though, since part of the undocumented contract right now is that the bytes served are the exact same bytes as the ones you ran the publish against. But there's no reason for that expectation to exist, and I doubt anyone is relying on that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096536240,run analyze on all databases as part of start up or publishing, https://github.com/simonw/datasette/issues/1618#issuecomment-1027653005,https://api.github.com/repos/simonw/datasette/issues/1618,1027653005,IC_kwDOBm6k_c49QL2N,9599,simonw,2022-02-02T07:22:13Z,2022-02-02T07:22:13Z,OWNER,"There's a workaround for this at the moment, which is to use parameterized SQL queries. For example, this: https://fivethirtyeight.datasettes.com/polls?sql=select+*+from+books+where+title+%3D+%3Atitle&title=The+Pragmatic+Programmer So the SQL query is `select * from books where title = :title` and then `&title=...` is added to the URL. The reason behind the quite aggressive pragma filtering is that SQLite allows you to execute pragmas using function calls, like this one: ```sql SELECT * FROM pragma_index_info('idx52'); ``` These can be nested arbitrarily deeply in sub-queries, so it's difficult to write a regular expression that will definitely catch them. I'm open to relaxing the regex a bit, but I need to be very confident that it's safe to do so. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1618#issuecomment-1027654979,https://api.github.com/repos/simonw/datasette/issues/1618,1027654979,IC_kwDOBm6k_c49QMVD,9599,simonw,2022-02-02T07:25:22Z,2022-02-02T07:25:22Z,OWNER,"But... I just noticed something I had missed in the docs for https://www.sqlite.org/pragma.html#pragfunc > Table-valued functions exist only for PRAGMAs that return results and that have no side-effects. So it's possible I'm being overly paranoid here after all: what I want to block here is people running things like `PRAGMA case_sensitive_like = 1` which could affect the global state for that connection and cause unexpected behaviour later on. So maybe I should allow all pragma functions. I previously allowed an allow-list of them in: - #761","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1618#issuecomment-1027656000,https://api.github.com/repos/simonw/datasette/issues/1618,1027656000,IC_kwDOBm6k_c49QMlA,9599,simonw,2022-02-02T07:27:14Z,2022-02-02T07:27:14Z,OWNER,"I also just realized that `pragma pragma_list` can be used to generate a list of all known pragmas for the connection: sqlite-utils fixtures.db 'pragma pragma_list' --fmt github | name | |---------------------------| | analysis_limit | | application_id | | auto_vacuum | | automatic_index | | busy_timeout | | cache_size | | cache_spill | | case_sensitive_like | | cell_size_check | | checkpoint_fullfsync | | collation_list | | compile_options | | count_changes | | data_version | | database_list | | default_cache_size | | defer_foreign_keys | | empty_result_callbacks | | encoding | | foreign_key_check | | foreign_key_list | | foreign_keys | | freelist_count | | full_column_names | | fullfsync | | function_list | | hard_heap_limit | | ignore_check_constraints | | incremental_vacuum | | index_info | | index_list | | index_xinfo | | integrity_check | | journal_mode | | journal_size_limit | | legacy_alter_table | | lock_proxy_file | | locking_mode | | max_page_count | | mmap_size | | module_list | | optimize | | page_count | | page_size | | pragma_list | | query_only | | quick_check | | read_uncommitted | | recursive_triggers | | reverse_unordered_selects | | schema_version | | secure_delete | | short_column_names | | shrink_memory | | soft_heap_limit | | synchronous | | table_info | | table_list | | table_xinfo | | temp_store | | temp_store_directory | | threads | | trusted_schema | | user_version | | wal_autocheckpoint | | wal_checkpoint | | writable_schema | So I could use that list to create a much more specific regular expression, which would then allow the word ""pragma"" to be used more freely while still protecting against any known pragma function being called.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1618#issuecomment-1027656518,https://api.github.com/repos/simonw/datasette/issues/1618,1027656518,IC_kwDOBm6k_c49QMtG,9599,simonw,2022-02-02T07:28:14Z,2022-02-02T07:31:30Z,OWNER,"I also need to consider if supposedly harmless side-effect free pragma functions could be used to work around the Datasette permissions system. My hunch is that wouldn't be a problem, because if you're allowing arbitrary SQL queries you're already letting people ignore the permissions system. One example: ``` sqlite-utils fixtures.db 'pragma database_list' -t seq name file ----- ------ ------------------------------------------------------ 0 main /Users/simon/Dropbox/Development/datasette/fixtures.db ``` Though it looks like I already allow-listed that one in #761: https://latest.datasette.io/_memory?sql=select+*+from+pragma_database_list%28%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1618#issuecomment-1027659018,https://api.github.com/repos/simonw/datasette/issues/1618,1027659018,IC_kwDOBm6k_c49QNUK,9599,simonw,2022-02-02T07:32:47Z,2022-02-02T07:32:47Z,OWNER,"I was hoping that `explain select ...` might be able to easily spot when people are calling PRAGMA functions, but this output doesn't look very helpful: ``` % sqlite-utils fixtures.db 'explain select * from pragma_database_list()' -t addr opcode p1 p2 p3 p4 p5 comment ------ ----------- ---- ---- ---- ----------------- ---- --------- 0 Init 0 11 0 0 1 VOpen 0 0 0 vtab:7F9C90AC3070 0 2 Integer 0 1 0 0 3 Integer 0 2 0 0 4 VFilter 0 10 1 0 5 VColumn 0 0 3 0 6 VColumn 0 1 4 0 7 VColumn 0 2 5 0 8 ResultRow 3 3 0 0 9 VNext 0 5 0 0 10 Halt 0 0 0 0 11 Transaction 0 0 35 0 1 12 Goto 0 1 0 0 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1615#issuecomment-1027659890,https://api.github.com/repos/simonw/datasette/issues/1615,1027659890,IC_kwDOBm6k_c49QNhy,9599,simonw,2022-02-02T07:34:17Z,2022-02-02T07:34:17Z,OWNER,"I've been thinking about this a bunch too. If I build anything along these lines it will be as part of the Datasette Cloud hosted service I'm working on, maybe as a free tier.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1117132741,Potential simplified publishing mechanism, https://github.com/simonw/datasette/issues/1533#issuecomment-1027669851,https://api.github.com/repos/simonw/datasette/issues/1533,1027669851,IC_kwDOBm6k_c49QP9b,9599,simonw,2022-02-02T07:51:57Z,2022-02-02T07:51:57Z,OWNER,"Documentation: https://docs.datasette.io/en/latest/json_api.html#discovering-the-json-for-a-page https://docs.datasette.io/en/latest/json_api.html top `--cors` section mentions the new `Access-Control-Expose-Headers: Link` header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065431383,"Add `Link: rel=""alternate""` header pointing to JSON for a table/query", https://github.com/simonw/datasette/issues/1533#issuecomment-1027672617,https://api.github.com/repos/simonw/datasette/issues/1533,1027672617,IC_kwDOBm6k_c49QQop,9599,simonw,2022-02-02T07:56:51Z,2022-02-02T07:56:51Z,OWNER,"Demos - these pages both have `<link rel=...` if you view source on them: - https://latest.datasette.io/fixtures/sortable - https://latest.datasette.io/fixtures/sortable/a,a And you can hit them with `curl` like so: ``` % curl -I 'https://latest.datasette.io/fixtures/sortable' HTTP/1.1 200 OK link: https://latest.datasette.io/fixtures/sortable.json; rel=""alternate""; type=""application/json+datasette"" cache-control: max-age=5 referrer-policy: no-referrer access-control-allow-origin: * access-control-allow-headers: Authorization access-control-expose-headers: Link content-type: text/html; charset=utf-8 x-databases: _memory, _internal, fixtures, extra_database Date: Wed, 02 Feb 2022 07:56:17 GMT Server: Google Frontend Transfer-Encoding: chunked % curl -I 'https://latest.datasette.io/fixtures/sortable/a,a' HTTP/1.1 200 OK link: https://latest.datasette.io/fixtures/sortable/a,a.json; rel=""alternate""; type=""application/json+datasette"" cache-control: max-age=5 referrer-policy: no-referrer access-control-allow-origin: * access-control-allow-headers: Authorization access-control-expose-headers: Link content-type: text/html; charset=utf-8 x-databases: _memory, _internal, fixtures, extra_database Date: Wed, 02 Feb 2022 07:56:24 GMT Server: Google Frontend Transfer-Encoding: chunked ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065431383,"Add `Link: rel=""alternate""` header pointing to JSON for a table/query", https://github.com/simonw/datasette/issues/1620#issuecomment-1028385067,https://api.github.com/repos/simonw/datasette/issues/1620,1028385067,IC_kwDOBm6k_c49S-kr,9599,simonw,2022-02-02T21:42:23Z,2022-02-02T21:42:23Z,OWNER,"``` % curl -s -I 'https://latest.datasette.io/' | grep link link: https://latest.datasette.io/.json; rel=""alternate""; type=""application/json+datasette"" % curl -s -I 'https://latest.datasette.io/fixtures' | grep link link: https://latest.datasette.io/fixtures.json; rel=""alternate""; type=""application/json+datasette"" % curl -s -I 'https://latest.datasette.io/fixtures?sql=select+1' | grep link link: https://latest.datasette.io/fixtures.json?sql=select+1; rel=""alternate""; type=""application/json+datasette"" % curl -s -I 'https://latest.datasette.io/-/plugins' | grep link link: https://latest.datasette.io/-/plugins.json; rel=""alternate""; type=""application/json+datasette"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041,"Link: rel=""alternate"" to JSON for queries too", https://github.com/simonw/datasette/issues/1620#issuecomment-1028374330,https://api.github.com/repos/simonw/datasette/issues/1620,1028374330,IC_kwDOBm6k_c49S786,9599,simonw,2022-02-02T21:28:16Z,2022-02-02T21:28:16Z,OWNER,I just realized I can refactor this to make it much simpler.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041,"Link: rel=""alternate"" to JSON for queries too", https://github.com/simonw/datasette/pull/1622#issuecomment-1028387529,https://api.github.com/repos/simonw/datasette/issues/1622,1028387529,IC_kwDOBm6k_c49S_LJ,22429695,codecov[bot],2022-02-02T21:45:21Z,2022-02-02T21:45:21Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1622](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fbaf317) into [main](https://codecov.io/gh/simonw/datasette/commit/8d5779acf0041cfd0db7f68f468419f9008b86ec?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8d5779a) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1622/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1622 +/- ## ======================================= Coverage 92.11% 92.11% ======================================= Files 34 34 Lines 4525 4525 ======================================= Hits 4168 4168 Misses 357 357 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [8d5779a...fbaf317](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122414274,Test against Python 3.11-dev, https://github.com/simonw/datasette/issues/1623#issuecomment-1028389953,https://api.github.com/repos/simonw/datasette/issues/1623,1028389953,IC_kwDOBm6k_c49S_xB,9599,simonw,2022-02-02T21:48:34Z,2022-02-02T21:48:34Z,OWNER,"A few other pages do that too, including: - https://latest.datasette.io/-/messages - https://latest.datasette.io/-/allow-debug","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122416919,/-/patterns returns link: alternate JSON header to 404, https://github.com/simonw/datasette/issues/1618#issuecomment-1028294089,https://api.github.com/repos/simonw/datasette/issues/1618,1028294089,IC_kwDOBm6k_c49SoXJ,770231,strada,2022-02-02T19:42:03Z,2022-02-02T19:42:03Z,NONE,"Thanks for looking into this. It might have been nice if `explain` surfaced these function calls. Looks like `explain query plan` does, but only for basic queries. ``` sqlite-utils fixtures.db 'explain query plan select * from pragma_function_list(), pragma_database_list(), pragma_module_list()' -t id parent notused detail ---- -------- --------- ------------------------------------------------ 4 0 0 SCAN pragma_function_list VIRTUAL TABLE INDEX 0: 8 0 0 SCAN pragma_database_list VIRTUAL TABLE INDEX 0: 12 0 0 SCAN pragma_module_list VIRTUAL TABLE INDEX 0: ``` ``` sqlite-utils fixtures.db 'explain query plan select * from pragma_function_list() as fl, pragma_database_list() as dl, pragma_module_list() as ml' -t id parent notused detail ---- -------- --------- ------------------------------ 4 0 0 SCAN fl VIRTUAL TABLE INDEX 0: 8 0 0 SCAN dl VIRTUAL TABLE INDEX 0: 12 0 0 SCAN ml VIRTUAL TABLE INDEX 0: ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305,"Reconsider policy on blocking queries containing the string ""pragma""", https://github.com/simonw/datasette/issues/1620#issuecomment-1028393259,https://api.github.com/repos/simonw/datasette/issues/1620,1028393259,IC_kwDOBm6k_c49TAkr,9599,simonw,2022-02-02T21:53:02Z,2022-02-02T21:53:02Z,OWNER,"I ran the following on https://www.google.com/ in the console to demonstrate that these work as intended: ```javascript [ ""https://latest.datasette.io/fixtures"", ""https://latest.datasette.io/fixtures?sql=select+1"", ""https://latest.datasette.io/fixtures/facetable"" ].forEach(async (url) => { response = await fetch(url, {method: ""HEAD""}); console.log(response.headers.get(""Link"")); }); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041,"Link: rel=""alternate"" to JSON for queries too", https://github.com/simonw/datasette/issues/1624#issuecomment-1028396866,https://api.github.com/repos/simonw/datasette/issues/1624,1028396866,IC_kwDOBm6k_c49TBdC,9599,simonw,2022-02-02T21:58:06Z,2022-02-02T21:58:06Z,OWNER,"It looks like this is because `IndexView` extends `BaseView` rather than extending `DataView` which is where all that CORS stuff happens: https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/datasette/views/index.py#L18-L21 Another thing I should address with the refactor project in: - #878 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122427321,Index page `/` has no CORS headers, https://github.com/simonw/datasette/issues/1623#issuecomment-1028397935,https://api.github.com/repos/simonw/datasette/issues/1623,1028397935,IC_kwDOBm6k_c49TBtv,9599,simonw,2022-02-02T21:59:43Z,2022-02-02T21:59:43Z,OWNER,Here's the new test: https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/tests/test_html.py#L927-L936,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122416919,/-/patterns returns link: alternate JSON header to 404, https://github.com/simonw/datasette/pull/1616#issuecomment-1028414871,https://api.github.com/repos/simonw/datasette/issues/1616,1028414871,IC_kwDOBm6k_c49TF2X,9599,simonw,2022-02-02T22:23:45Z,2022-02-02T22:23:45Z,OWNER,First stable Black release!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1119413338,Bump black from 21.12b0 to 22.1.0, https://github.com/simonw/datasette/pull/1617#issuecomment-1028419517,https://api.github.com/repos/simonw/datasette/issues/1617,1028419517,IC_kwDOBm6k_c49TG-9,22429695,codecov[bot],2022-02-02T22:30:26Z,2022-02-03T01:36:07Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1617](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (af293c9) into [main](https://codecov.io/gh/simonw/datasette/commit/2aa686c6554bf6b8230eb5b3019574df6cc99225?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2aa686c) will **increase** coverage by `0.06%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1617/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1617 +/- ## ========================================== + Coverage 92.09% 92.16% +0.06% ========================================== Files 34 34 Lines 4518 4531 +13 ========================================== + Hits 4161 4176 +15 + Misses 357 355 -2 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.37% <100.00%> (ø)` | | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.19% <0.00%> (ø)` | | | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.79% <0.00%> (+<0.01%)` | :arrow_up: | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.49% <0.00%> (+0.07%)` | :arrow_up: | | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `95.09% <0.00%> (+2.38%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2aa686c...af293c9](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1120990806,"Ensure template_path always uses ""/"" to match jinja", https://github.com/simonw/datasette/pull/1626#issuecomment-1028423514,https://api.github.com/repos/simonw/datasette/issues/1626,1028423514,IC_kwDOBm6k_c49TH9a,22429695,codecov[bot],2022-02-02T22:36:37Z,2022-02-02T22:39:52Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1626](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4b4d0e1) into [main](https://codecov.io/gh/simonw/datasette/commit/b5e6b1a9e1332fca3effe45d55dd06ee4249f163?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b5e6b1a) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1626/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1626 +/- ## ======================================= Coverage 92.16% 92.16% ======================================= Files 34 34 Lines 4531 4531 ======================================= Hits 4176 4176 Misses 355 355 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b5e6b1a...4b4d0e1](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122451096,Try test suite against macOS and Windows, https://github.com/simonw/datasette/pull/1626#issuecomment-1028420821,https://api.github.com/repos/simonw/datasette/issues/1626,1028420821,IC_kwDOBm6k_c49THTV,9599,simonw,2022-02-02T22:32:26Z,2022-02-02T22:33:31Z,OWNER,"That broke on a macOS test: https://github.com/simonw/datasette/runs/5044036993?check_suite_focus=true I'm going to remove macOS and Ubuntu and just try Windows purely to see what happens there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122451096,Try test suite against macOS and Windows, https://github.com/simonw/datasette/issues/1534#issuecomment-1028461220,https://api.github.com/repos/simonw/datasette/issues/1534,1028461220,IC_kwDOBm6k_c49TRKk,9599,simonw,2022-02-02T23:39:33Z,2022-02-02T23:39:33Z,OWNER,"I've decided not to do this, because of the risk that Cloudflare could cache the JSON version for an HTML page or vice-versa.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388,Maybe return JSON from HTML pages if `Accept: application/json` is sent, https://github.com/simonw/datasette/issues/1545#issuecomment-1028517268,https://api.github.com/repos/simonw/datasette/issues/1545,1028517268,IC_kwDOBm6k_c49Te2U,9599,simonw,2022-02-03T01:26:53Z,2022-02-03T01:26:53Z,OWNER,"I understand the problem now! https://github.com/pallets/jinja/issues/1378#issuecomment-812410922 > Jinja template names/paths are not always filesystem paths. So regardless of the OS Jinja always uses forward slashes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249,Custom pages don't work on windows, https://github.com/simonw/datasette/pull/1626#issuecomment-1028515161,https://api.github.com/repos/simonw/datasette/issues/1626,1028515161,IC_kwDOBm6k_c49TeVZ,9599,simonw,2022-02-03T01:22:43Z,2022-02-03T01:22:43Z,OWNER,"OK, the tests do NOT pass against Windows! https://github.com/simonw/datasette/runs/5044105941 <img width=""767"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/152265501-5b5af51b-0fae-4aa7-b86d-7b61aba24b3c.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122451096,Try test suite against macOS and Windows, https://github.com/simonw/datasette/pull/1617#issuecomment-1028517073,https://api.github.com/repos/simonw/datasette/issues/1617,1028517073,IC_kwDOBm6k_c49TezR,9599,simonw,2022-02-03T01:26:32Z,2022-02-03T01:26:32Z,OWNER,"Aha I understand the problem now! https://github.com/pallets/jinja/issues/1378#issuecomment-812410922 > Jinja template names/paths are not always filesystem paths. So regardless of the OS Jinja always uses forward slashes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1120990806,"Ensure template_path always uses ""/"" to match jinja", https://github.com/simonw/datasette/pull/1617#issuecomment-1028519382,https://api.github.com/repos/simonw/datasette/issues/1617,1028519382,IC_kwDOBm6k_c49TfXW,9599,simonw,2022-02-03T01:31:25Z,2022-02-03T01:31:25Z,OWNER,"I was hoping to get the test suite running on Windows before merging this PR but that looks like it will be a BIG job, see: - #1627 So I'm going to merge this one as-is for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1120990806,"Ensure template_path always uses ""/"" to match jinja", https://github.com/simonw/datasette/issues/1545#issuecomment-1028535868,https://api.github.com/repos/simonw/datasette/issues/1545,1028535868,IC_kwDOBm6k_c49TjY8,9599,simonw,2022-02-03T02:08:30Z,2022-02-03T02:08:30Z,OWNER,"Filed an issue with Jinja suggesting a documentation update: - https://github.com/pallets/jinja/issues/1578","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249,Custom pages don't work on windows, https://github.com/simonw/datasette/issues/1080#issuecomment-1029695083,https://api.github.com/repos/simonw/datasette/issues/1080,1029695083,IC_kwDOBm6k_c49X-Zr,9599,simonw,2022-02-04T06:24:40Z,2022-02-04T06:25:18Z,OWNER,"An initial prototype of that in my local `group-count` branch quickly started running into problems: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index be9e9c3..d30efe1 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -105,8 +105,12 @@ class RowTableShared(DataView): type_ = ""integer"" notnull = 0 else: - type_ = column_details[r[0]].type - notnull = column_details[r[0]].notnull + try: + type_ = column_details[r[0]].type + notnull = column_details[r[0]].notnull + except KeyError: # Probably count(*) + type_ = ""integer"" + notnull = False columns.append( { ""name"": r[0], @@ -613,6 +617,15 @@ class TableView(RowTableShared): offset=offset, ) + # If ?_group_count we convert the SQL query here + group_count = request.args.getlist(""_group_count"") + if group_count: + wrapped_sql = ""select {cols}, count(*) from ({sql}) group by {cols}"".format( + cols="", "".join(group_count), + sql=sql, + ) + sql = wrapped_sql + if request.args.get(""_timelimit""): extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit"")) ``` Resulted in errors like this one: ``` pk_path = path_from_row_pks(row, pks, not pks, False) File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 82, in path_from_row_pks bits = [ File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 83, in <listcomp> row[pk][""value""] if isinstance(row[pk], dict) else row[pk] for pk in pks IndexError: No item with that key ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them", https://github.com/simonw/datasette/issues/1080#issuecomment-1029691693,https://api.github.com/repos/simonw/datasette/issues/1080,1029691693,IC_kwDOBm6k_c49X9kt,9599,simonw,2022-02-04T06:16:45Z,2022-02-04T06:16:45Z,OWNER,"Had a new, different idea for how this could work: support a `?_group_count=colname` parameter to the table view, which turns the page into a `select colname, count(*) ... group by colname` query - but keeps things like the filter interface, facet selection, search box and so on.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them", https://github.com/simonw/datasette/pull/1629#issuecomment-1029980337,https://api.github.com/repos/simonw/datasette/issues/1629,1029980337,IC_kwDOBm6k_c49ZECx,22429695,codecov[bot],2022-02-04T13:21:09Z,2022-02-04T13:21:09Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1629?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1629](https://codecov.io/gh/simonw/datasette/pull/1629?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1c0d848) into [main](https://codecov.io/gh/simonw/datasette/commit/1af1041f91a9b91b321078d354132d1df5204660?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1af1041) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1629/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1629?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1629 +/- ## ======================================= Coverage 92.16% 92.16% ======================================= Files 34 34 Lines 4531 4531 ======================================= Hits 4176 4176 Misses 355 355 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1629?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1629?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1af1041...1c0d848](https://codecov.io/gh/simonw/datasette/pull/1629?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124191982,"Update pytest requirement from <6.3.0,>=5.2.2 to >=5.2.2,<7.1.0", https://github.com/simonw/datasette/issues/1576#issuecomment-1030528532,https://api.github.com/repos/simonw/datasette/issues/1576,1030528532,IC_kwDOBm6k_c49bJ4U,9599,simonw,2022-02-05T05:09:57Z,2022-02-05T05:09:57Z,OWNER,Needs documentation. I'll document `from datasette.tracer import trace` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-1030525218,https://api.github.com/repos/simonw/datasette/issues/1576,1030525218,IC_kwDOBm6k_c49bJEi,9599,simonw,2022-02-05T04:45:11Z,2022-02-05T04:45:11Z,OWNER,"Got a prototype working with `contextvars` - it identified two parallel executing queries using the patch from above: ![CleanShot 2022-02-04 at 20 41 50@2x](https://user-images.githubusercontent.com/9599/152628949-cf766b13-13cf-4831-b48d-2f23cadb6a05.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/1576#issuecomment-1030530071,https://api.github.com/repos/simonw/datasette/issues/1576,1030530071,IC_kwDOBm6k_c49bKQX,9599,simonw,2022-02-05T05:21:35Z,2022-02-05T05:21:35Z,OWNER,New documentation section: https://docs.datasette.io/en/latest/internals.html#datasette-tracer,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951,Traces should include SQL executed by subtasks created with `asyncio.gather`, https://github.com/simonw/datasette/issues/957#issuecomment-1030762140,https://api.github.com/repos/simonw/datasette/issues/957,1030762140,IC_kwDOBm6k_c49cC6c,9599,simonw,2022-02-06T06:36:41Z,2022-02-06T06:36:41Z,OWNER,Documented here: https://docs.datasette.io/en/latest/internals.html#import-shortcuts,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes, https://github.com/simonw/datasette/issues/1176#issuecomment-1030762279,https://api.github.com/repos/simonw/datasette/issues/1176,1030762279,IC_kwDOBm6k_c49cC8n,9599,simonw,2022-02-06T06:38:08Z,2022-02-06T06:41:37Z,OWNER,"Might do this using Sphinx auto-generated function and class documentation hooks, as seen here in `sqlite-utils`: https://sqlite-utils.datasette.io/en/stable/python-api.html#spatialite-helpers This would encourage me to add really good docstrings. ``` .. _python_api_gis_find_spatialite: Finding SpatiaLite ------------------ .. autofunction:: sqlite_utils.utils.find_spatialite ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/957#issuecomment-1030761625,https://api.github.com/repos/simonw/datasette/issues/957,1030761625,IC_kwDOBm6k_c49cCyZ,9599,simonw,2022-02-06T06:30:32Z,2022-02-06T06:31:44Z,OWNER,"I'm just going with: ```python from datasette import Response from datasette import Forbidden from datasette import NotFound from datasette import hookimpl from datasette import actor_matches_allow ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes, https://github.com/simonw/datasette/issues/932#issuecomment-1030940407,https://api.github.com/repos/simonw/datasette/issues/932,1030940407,IC_kwDOBm6k_c49cub3,9599,simonw,2022-02-06T23:31:22Z,2022-02-06T23:31:22Z,OWNER,"Great argument for doing this from a conversation on Twitter about documentation-driven development: > Long ago, when the majority of commercial programs were desktop apps, I've read a very wise advice: The user manual should be written first, before even a single line if code. https://twitter.com/b11c/status/1490466703175823362","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation, https://github.com/simonw/datasette/issues/1176#issuecomment-1031108559,https://api.github.com/repos/simonw/datasette/issues/1176,1031108559,IC_kwDOBm6k_c49dXfP,9599,simonw,2022-02-07T06:11:27Z,2022-02-07T06:11:27Z,OWNER,I'm going with `@documented` as the decorator for functions that should be documented.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031122800,https://api.github.com/repos/simonw/datasette/issues/1176,1031122800,IC_kwDOBm6k_c49da9w,9599,simonw,2022-02-07T06:34:21Z,2022-02-07T06:34:21Z,OWNER,"New section is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module But it's not correctly displaying the new autodoc stuff: <img width=""688"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/152736915-e6470e0a-05e2-4612-a2ff-ebd4f51b65cb.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031126547,https://api.github.com/repos/simonw/datasette/issues/1176,1031126547,IC_kwDOBm6k_c49db4T,9599,simonw,2022-02-07T06:42:58Z,2022-02-07T06:42:58Z,OWNER,"That fixed it: https://docs.datasette.io/en/latest/internals.html#parse-metadata-content <img width=""695"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/152737872-595e9a14-1a0d-47ff-9509-63ee765c0c5d.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031126801,https://api.github.com/repos/simonw/datasette/issues/1176,1031126801,IC_kwDOBm6k_c49db8R,9599,simonw,2022-02-07T06:43:31Z,2022-02-07T06:43:31Z,OWNER,Here's the new test: https://github.com/simonw/datasette/blob/03305ea183b1534bc4cef3a721fe5f3700273b84/tests/test_docs.py#L91-L104,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031123719,https://api.github.com/repos/simonw/datasette/issues/1176,1031123719,IC_kwDOBm6k_c49dbMH,9599,simonw,2022-02-07T06:36:32Z,2022-02-07T06:36:32Z,OWNER,"https://github.com/simonw/sqlite-utils/blob/main/.readthedocs.yaml looks like this (it works correctly): ```yaml version: 2 sphinx: configuration: docs/conf.py python: version: ""3.8"" install: - method: pip path: . extra_requirements: - docs ``` Compare to the current Datasette one here: https://github.com/simonw/datasette/blob/d9b508ffaa91f9f1840b366f5d282712d445f16b/.readthedocs.yaml#L1-L13 Looks like I need this bit: ```python python: version: ""3.8"" install: - method: pip path: . extra_requirements: - docs ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1176#issuecomment-1031125347,https://api.github.com/repos/simonw/datasette/issues/1176,1031125347,IC_kwDOBm6k_c49dblj,9599,simonw,2022-02-07T06:40:16Z,2022-02-07T06:40:16Z,OWNER,"Read The Docs error: > Problem in your project's configuration. Invalid ""python.version"": .readthedocs.yaml: Invalid configuration option: python.version. Make sure the key name is correct.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions", https://github.com/simonw/datasette/issues/1439#issuecomment-1031141849,https://api.github.com/repos/simonw/datasette/issues/1439,1031141849,IC_kwDOBm6k_c49dfnZ,9599,simonw,2022-02-07T07:11:11Z,2022-02-07T07:11:11Z,OWNER,"I added a Link header to solve this problem for the JSON version in: - #1533 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/pull/1593#issuecomment-1031455498,https://api.github.com/repos/simonw/datasette/issues/1593,1031455498,IC_kwDOBm6k_c49esMK,49699333,dependabot[bot],2022-02-07T13:13:22Z,2022-02-07T13:13:22Z,CONTRIBUTOR,Superseded by #1631.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1101705012,"Update pytest-asyncio requirement from <0.17,>=0.10 to >=0.10,<0.18", https://github.com/simonw/datasette/pull/1631#issuecomment-1031463789,https://api.github.com/repos/simonw/datasette/issues/1631,1031463789,IC_kwDOBm6k_c49euNt,22429695,codecov[bot],2022-02-07T13:21:48Z,2022-02-07T13:21:48Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1631?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1631](https://codecov.io/gh/simonw/datasette/pull/1631?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (62eed84) into [main](https://codecov.io/gh/simonw/datasette/commit/03305ea183b1534bc4cef3a721fe5f3700273b84?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (03305ea) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1631/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1631?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1631 +/- ## ======================================= Coverage 92.19% 92.19% ======================================= Files 34 34 Lines 4546 4546 ======================================= Hits 4191 4191 Misses 355 355 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1631?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1631?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [03305ea...62eed84](https://codecov.io/gh/simonw/datasette/pull/1631?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125973221,"Update pytest-asyncio requirement from <0.17,>=0.10 to >=0.10,<0.19", https://github.com/simonw/datasette/issues/1632#issuecomment-1032032686,https://api.github.com/repos/simonw/datasette/issues/1632,1032032686,IC_kwDOBm6k_c49g5Gu,9599,simonw,2022-02-07T23:16:10Z,2022-02-07T23:16:10Z,OWNER,"I found this bug while trying to get the following to work: datasette /data/one.db /data/two.db /data/*.db --create I want this to create any missing database files on startup out of that literal list of `one.db` and `two.db` and to also open any other `*.db` files in that folder - needed for `datasette-publish-fly` in https://github.com/simonw/datasette-publish-fly/pull/12#issuecomment-1032029874","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032034015,https://api.github.com/repos/simonw/datasette/issues/1632,1032034015,IC_kwDOBm6k_c49g5bf,9599,simonw,2022-02-07T23:17:57Z,2022-02-07T23:17:57Z,OWNER,I'm going to fix this in a 0.60.2 bug fix release.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032036525,https://api.github.com/repos/simonw/datasette/issues/1632,1032036525,IC_kwDOBm6k_c49g6Ct,9599,simonw,2022-02-07T23:19:59Z,2022-02-07T23:19:59Z,OWNER,"I'm going to fix this in the CLI code itself, rather than fixing it in the `Datasette` constructor. That way if someone has a truly weird reason to want this behaviour they can construct Datasette directly. https://github.com/simonw/datasette/blob/03305ea183b1534bc4cef3a721fe5f3700273b84/datasette/cli.py#L535-L550","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032037391,https://api.github.com/repos/simonw/datasette/issues/1632,1032037391,IC_kwDOBm6k_c49g6QP,9599,simonw,2022-02-07T23:21:07Z,2022-02-07T23:21:07Z,OWNER,"For the record, here's the code that picks the `one_2` name if that stem is already used as a database name: https://github.com/simonw/datasette/blob/03305ea183b1534bc4cef3a721fe5f3700273b84/datasette/app.py#L401-L417","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032050489,https://api.github.com/repos/simonw/datasette/issues/1632,1032050489,IC_kwDOBm6k_c49g9c5,9599,simonw,2022-02-07T23:39:11Z,2022-02-07T23:42:08Z,OWNER,"That implementation broke on Python 3.6 - which is still a supported Python version for the 0.60.x branch - `test_homepage` failed. ``` > assert ( ""2 rows in 1 table, 5 rows in 4 hidden tables, 1 view"" == counts_p.text.strip() ) E AssertionError: assert '2 rows in 1 ...ables, 1 view' == '1 table, 4 h...ables, 1 view' E - 1 table, 4 hidden tables, 1 view E + 2 rows in 1 table, 5 rows in 4 hidden tables, 1 view E ? ++++++++++ ++++++++++ ``` That's because this idiom isn't guaranteed to preserve order in versions earlier than Python 3.7: https://github.com/simonw/datasette/blob/fa5fc327adbbf70656ac533912f3fc0526a3873d/datasette/cli.py#L552-L553 I could say that `0.60.2` is the first version to require Python 3.7 - but that feels a little surprising. I'm going to use a different idiom for order-preserving de-duplication from [this StackOverflow](https://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-whilst-preserving-order) instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1632#issuecomment-1032057472,https://api.github.com/repos/simonw/datasette/issues/1632,1032057472,IC_kwDOBm6k_c49g_KA,9599,simonw,2022-02-07T23:50:01Z,2022-02-07T23:50:01Z,OWNER,Released in https://github.com/simonw/datasette/releases/tag/0.60.2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126604194,"datasette one.db one.db opens database twice, as one and one_2", https://github.com/simonw/datasette/issues/1607#issuecomment-1033403664,https://api.github.com/repos/simonw/datasette/issues/1607,1033403664,IC_kwDOBm6k_c49mH0Q,9599,simonw,2022-02-09T06:42:02Z,2022-02-09T06:42:02Z,OWNER,"Deployed a new build of https://github.com/simonw/calands-datasette/actions/workflows/build-and-deploy.yml for a live demo: https://calands.datasettes.com/-/versions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109783030,More detailed information about installed SpatiaLite version, https://github.com/simonw/datasette/issues/236#issuecomment-1033772902,https://api.github.com/repos/simonw/datasette/issues/236,1033772902,IC_kwDOBm6k_c49nh9m,1376648,jordaneremieff,2022-02-09T13:40:52Z,2022-02-09T13:40:52Z,NONE,"Hi @simonw, I've received some inquiries over the last year or so about Datasette and how it might be supported by [Mangum](https://github.com/jordaneremieff/mangum). I maintain Mangum which is, as far as I know, the only project that provides support for ASGI applications in AWS Lambda. If there is anything that I can help with here, please let me know because I think what Datasette provides to the community (even beyond OSS) is noble and worthy of special consideration.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin, https://github.com/simonw/datasette/issues/1633#issuecomment-1034222709,https://api.github.com/repos/simonw/datasette/issues/1633,1034222709,IC_kwDOBm6k_c49pPx1,6613091,henrikek,2022-02-09T21:47:02Z,2022-02-09T21:47:02Z,NONE,Is this the correct solution to add the base_url row to url_builder.py?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1129052172,base_url or prefix does not work with _exact match, https://github.com/simonw/datasette/issues/1634#issuecomment-1035664928,https://api.github.com/repos/simonw/datasette/issues/1634,1035664928,IC_kwDOBm6k_c49uv4g,9599,simonw,2022-02-11T00:10:07Z,2022-02-11T00:10:23Z,OWNER,Could also bump this up to Python 3.10: https://github.com/simonw/datasette/blob/5619069968ab39fd44c44a1888965e361c6f7fb9/Dockerfile#L1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1131295060,Update Dockerfile generated by `datasette publish`, https://github.com/simonw/datasette/issues/1634#issuecomment-1035664412,https://api.github.com/repos/simonw/datasette/issues/1634,1035664412,IC_kwDOBm6k_c49uvwc,9599,simonw,2022-02-11T00:09:18Z,2022-02-11T00:09:18Z,OWNER,Starting it with `FROM datasetteproject/datasette` might be a good idea. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1131295060,Update Dockerfile generated by `datasette publish`, https://github.com/simonw/datasette/issues/1634#issuecomment-1035667060,https://api.github.com/repos/simonw/datasette/issues/1634,1035667060,IC_kwDOBm6k_c49uwZ0,9599,simonw,2022-02-11T00:13:22Z,2022-02-11T00:13:22Z,OWNER,Looks like `3.10.2` is the latest: https://hub.docker.com/_/python?tab=tags&page=1&name=3.10.2-slim-bu,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1131295060,Update Dockerfile generated by `datasette publish`, https://github.com/simonw/datasette/issues/1660#issuecomment-1073355032,https://api.github.com/repos/simonw/datasette/issues/1660,1073355032,IC_kwDOBm6k_c4_-hkY,9599,simonw,2022-03-20T21:46:43Z,2022-03-20T21:46:43Z,OWNER,I think the way to get rid of most of the remaining complexity in `DataView` is to refactor how CSV stuff works - pulling it in line with other export factors and extracting the streaming mechanism. Opening a fresh issue for that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1672#issuecomment-1073355818,https://api.github.com/repos/simonw/datasette/issues/1672,1073355818,IC_kwDOBm6k_c4_-hwq,9599,simonw,2022-03-20T21:52:38Z,2022-03-20T21:52:38Z,OWNER,"That means taking on these issues: - https://github.com/simonw/datasette/issues/1101 - https://github.com/simonw/datasette/issues/1096 - https://github.com/simonw/datasette/issues/1062","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174697144,Refactor CSV handling code out of DataView, https://github.com/simonw/datasette/issues/1673#issuecomment-1073361986,https://api.github.com/repos/simonw/datasette/issues/1673,1073361986,IC_kwDOBm6k_c4_-jRC,9599,simonw,2022-03-20T22:31:41Z,2022-03-20T22:34:06Z,OWNER,"Maybe it's because `supports_table_xinfo()` creates a brand new in-memory SQLite connection every time you call it? https://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/sqlite.py#L22-L35 Actually no, I'm caching that already: https://github.com/simonw/datasette/blob/798f075ef9b98819fdb564f9f79c78975a0f71e8/datasette/utils/sqlite.py#L12-L19","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174708375,Streaming CSV spends a lot of time in `table_column_details`, https://github.com/simonw/datasette/issues/1355#issuecomment-1073362979,https://api.github.com/repos/simonw/datasette/issues/1355,1073362979,IC_kwDOBm6k_c4_-jgj,9599,simonw,2022-03-20T22:38:53Z,2022-03-20T22:38:53Z,OWNER,"Built a research prototype: ```diff diff --git a/datasette/app.py b/datasette/app.py index 5c8101a..5cd3e63 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1,6 +1,7 @@ import asyncio import asgi_csrf import collections +import contextlib import datetime import functools import glob @@ -1490,3 +1491,11 @@ class DatasetteClient: return await client.request( method, self._fix(path, avoid_path_rewrites), **kwargs ) + + @contextlib.asynccontextmanager + async def stream(self, method, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + print(""async with as client"") + async with client.stream(method, self._fix(path), **kwargs) as response: + print(""async with client.stream about to yield response"") + yield response diff --git a/datasette/cli.py b/datasette/cli.py index 3c6e1b2..3025ead 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -585,11 +585,19 @@ def serve( asyncio.get_event_loop().run_until_complete(check_databases(ds)) if get: - client = TestClient(ds) - response = client.get(get) - click.echo(response.text) - exit_code = 0 if response.status == 200 else 1 - sys.exit(exit_code) + + async def _run_get(): + print(""_run_get"") + async with ds.client.stream(""GET"", get) as response: + print(""Got response:"", response) + async for chunk in response.aiter_bytes(chunk_size=1024): + print("" chunk"") + sys.stdout.buffer.write(chunk) + sys.stdout.buffer.flush() + exit_code = 0 if response.status_code == 200 else 1 + sys.exit(exit_code) + + asyncio.get_event_loop().run_until_complete(_run_get()) return # Start the server ``` But for some reason it didn't appear to stream out the response - it would print this out: ``` % datasette covid.db --get '/covid/ny_times_us_counties.csv?_size=10&_stream=on' _run_get async with as client ``` And then hang. I would expect it to start printing out chunks of CSV data here, but instead it looks like it waited for everything to be generated before returning anything to the console. No idea why. I dropped this for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910088936,datasette --get should efficiently handle streaming CSV, https://github.com/simonw/datasette/issues/1674#issuecomment-1073366436,https://api.github.com/repos/simonw/datasette/issues/1674,1073366436,IC_kwDOBm6k_c4_-kWk,9599,simonw,2022-03-20T22:58:40Z,2022-03-20T22:58:40Z,OWNER,"This will probably happen as part of turning this into an officially documented API that serves the template context for the homepage: - #1510","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174717287,Tweak design of /.json, https://github.com/simonw/datasette/issues/1510#issuecomment-1073366630,https://api.github.com/repos/simonw/datasette/issues/1510,1073366630,IC_kwDOBm6k_c4_-kZm,9599,simonw,2022-03-20T22:59:33Z,2022-03-20T22:59:33Z,OWNER,"I really like the idea of making this effectively the same thing as the fully documented, stable JSON API that comes as part of 1.0. If you want to know what will be available to your templates, consult the API documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054244712,Datasette 1.0 documented template context (maybe via API docs), https://github.com/simonw/datasette/issues/1663#issuecomment-1071519407,https://api.github.com/repos/simonw/datasette/issues/1663,1071519407,IC_kwDOBm6k_c4_3hav,9599,simonw,2022-03-17T21:32:35Z,2022-03-17T21:32:35Z,OWNER,"Updated docs: - https://docs.datasette.io/en/latest/internals.html#datasette-class - https://docs.datasette.io/en/latest/internals.html#db-hash","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170554975,Document the internals that were used in datasette-hashed-urls, https://github.com/simonw/datasette/issues/1661#issuecomment-1071706993,https://api.github.com/repos/simonw/datasette/issues/1661,1071706993,IC_kwDOBm6k_c4_4PNx,9599,simonw,2022-03-17T22:42:21Z,2022-03-17T22:42:21Z,OWNER,"As part of this I'm going to get rid of this mechanism: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/base.py#L170-L173 Unwrapping `request.scope[""url_route""][""kwargs""]` into keyword argument to view functions just made the code harder to follow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1071793307,https://api.github.com/repos/simonw/datasette/issues/1661,1071793307,IC_kwDOBm6k_c4_4kSb,9599,simonw,2022-03-17T23:17:32Z,2022-03-17T23:17:32Z,OWNER,"Surprisingly I managed to break https://latest.datasette.io/fixtures/custom_foreign_key_label while working on this change: ![CleanShot 2022-03-17 at 16 16 54@2x](https://user-images.githubusercontent.com/9599/158909271-717b65e8-cfcc-44c4-b1cc-f34478b0f803.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1071797707,https://api.github.com/repos/simonw/datasette/issues/1661,1071797707,IC_kwDOBm6k_c4_4lXL,9599,simonw,2022-03-17T23:19:24Z,2022-03-17T23:19:24Z,OWNER,"Moving this to PR so I can comment on individual lines: - #1664","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1071803114,https://api.github.com/repos/simonw/datasette/issues/1664,1071803114,IC_kwDOBm6k_c4_4mrq,9599,simonw,2022-03-17T23:22:00Z,2022-03-17T23:22:00Z,OWNER,"Surprisingly I managed to break https://latest.datasette.io/fixtures/custom_foreign_key_label while working on this change: ![CleanShot 2022-03-17 at 16 16 54@2x](https://user-images.githubusercontent.com/9599/158909271-717b65e8-cfcc-44c4-b1cc-f34478b0f803.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1071809988,https://api.github.com/repos/simonw/datasette/issues/1664,1071809988,IC_kwDOBm6k_c4_4oXE,9599,simonw,2022-03-17T23:24:57Z,2022-03-17T23:24:57Z,OWNER,"My hunch is that this is broken because of this: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/app.py#L1098-L1107 Note how the table uses `table_and_format` but the row uses just `table` - I think there's code that's getting confused by this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1071813296,https://api.github.com/repos/simonw/datasette/issues/1664,1071813296,IC_kwDOBm6k_c4_4pKw,9599,simonw,2022-03-17T23:26:22Z,2022-03-17T23:26:22Z,OWNER,Probably caused by the convoluted code is `get_format()`: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/base.py#L466-L481,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/issues/1605#issuecomment-1072907200,https://api.github.com/repos/simonw/datasette/issues/1605,1072907200,IC_kwDOBm6k_c4_80PA,9599,simonw,2022-03-19T00:52:54Z,2022-03-19T00:53:45Z,OWNER,"Had a thought about the implementation of this: it could make a really neat plugin. Something like `datasette-export` which adds a `export` command using https://docs.datasette.io/en/stable/plugin_hooks.html#register-commands-cli - then you could run: datasette export my-export-dir mydatabase.db -m metadata.json --template-dir templates/ And the command would then: - Create a `Datasette()` instance with those databases/metadata/etc - Execute`await datasette.client.get(""/"")` to get the homepage HTML - Parse the HTML using BeautifulSoup to find all `a[href]`, `link[href]`, `script[src]`, `img[src]` elements that reference a relative path as opposed to one that starts with `http://` - Write out the homepage to `my-export-dir/index.html` - Recursively fetch and dump all of the other pages and assets that it found too All of that HTML parsing may be over-complicating things. It could alternatively accept options for which pages you want to export: ``` datasette export my-export-dir \ mydatabase.db -m metadata.json --template-dir templates/ \ --path / \ --path /mydatabase ... ``` Or a really wild option: it could allow you to define the paths you want to export using a SQL query: ``` datasette export my-export-dir \ mydatabase.db -m metadata.json --template-dir templates/ \ --sql "" select '/' as path, 'index.html' as filename union all select '/mydatabase/articles/' || id as path, 'article-' || id || '.html' as filename from articles union all select '/mydatabase/tags/' || tag as path, 'tag-' || tag || '.html' as filename from tags "" ``` Which would save these files: - `index.html` as the content of `/` - `article-1.html` (and more) as the content of `/mydatabase/articles/1` - `tag-python.html` (and more) as the content of `/mydatabase/tags/python`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1228#issuecomment-1072907610,https://api.github.com/repos/simonw/datasette/issues/1228,1072907610,IC_kwDOBm6k_c4_80Va,9599,simonw,2022-03-19T00:55:29Z,2022-03-19T00:55:29Z,OWNER,"It looks to me like something is causing the faceting query here to return a string when it was expected to return a number: https://github.com/simonw/datasette/blob/32963018e7edfab1233de7c7076c428d0e5c7813/datasette/facets.py#L153-L170 I can't think of any way that a `count(*) as n` would turn into a string though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/issues/1228#issuecomment-1072907680,https://api.github.com/repos/simonw/datasette/issues/1228,1072907680,IC_kwDOBm6k_c4_80Wg,9599,simonw,2022-03-19T00:55:48Z,2022-03-19T00:55:48Z,OWNER,... unless your data had a column called `n`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/issues/1228#issuecomment-1072908029,https://api.github.com/repos/simonw/datasette/issues/1228,1072908029,IC_kwDOBm6k_c4_80b9,9599,simonw,2022-03-19T00:57:54Z,2022-03-19T00:57:54Z,OWNER,"Yes! That's the problem. I was able to replicate it like so: ``` echo '[{ ""n"": ""one"", ""abc"": 1 }, { ""n"": ""one"", ""abc"": 2 }, { ""n"": ""two"", ""abc"": 3 }]' | sqlite-utils insert column-called-n.db t - ``` <img width=""564"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/159100494-9a072e3c-7bad-4fc5-b90e-b5078c11fc44.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/issues/1228#issuecomment-1072915936,https://api.github.com/repos/simonw/datasette/issues/1228,1072915936,IC_kwDOBm6k_c4_82Xg,9599,simonw,2022-03-19T01:50:27Z,2022-03-19T01:50:27Z,OWNER,Demo: https://latest.datasette.io/fixtures/facetable - which now has a column called `n`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/issues/1666#issuecomment-1072933875,https://api.github.com/repos/simonw/datasette/issues/1666,1072933875,IC_kwDOBm6k_c4_86vz,9599,simonw,2022-03-19T04:03:42Z,2022-03-19T04:03:42Z,OWNER,Tests so far: https://github.com/simonw/datasette/blob/711767bcd3c1e76a0861fe7f24069ff1c8efc97a/tests/test_routes.py#L12-L34,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174162781,Refactor URL routing to enable testing, https://github.com/simonw/datasette/issues/1561#issuecomment-1072939780,https://api.github.com/repos/simonw/datasette/issues/1561,1072939780,IC_kwDOBm6k_c4_88ME,9599,simonw,2022-03-19T04:45:40Z,2022-03-19T04:45:40Z,OWNER,"I ended up moving hashed URL mode out to a plugin in: - #647 If you're still interested in using it with `_memory` please open an issue in that repo here: https://github.com/simonw/datasette-hashed-urls","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082765654,"add hash id to ""_memory"" url if hashed url mode is turned on and crossdb is also turned on", https://github.com/simonw/datasette/issues/1228#issuecomment-1072954795,https://api.github.com/repos/simonw/datasette/issues/1228,1072954795,IC_kwDOBm6k_c4_8_2r,7107523,Kabouik,2022-03-19T06:44:40Z,2022-03-19T06:44:40Z,NONE,"> ... unless your data had a column called `n`? Exactly, that's highly likely even though I can't double check from this computer just now. Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810397025,500 error caused by faceting if a column called `n` exists, https://github.com/simonw/datasette/pull/1664#issuecomment-1072890205,https://api.github.com/repos/simonw/datasette/issues/1664,1072890205,IC_kwDOBm6k_c4_8wFd,9599,simonw,2022-03-18T23:43:15Z,2022-03-18T23:43:15Z,OWNER,"Now almost everything is working except for foreign key expansion: ![CleanShot 2022-03-18 at 16 41 39@2x](https://user-images.githubusercontent.com/9599/159097349-6f41dfdf-5bab-449b-a148-5cda3df6534c.png) Using the debugger I tracked it down to this code: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/table.py#L708-L715 Turns out `default_labels` there is `None` - and it's a parameter to that `data()` method: https://github.com/simonw/datasette/blob/30e5f0e67c38054a8087a2a4eae3fc4d1779af90/datasette/views/table.py#L325-L334 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1072890524,https://api.github.com/repos/simonw/datasette/issues/1664,1072890524,IC_kwDOBm6k_c4_8wKc,9599,simonw,2022-03-18T23:44:33Z,2022-03-19T00:06:51Z,OWNER,Looks like that was set here: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L490-L492,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1072898797,https://api.github.com/repos/simonw/datasette/issues/1664,1072898797,IC_kwDOBm6k_c4_8yLt,9599,simonw,2022-03-19T00:11:09Z,2022-03-19T00:11:09Z,OWNER,Still need to remove it from the documentation and do something about that `hash_urls` setting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/pull/1664#issuecomment-1072898923,https://api.github.com/repos/simonw/datasette/issues/1664,1072898923,IC_kwDOBm6k_c4_8yNr,9599,simonw,2022-03-19T00:11:33Z,2022-03-19T00:11:33Z,OWNER,I'm going to land this and handle those in separate commits.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173017980,Remove hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1072901159,https://api.github.com/repos/simonw/datasette/issues/1661,1072901159,IC_kwDOBm6k_c4_8ywn,9599,simonw,2022-03-19T00:20:27Z,2022-03-19T00:20:27Z,OWNER,I can remove the `default_cache_ttl_hashed` setting too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1072904703,https://api.github.com/repos/simonw/datasette/issues/1661,1072904703,IC_kwDOBm6k_c4_8zn_,9599,simonw,2022-03-19T00:37:36Z,2022-03-19T00:37:36Z,OWNER,Updated docs: https://docs.datasette.io/en/latest/performance.html#datasette-hashed-urls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1662#issuecomment-1072905467,https://api.github.com/repos/simonw/datasette/issues/1662,1072905467,IC_kwDOBm6k_c4_8zz7,9599,simonw,2022-03-19T00:42:23Z,2022-03-19T00:42:23Z,OWNER,"Those client-side SQLite tricks are _really_ neat. `datasette publish` defaults to configuring it so the raw SQLite database can be downloaded from `/fixtures.db` - and this issue updated it to be served with a CORS header that would allow client-side scripts to load the file: - #1057 If you're not going to run any server-side code at all you don't need Datasette for this - you can upload the SQLite database file to any static hosting with CORS headers and load it into the client that way. In terms of static publishing, I do think there's something interesting about using Datasette to generate static sites. There's an issue discussing options for that over here: - #1605","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170497629,[feature request] Publish to fully static website, https://github.com/simonw/datasette/issues/878#issuecomment-1073037939,https://api.github.com/repos/simonw/datasette/issues/878,1073037939,IC_kwDOBm6k_c4_9UJz,9599,simonw,2022-03-19T16:19:30Z,2022-03-19T16:19:30Z,OWNER,"On revisiting https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2 a few months later I'm having second thoughts about using `@inject` on the `main()` method. But I still like the pattern as a way to resolve more complex cases like ""to generate GeoJSON of the expanded view with labels, the label expansion code needs to run once at some before the GeoJSON formatting code does"". So I'm going to stick with it a tiny bit longer, but maybe try to make it a lot more explicit when it's going to happen rather than having the main view methods themselves also use async DI.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins", https://github.com/simonw/datasette/issues/1666#issuecomment-1073039241,https://api.github.com/repos/simonw/datasette/issues/1666,1073039241,IC_kwDOBm6k_c4_9UeJ,9599,simonw,2022-03-19T16:28:15Z,2022-03-19T16:28:15Z,OWNER,This is more interesting if it also asserts against the captured matches from the pattern.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174162781,Refactor URL routing to enable testing, https://github.com/simonw/datasette/issues/1666#issuecomment-1073039670,https://api.github.com/repos/simonw/datasette/issues/1666,1073039670,IC_kwDOBm6k_c4_9Uk2,9599,simonw,2022-03-19T16:31:08Z,2022-03-19T16:31:57Z,OWNER,"This does make it more interesting - it also highlights how inconsistent the way the capturing works is. Especially `as_format` which can be `None` or `""""` or `.json` or `json` or not used at all in the case of `TableView`. https://github.com/simonw/datasette/blob/764738dfcb16cd98b0987d443f59d5baa9d3c332/tests/test_routes.py#L12-L36","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174162781,Refactor URL routing to enable testing, https://github.com/simonw/datasette/issues/1667#issuecomment-1073040072,https://api.github.com/repos/simonw/datasette/issues/1667,1073040072,IC_kwDOBm6k_c4_9UrI,9599,simonw,2022-03-19T16:34:02Z,2022-03-19T16:34:02Z,OWNER,"I called it `as_format` to avoid clashing with the Python built-in `format()` function when these things were turned into keyword arguments, but now that they're not I can use `format` instead. I think I'm going to go with `database`, `table`, `format` and `pks`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174302994,Make route matched pattern groups more consistent, https://github.com/simonw/datasette/issues/1667#issuecomment-1073042554,https://api.github.com/repos/simonw/datasette/issues/1667,1073042554,IC_kwDOBm6k_c4_9VR6,9599,simonw,2022-03-19T16:50:01Z,2022-03-19T16:52:35Z,OWNER,"OK, I've made this more consistent - I still need to address the fact that `format` can be `.json` or `json` or not used at all before I close this issue. https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/tests/test_routes.py#L15-L35","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174302994,Make route matched pattern groups more consistent, https://github.com/simonw/datasette/issues/1668#issuecomment-1073043350,https://api.github.com/repos/simonw/datasette/issues/1668,1073043350,IC_kwDOBm6k_c4_9VeW,9599,simonw,2022-03-19T16:54:26Z,2022-03-19T16:54:26Z,OWNER,"The `Database` class already has a `path` property but it means something else - it's the path to the `.db` file on disk: https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/database.py#L29-L50 So need a different name for the path-that-is-used-in-the-URL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073043433,https://api.github.com/repos/simonw/datasette/issues/1668,1073043433,IC_kwDOBm6k_c4_9Vfp,9599,simonw,2022-03-19T16:54:55Z,2022-03-19T20:01:19Z,OWNER,"Options: - `route_path` - `url_path` - `route` I like `route_path`, or maybe `route`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073043713,https://api.github.com/repos/simonw/datasette/issues/1668,1073043713,IC_kwDOBm6k_c4_9VkB,9599,simonw,2022-03-19T16:56:19Z,2022-03-19T16:56:19Z,OWNER,"Worth noting that the `name` right now is picked automatically to avoid conflicts: https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/app.py#L397-L413","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073073547,https://api.github.com/repos/simonw/datasette/issues/1668,1073073547,IC_kwDOBm6k_c4_9c2L,9599,simonw,2022-03-19T20:06:07Z,2022-03-19T20:06:07Z,OWNER,"Implementing this is a little tricky because there's a whole lot of code that expects the `database` captured by the URL routing to be the name used to look up the database in `datasette.databases` - or via `.get_database()`. The `DataView.get()` method is a good example of the trickyness here. It even has code that dispatches out to plugin hooks that take `database` as a parameter. https://github.com/simonw/datasette/blob/61419388c134001118aaf7dfb913562d467d7913/datasette/views/base.py#L383-L555 All the more reason to get rid of that `BaseView -> DataView -> TableView` hierarchy entirely: - #1660","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1660#issuecomment-1073073599,https://api.github.com/repos/simonw/datasette/issues/1660,1073073599,IC_kwDOBm6k_c4_9c2_,9599,simonw,2022-03-19T20:06:40Z,2022-03-19T20:06:40Z,OWNER,"This blocks: - #1668","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1668#issuecomment-1073073579,https://api.github.com/repos/simonw/datasette/issues/1668,1073073579,IC_kwDOBm6k_c4_9c2r,9599,simonw,2022-03-19T20:06:27Z,2022-03-19T20:06:27Z,OWNER,Marking this as blocked until #1660 is done.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073075697,https://api.github.com/repos/simonw/datasette/issues/1668,1073075697,IC_kwDOBm6k_c4_9dXx,9599,simonw,2022-03-19T20:24:06Z,2022-03-19T20:24:06Z,OWNER,"Right now if a database has a `.` in its name e.g. `fixtures.dot` the URL to that database is: /fixtures~2Edot But the output on `/-/databases` doesn't reflect that, it still shows the name with the dot.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073075913,https://api.github.com/repos/simonw/datasette/issues/1668,1073075913,IC_kwDOBm6k_c4_9dbJ,9599,simonw,2022-03-19T20:25:46Z,2022-03-19T20:26:08Z,OWNER,"The output of `/.json` DOES use `path` to mean the URL path, not the path to the file on disk: ``` { ""fixtures.dot"": { ""name"": ""fixtures.dot"", ""hash"": null, ""color"": ""631f11"", ""path"": ""/fixtures~2Edot"", ``` So that's a problem already: having `db.path` refer to something different from that JSON is inconsistent.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073076015,https://api.github.com/repos/simonw/datasette/issues/1668,1073076015,IC_kwDOBm6k_c4_9dcv,9599,simonw,2022-03-19T20:26:32Z,2022-03-19T20:26:32Z,OWNER,I'm inclined to redefine `ds.path` to `ds.file_path` to fix this. Or `ds.filepath`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073076110,https://api.github.com/repos/simonw/datasette/issues/1668,1073076110,IC_kwDOBm6k_c4_9deO,9599,simonw,2022-03-19T20:27:22Z,2022-03-19T20:27:22Z,OWNER,"The docs do currently describe `path` as the filesystem path here: https://docs.datasette.io/en/stable/internals.html#database-class <img width=""720"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/159137373-5bbc7fea-7544-42b3-8423-be4e11eb4d52.png""> Good thing I'm not at 1.0 yet so I can change that!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073076136,https://api.github.com/repos/simonw/datasette/issues/1668,1073076136,IC_kwDOBm6k_c4_9deo,9599,simonw,2022-03-19T20:27:44Z,2022-03-19T20:27:44Z,OWNER,"Pretty sure changing it will break some existing plugins though, including likely Datasette Desktop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073076187,https://api.github.com/repos/simonw/datasette/issues/1668,1073076187,IC_kwDOBm6k_c4_9dfb,9599,simonw,2022-03-19T20:28:20Z,2022-03-19T20:28:20Z,OWNER,I'm going to keep `path` as the path to the file on disk. I'll pick a new name for what is currently `path` in that undocumented JSON API.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1667#issuecomment-1073076624,https://api.github.com/repos/simonw/datasette/issues/1667,1073076624,IC_kwDOBm6k_c4_9dmQ,9599,simonw,2022-03-19T20:31:44Z,2022-03-19T20:31:44Z,OWNER,I can now read `format` from `request.url_vars` and delete this code entirely: https://github.com/simonw/datasette/blob/b9c2b1cfc8692b9700416db98721fa3ec982f6be/datasette/views/base.py#L375-L381,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174302994,Make route matched pattern groups more consistent, https://github.com/simonw/datasette/issues/1668#issuecomment-1073097394,https://api.github.com/repos/simonw/datasette/issues/1668,1073097394,IC_kwDOBm6k_c4_9iqy,9599,simonw,2022-03-19T20:56:35Z,2022-03-19T20:56:35Z,OWNER,"I'm trying to think if there's any reason not to use `route` for this. Would I possibly want to use that noun for something else in the future? I like it more than `route_path` because it has no underscore. Decision made: I'm going with `route`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073112104,https://api.github.com/repos/simonw/datasette/issues/1668,1073112104,IC_kwDOBm6k_c4_9mQo,9599,simonw,2022-03-19T21:08:21Z,2022-03-19T21:08:21Z,OWNER,"I think I've got this working but I need to write a test for it that covers the rare case when the route is not the same thing as the database name. I'll do that with a new test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073125334,https://api.github.com/repos/simonw/datasette/issues/1668,1073125334,IC_kwDOBm6k_c4_9pfW,9599,simonw,2022-03-19T22:53:55Z,2022-03-19T22:53:55Z,OWNER,"Need to update documentation in a few places - e.g. https://docs.datasette.io/en/stable/internals.html#remove-database-name > This removes a database that has been previously added. `name=` is the unique name of that database, used in its URL path.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073126264,https://api.github.com/repos/simonw/datasette/issues/1668,1073126264,IC_kwDOBm6k_c4_9pt4,9599,simonw,2022-03-19T22:59:30Z,2022-03-19T22:59:30Z,OWNER,"Also need to update the `datasette.urls` methods that construct the URL to a database/table/row - they take the database name but they need to know to look for the route. Need to add tests that check the links in the HTML and can confirm this is working correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073135433,https://api.github.com/repos/simonw/datasette/issues/1668,1073135433,IC_kwDOBm6k_c4_9r9J,9599,simonw,2022-03-20T00:20:36Z,2022-03-20T00:20:36Z,OWNER,"Building this plugin instantly revealed that all of the links - on the homepage and the database page and so on - are incorrect: ```python from datasette import hookimpl @hookimpl def startup(datasette): db = datasette.get_database(""fixtures2"") db.route = ""alternative-route"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073134206,https://api.github.com/repos/simonw/datasette/issues/1668,1073134206,IC_kwDOBm6k_c4_9rp-,9599,simonw,2022-03-20T00:12:03Z,2022-03-20T00:12:03Z,OWNER,I'd like to have a live demo of this up on `latest.datasette.io` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073134816,https://api.github.com/repos/simonw/datasette/issues/1668,1073134816,IC_kwDOBm6k_c4_9rzg,9599,simonw,2022-03-20T00:16:22Z,2022-03-20T00:16:22Z,OWNER,I'm going to add a `fixtures2.db` database which has that as the name but `alternative-route` as the route. I'll set that up using a custom plugin in the `plugins/` folder that gets deployed by https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073136686,https://api.github.com/repos/simonw/datasette/issues/1668,1073136686,IC_kwDOBm6k_c4_9sQu,9599,simonw,2022-03-20T00:31:13Z,2022-03-20T00:31:13Z,OWNER,"That demo is now live: - https://latest.datasette.io/alternative-route - https://latest.datasette.io/alternative-route/attraction_characteristic","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1668#issuecomment-1073136896,https://api.github.com/repos/simonw/datasette/issues/1668,1073136896,IC_kwDOBm6k_c4_9sUA,9599,simonw,2022-03-20T00:33:23Z,2022-03-20T00:33:23Z,OWNER,I'm going to release this as a 0.61 alpha so I can more easily depend on it from `datasette-hashed-urls`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174306154,"Introduce concept of a database `route`, separate from its name", https://github.com/simonw/datasette/issues/1669#issuecomment-1073137170,https://api.github.com/repos/simonw/datasette/issues/1669,1073137170,IC_kwDOBm6k_c4_9sYS,9599,simonw,2022-03-20T00:35:52Z,2022-03-20T00:35:52Z,OWNER,https://github.com/simonw/datasette/compare/0.60.2...main,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174404647,Release 0.61 alpha, https://github.com/simonw/datasette/issues/1669#issuecomment-1073143413,https://api.github.com/repos/simonw/datasette/issues/1669,1073143413,IC_kwDOBm6k_c4_9t51,9599,simonw,2022-03-20T01:24:36Z,2022-03-20T01:24:36Z,OWNER,https://github.com/simonw/datasette/releases/tag/0.61a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174404647,Release 0.61 alpha, https://github.com/simonw/datasette/issues/1439#issuecomment-1059802318,https://api.github.com/repos/simonw/datasette/issues/1439,1059802318,IC_kwDOBm6k_c4_K0zO,9599,simonw,2022-03-05T17:34:33Z,2022-03-05T17:34:33Z,OWNER,"Wrote documentation: <img width=""741"" alt=""Dash encoding. Datasette uses a custom encoding scheme in some places, called dash encoding. This is primarily used for table names and row primary keys, to avoid any confusion between / characters in those values and the Datasette URL that references them. Dash encoding applies the following rules, in order: 1. All single - characters are replaced by -- 2. . characters are replaced by -. 3. / characters are replaced by ./ These rules are applied in reverse order to decode a dash encoded string."" src=""https://user-images.githubusercontent.com/9599/156893903-5723f60e-e054-4365-84bc-f3084d11183d.png""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1647#issuecomment-1059804577,https://api.github.com/repos/simonw/datasette/issues/1647,1059804577,IC_kwDOBm6k_c4_K1Wh,9599,simonw,2022-03-05T17:49:46Z,2022-03-05T17:49:46Z,OWNER,My best guess is that this is an undocumented change in SQLite 3.38 - I get that test failure with that SQLite version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1647#issuecomment-1059807598,https://api.github.com/repos/simonw/datasette/issues/1647,1059807598,IC_kwDOBm6k_c4_K2Fu,9599,simonw,2022-03-05T18:06:56Z,2022-03-05T18:08:00Z,OWNER,"Had a look through the commits in https://github.com/sqlite/sqlite/compare/version-3.37.2...version-3.38.0 but couldn't see anything obvious that might have caused this. Really wish I had a good mechanism for running the test suite against different SQLite versions! May have to revisit this old trick: https://til.simonwillison.net/sqlite/ld-preload","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1647#issuecomment-1059823119,https://api.github.com/repos/simonw/datasette/issues/1647,1059823119,IC_kwDOBm6k_c4_K54P,9599,simonw,2022-03-05T19:56:27Z,2022-03-05T19:56:27Z,OWNER,Updated this TIL with extra patterns I figured out: https://til.simonwillison.net/sqlite/ld-preload,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/pull/1648#issuecomment-1059823151,https://api.github.com/repos/simonw/datasette/issues/1648,1059823151,IC_kwDOBm6k_c4_K54v,22429695,codecov[bot],2022-03-05T19:56:41Z,2022-03-07T15:38:08Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1648](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (32548b8) into [main](https://codecov.io/gh/simonw/datasette/commit/7d24fd405f3c60e4c852c5d746c91aa2ba23cf5b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7d24fd4) will **increase** coverage by `0.02%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1648/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1648 +/- ## ========================================== + Coverage 92.03% 92.05% +0.02% ========================================== Files 34 34 Lines 4557 4570 +13 ========================================== + Hits 4194 4207 +13 Misses 363 363 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/url\_builder.py](https://codecov.io/gh/simonw/datasette/pull/1648/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3VybF9idWlsZGVyLnB5) | `100.00% <100.00%> (ø)` | | | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1648/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.97% <100.00%> (+0.10%)` | :arrow_up: | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1648/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.49% <100.00%> (ø)` | | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1648/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.19% <100.00%> (ø)` | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [7d24fd4...32548b8](https://codecov.io/gh/simonw/datasette/pull/1648?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/issues/1647#issuecomment-1059819628,https://api.github.com/repos/simonw/datasette/issues/1647,1059819628,IC_kwDOBm6k_c4_K5Bs,9599,simonw,2022-03-05T19:28:54Z,2022-03-05T19:28:54Z,OWNER,"OK, using that trick worked for testing this: docker run -it -p 8001:8001 ubuntu Then inside that container: apt-get install -y python3 build-essential tcl wget python3-pip git python3.8-venv For each version of SQLite I wanted to test I needed to figure out the tarball URL - for example, for `3.38.0` I navigated to https://www.sqlite.org/src/timeline?t=version-3.38.0 and clicked the ""checkin"" link and copied the tarball link: https://www.sqlite.org/src/tarball/40fa792d/SQLite-40fa792d.tar.gz Then to build it (the `CPPFLAGS` took some trial and error): ``` cd /tmp wget https://www.sqlite.org/src/tarball/40fa792d/SQLite-40fa792d.tar.gz tar -xzvf SQLite-40fa792d.tar.gz cd SQLite-40fa792d CPPFLAGS=""-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1"" ./configure make ``` Then to test with Datasette: ``` cd /tmp git clone https://github.com/simonw/datasette cd datasette python3 -m venv venv source venv/bin/activate pip install wheel # So bdist_wheel works in next step pip install -e '.[test]' LD_PRELOAD=/tmp/SQLite-40fa792d/.libs/libsqlite3.so pytest ``` After some trial and error I proved that those tests passed with 3.36.0: ``` cd /tmp wget https://www.sqlite.org/src/tarball/5c9a6c06/SQLite-5c9a6c06.tar.gz tar -xzvf SQLite-5c9a6c06.tar.gz cd SQLite-5c9a6c06 CPPFLAGS=""-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1"" ./configure make cd /tmp/datasette LD_PRELOAD=/tmp/SQLite-5c9a6c06/.libs/libsqlite3.so pytest tests/test_internals_database.py ``` BUT failed with 3.37.0: ``` # 3.37.0 cd /tmp wget https://www.sqlite.org/src/tarball/bd41822c/SQLite-bd41822c.tar.gz tar -xzvf SQLite-bd41822c.tar.gz cd SQLite-bd41822c CPPFLAGS=""-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1"" ./configure make cd /tmp/datasette LD_PRELOAD=/tmp/SQLite-bd41822c/.libs/libsqlite3.so pytest tests/test_internals_database.py ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1647#issuecomment-1059821674,https://api.github.com/repos/simonw/datasette/issues/1647,1059821674,IC_kwDOBm6k_c4_K5hq,9599,simonw,2022-03-05T19:44:32Z,2022-03-05T19:44:32Z,OWNER,"I thought I'd need to introduce https://dirty-equals.helpmanual.io/types/string/ to help write tests for this, but I think I've found a good alternative that doesn't need a new dependency.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160407071,Test failures with SQLite 3.37.0+ due to column affinity case, https://github.com/simonw/datasette/issues/1439#issuecomment-1059822151,https://api.github.com/repos/simonw/datasette/issues/1439,1059822151,IC_kwDOBm6k_c4_K5pH,9599,simonw,2022-03-05T19:48:35Z,2022-03-05T19:48:35Z,OWNER,Those new docs: https://github.com/simonw/datasette/blob/d1cb73180b4b5a07538380db76298618a5fc46b6/docs/internals.rst#dash-encoding,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059822391,https://api.github.com/repos/simonw/datasette/issues/1439,1059822391,IC_kwDOBm6k_c4_K5s3,9599,simonw,2022-03-05T19:50:12Z,2022-03-05T19:50:12Z,OWNER,I'm going to move this work to a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059836599,https://api.github.com/repos/simonw/datasette/issues/1439,1059836599,IC_kwDOBm6k_c4_K9K3,9599,simonw,2022-03-05T21:52:10Z,2022-03-05T21:52:10Z,OWNER,Blogged about this here: https://simonwillison.net/2022/Mar/5/dash-encoding/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1645#issuecomment-1059634688,https://api.github.com/repos/simonw/datasette/issues/1645,1059634688,IC_kwDOBm6k_c4_KL4A,9599,simonw,2022-03-05T01:06:08Z,2022-03-05T01:06:08Z,OWNER,"It sounds like you can workaround this with Varnish configuration for the moment, but I'm going to bump this up the list of things to fix - it's particularly relevant now as I'd like to get a solution in place before Datasette 1.0, since it's likely to be beneficial to plugins and hence should be part of the stable, documented plugin interface.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1154399841,"Sensible `cache-control` headers for static assets, including those served by plugins", https://github.com/simonw/datasette/issues/1645#issuecomment-1059633902,https://api.github.com/repos/simonw/datasette/issues/1645,1059633902,IC_kwDOBm6k_c4_KLru,9599,simonw,2022-03-05T01:03:06Z,2022-03-05T01:03:06Z,OWNER,"I agree: this is bad. Ideally, content served from `/static/` would apply best practices for static content serving - which to my mind means the following: - Where possible, serve with a far-future cache expiry header and use an asset URL that changes when the file itself changes - For assets without that, support conditional GET to avoid transferring the whole asset if it hasn't changed - Some kind of sensible mechanism for setting cache TTLs on assets that don't have a unique-file-per-version - in particular assets that might be served from plugins. Datasette half-implemented the first of these: if you view source on https://latest.datasette.io/ you'll see it links to `/-/static/app.css?cead5a` - which in the template looks like this: https://github.com/simonw/datasette/blob/dd94157f8958bdfe9f45575add934ccf1aba6d63/datasette/templates/base.html#L5 I had forgotten I had implemented this! Here is how it is calculated: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/app.py#L510-L516 So `app.css` right now could be safely served with a far-future cache header... only it isn't: ``` ~ % curl -i 'https://latest.datasette.io/-/static/app.css?cead5a' HTTP/2 200 content-type: text/css x-databases: _memory, _internal, fixtures, extra_database x-cloud-trace-context: 9ddc825620eb53d30fc127d1c750f342 date: Sat, 05 Mar 2022 01:01:53 GMT server: Google Frontend content-length: 16178 ``` The larger question though is what to do about other assets. I'm particularly interested in plugin assets, since visualization plugins like `datasette-vega` and `datasette-cluster-map` ship with large amounts of JavaScript and I'd really like that to be sensibly cached by default.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1154399841,"Sensible `cache-control` headers for static assets, including those served by plugins", https://github.com/simonw/datasette/issues/1645#issuecomment-1059634412,https://api.github.com/repos/simonw/datasette/issues/1645,1059634412,IC_kwDOBm6k_c4_KLzs,9599,simonw,2022-03-05T01:04:53Z,2022-03-05T01:04:53Z,OWNER,"The existing `app_css_hash` already isn't good enough, because I built that before `table.js` existed, and that file should obviously be smartly cached too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1154399841,"Sensible `cache-control` headers for static assets, including those served by plugins", https://github.com/simonw/datasette/issues/1640#issuecomment-1059638778,https://api.github.com/repos/simonw/datasette/issues/1640,1059638778,IC_kwDOBm6k_c4_KM36,9599,simonw,2022-03-05T01:19:00Z,2022-03-05T01:19:00Z,OWNER,"The reason I implemented it like this was to support things like the `curl` progress bar if users decide to serve up large files using the `--static` mechanism. Here's the code that hooks it up to the URL resolver: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/app.py#L1001-L1005 Which uses this function: https://github.com/simonw/datasette/blob/a6ff123de5464806441f6a6f95145c9a83b7f20b/datasette/utils/asgi.py#L285-L310 One option here would be to support a workaround that looks something like this: http://localhost:8001/my-static/log.txt?_unknown_size=1` The URL routing code could then look out for that `?_unknown_size=1` option and, if it's present, omit the `content-length` header entirely. It's a bit of a cludge, but it would be pretty straight-forward to implement. Would that work for you @broccolihighkicks?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1148725876,"Support static assets where file length may change, e.g. logs", https://github.com/simonw/datasette/issues/1642#issuecomment-1059635969,https://api.github.com/repos/simonw/datasette/issues/1642,1059635969,IC_kwDOBm6k_c4_KMMB,9599,simonw,2022-03-05T01:11:17Z,2022-03-05T01:11:17Z,OWNER,"`pip install datasette` in a fresh virtual environment doesn't show any warnings. Neither does `pip install -e '.'` in a fresh checkout. Or `pip install -e '.[test]'`. Closing this as can't reproduce.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1152072027,Dependency issue with asgiref and uvicorn, https://github.com/simonw/datasette/issues/1640#issuecomment-1059636420,https://api.github.com/repos/simonw/datasette/issues/1640,1059636420,IC_kwDOBm6k_c4_KMTE,9599,simonw,2022-03-05T01:13:26Z,2022-03-05T01:13:26Z,OWNER,"Hah, this is certainly unexpected. It looks like this is the code in question: https://github.com/simonw/datasette/blob/a6ff123de5464806441f6a6f95145c9a83b7f20b/datasette/utils/asgi.py#L259-L266 You're right: it assumes that the file it is serving won't change length while it is serving it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1148725876,"Support static assets where file length may change, e.g. logs", https://github.com/simonw/datasette/pull/1648#issuecomment-1060065736,https://api.github.com/repos/simonw/datasette/issues/1648,1060065736,IC_kwDOBm6k_c4_L1HI,9599,simonw,2022-03-06T23:43:00Z,2022-03-06T23:43:11Z,OWNER,"> * Maybe use dash encoding for database name too? Yes, I'm going to do this. At the moment if a DB file is called `fixx%tures.db` when you run it in Datasette the path is `/fix%2525tures` - which is liable to break.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/pull/1648#issuecomment-1060067031,https://api.github.com/repos/simonw/datasette/issues/1648,1060067031,IC_kwDOBm6k_c4_L1bX,9599,simonw,2022-03-06T23:50:40Z,2022-03-06T23:58:31Z,OWNER,"I may have to do extra work here ```python def database(self, database, format=None): db = self.ds.databases[database] if self.ds.setting(""hash_urls"") and db.hash: path = self.path( f""{dash_encode(database)}-{db.hash[:HASH_LENGTH]}"", format=format ) else: path = self.path(dash_encode(database), format=format) return path ``` The URLs that incorporate a hash have a `dbname-hash` format - will that `-` in the middle there mess up the dash decoding mechanism? I think it will. Might be able to solve that like so: ```python async def resolve_db_name(self, request, db_name, **kwargs): hash = None name = None decoded_name = dash_decode(db_name) if decoded_name not in self.ds.databases and ""-"" in db_name: # No matching DB found, maybe it's a name-hash? name_bit, hash_bit = db_name.rsplit(""-"", 1) if dash_decode(name_bit) not in self.ds.databases: raise NotFound(f""Database not found: {name}"") else: name = dash_decode(name_bit) hash = hash_bit else: name = decoded_name ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/issues/1439#issuecomment-1059850369,https://api.github.com/repos/simonw/datasette/issues/1439,1059850369,IC_kwDOBm6k_c4_LAiB,9599,simonw,2022-03-05T23:28:56Z,2022-03-05T23:28:56Z,OWNER,"Lots of great conversations about the dash encoding implementation on Twitter: https://twitter.com/simonw/status/1500228316309061633 @dracos helped me figure out a simpler regex: https://twitter.com/dracos/status/1500236433809973248 `^/(?P<database>[^/]+)/(?P<table>[^\/\-\.]*|\-/|\-\.|\-\-)*(?P<format>\.\w+)?$` ![image](https://user-images.githubusercontent.com/9599/156903088-c01933ae-4713-4e91-8d71-affebf70b945.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059851259,https://api.github.com/repos/simonw/datasette/issues/1439,1059851259,IC_kwDOBm6k_c4_LAv7,9599,simonw,2022-03-05T23:35:47Z,2022-03-05T23:35:59Z,OWNER,"This [comment from glyph](https://twitter.com/glyph/status/1500244937312329730) got me thinking: > Have you considered replacing % with some other character and then using percent-encoding? What happens if a table name includes a `%` character and that ends up getting mangled by a misbehaving proxy? I should consider `%` in the escaping system too. And maybe go with that suggestion of using percent-encoding directly but with a different character.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059853526,https://api.github.com/repos/simonw/datasette/issues/1439,1059853526,IC_kwDOBm6k_c4_LBTW,9599,simonw,2022-03-05T23:49:59Z,2022-03-05T23:49:59Z,OWNER,"I want to try regular percentage encoding, except that it also encodes both the `-` and the `.` characters, AND it uses `-` instead of `%` as the encoding character. Should check what it does with emoji too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059854864,https://api.github.com/repos/simonw/datasette/issues/1439,1059854864,IC_kwDOBm6k_c4_LBoQ,9599,simonw,2022-03-05T23:59:05Z,2022-03-05T23:59:05Z,OWNER,"OK, for that percentage thing: the Python core implementation of URL percentage escaping deliberately ignores two of the characters we want to escape: `.` and `-`: https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Lib/urllib/parse.py#L780-L783 ```python _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-~') ``` It also defaults to skipping `/` (passed as a `safe=` parameter to various things). I'm going to try borrowing and modifying the core of the Python implementation: https://github.com/python/cpython/blob/6927632492cbad86a250aa006c1847e03b03e70b/Lib/urllib/parse.py#L795-L814 ```python class _Quoter(dict): """"""A mapping from bytes numbers (in range(0,256)) to strings. String values are percent-encoded byte values, unless the key < 128, and in either of the specified safe set, or the always safe set. """""" # Keeps a cache internally, via __missing__, for efficiency (lookups # of cached keys don't call Python code at all). def __init__(self, safe): """"""safe: bytes object."""""" self.safe = _ALWAYS_SAFE.union(safe) def __repr__(self): return f""<Quoter {dict(self)!r}>"" def __missing__(self, b): # Handle a cache miss. Store quoted string in cache and return. res = chr(b) if b in self.safe else '%{:02X}'.format(b) self[b] = res return res ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059855418,https://api.github.com/repos/simonw/datasette/issues/1439,1059855418,IC_kwDOBm6k_c4_LBw6,9599,simonw,2022-03-06T00:00:53Z,2022-03-06T00:04:18Z,OWNER,"```python _ESCAPE_SAFE = frozenset( b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789_' ) # I removed b'.-~') class Quoter(dict): # Keeps a cache internally, via __missing__ def __missing__(self, b): # Handle a cache miss. Store quoted string in cache and return. res = chr(b) if b in _ESCAPE_SAFE else '-{:02X}'.format(b) self[b] = res return res quoter = Quoter().__getitem__ ''.join([quoter(char) for char in b'foo/bar.csv']) # 'foo-2Fbar-2Ecsv' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059863997,https://api.github.com/repos/simonw/datasette/issues/1439,1059863997,IC_kwDOBm6k_c4_LD29,505230,karlcow,2022-03-06T00:57:57Z,2022-03-06T00:57:57Z,NONE,"Probably too late… but I have just seen this because http://simonwillison.net/2022/Mar/5/dash-encoding/#atom-everything And it reminded me of comma tools at W3C. http://www.w3.org/,tools Example, the text version of W3C homepage https://www.w3.org/,text > The challenge comes down to telling the difference between the following: > > * `/db/table` - an HTML table page `/db/table` > * `/db/table.csv` - the CSV version of `/db/table` `/db/table,csv` > * `/db/table.csv` - no this one is actually a database table called `table.csv` `/db/table.csv` > * `/db/table.csv.csv` - the CSV version of `/db/table.csv` `/db/table.csv,csv` > * `/db/table.csv.csv.csv` and so on... `/db/table.csv.csv,csv` I haven't checked all the cases in the thread.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1059864154,https://api.github.com/repos/simonw/datasette/issues/1439,1059864154,IC_kwDOBm6k_c4_LD5a,9599,simonw,2022-03-06T00:59:04Z,2022-03-06T00:59:04Z,OWNER,"Needs more testing, but this seems to work for decoding the percent-escaped-with-dashes format: `urllib.parse.unquote(s.replace('-', '%'))`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/pull/1589#issuecomment-1059875687,https://api.github.com/repos/simonw/datasette/issues/1589,1059875687,IC_kwDOBm6k_c4_LGtn,9599,simonw,2022-03-06T01:58:25Z,2022-03-06T01:58:25Z,OWNER,Thanks for catching this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1098275181,Typo in docs about default redirect status code, https://github.com/simonw/datasette/issues/1439#issuecomment-1059903309,https://api.github.com/repos/simonw/datasette/issues/1439,1059903309,IC_kwDOBm6k_c4_LNdN,9599,simonw,2022-03-06T06:17:51Z,2022-03-06T06:17:51Z,OWNER,"Suggestion from a conversation with Seth Michael Larson: it would be neat if plugins could easily integrate with whatever scheme this ends up using, maybe with the `/db/table/-/plugin-name` standardized pattern or similar. Making it easy for plugins to do the right, consistent thing is a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/pull/1648#issuecomment-1060016221,https://api.github.com/repos/simonw/datasette/issues/1648,1060016221,IC_kwDOBm6k_c4_LpBd,9599,simonw,2022-03-06T18:37:59Z,2022-03-06T18:37:59Z,OWNER,"Change of plan: based on extensive conversations on Twitter - see https://github.com/simonw/datasette/issues/1439#issuecomment-1059851259 - I'm going to try a variant of this which is basically percent-encoding but with a hyphen instead of a percent symbol. Reason being that the current scheme doesn't handle the case of `%` being part of the table name, which could cause weird breakage due to some proxies decoding percent encoding before it gets to Datasette.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/pull/1649#issuecomment-1060021753,https://api.github.com/repos/simonw/datasette/issues/1649,1060021753,IC_kwDOBm6k_c4_LqX5,22429695,codecov[bot],2022-03-06T19:13:09Z,2022-03-06T19:13:09Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1649](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (59b2c16) into [main](https://codecov.io/gh/simonw/datasette/commit/0499f174c063283aa9b589d475a32077aaf7adc5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0499f17) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1649/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1649 +/- ## ======================================= Coverage 92.03% 92.03% ======================================= Files 34 34 Lines 4557 4557 ======================================= Hits 4194 4194 Misses 363 363 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1649/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.86% <ø> (ø)` | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0499f17...59b2c16](https://codecov.io/gh/simonw/datasette/pull/1649?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160677684,Add /opt/homebrew to where spatialite extension can be found, https://github.com/simonw/datasette/pull/1648#issuecomment-1060034562,https://api.github.com/repos/simonw/datasette/issues/1648,1060034562,IC_kwDOBm6k_c4_LtgC,9599,simonw,2022-03-06T20:36:12Z,2022-03-06T20:36:12Z,OWNER,"Updated documentation: ![image](https://user-images.githubusercontent.com/9599/156941171-89778c12-41bc-4951-97f2-ecc805025a53.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/pull/1648#issuecomment-1060044592,https://api.github.com/repos/simonw/datasette/issues/1648,1060044592,IC_kwDOBm6k_c4_Lv8w,9599,simonw,2022-03-06T21:42:35Z,2022-03-06T21:42:35Z,OWNER,"For consistency, I'm going to change how `?_next=` tokens work too. Right now they work like this: https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/views/table.py#L501-L507 https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/utils/__init__.py#L114-L116 I'm going to change those to use dash-encoding instead. I considered looking for `%` in those values and replacing that as `-` too, but since Datasette isn't 1.0 yet I'm going to risk breaking any pagination tokens that people might have saved away somewhere!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/issues/1439#issuecomment-1060044007,https://api.github.com/repos/simonw/datasette/issues/1439,1060044007,IC_kwDOBm6k_c4_Lvzn,9599,simonw,2022-03-06T21:38:15Z,2022-03-06T21:38:15Z,OWNER,"Test: https://github.com/simonw/datasette/blob/d2e3fe3facf0ed0abf8b00cd54463af90dd6904d/tests/test_utils.py#L651-L666 One big advantage to this scheme is that redirecting old links to `%2F` pages (e.g. https://fivethirtyeight.datasettes.com/fivethirtyeight/twitter-ratio%2Fsenators) is easy - if you see a `%` in the `raw_path`, redirect to that page with the `%` replaced by `-`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/pull/1648#issuecomment-1060056510,https://api.github.com/repos/simonw/datasette/issues/1648,1060056510,IC_kwDOBm6k_c4_Ly2-,9599,simonw,2022-03-06T23:02:05Z,2022-03-06T23:04:24Z,OWNER,"Just spotted this: https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/views/base.py#L203-L216 Maybe the db name should use dash encoding too? If so, relevant code includes this bit: https://github.com/simonw/datasette/blob/de810f49cc57a4f88e4a1553d26c579253ce4531/datasette/url_builder.py#L30-L38","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160432941,Use dash encoding for table names and row primary keys in URLs, https://github.com/simonw/datasette/issues/1651#issuecomment-1060853226,https://api.github.com/repos/simonw/datasette/issues/1651,1060853226,IC_kwDOBm6k_c4_O1Xq,9599,simonw,2022-03-07T16:04:26Z,2022-03-07T16:04:26Z,OWNER,"Here's the relevant code: https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/utils/__init__.py#L753-L772 https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L451-L479","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1650#issuecomment-1060863311,https://api.github.com/repos/simonw/datasette/issues/1650,1060863311,IC_kwDOBm6k_c4_O31P,9599,simonw,2022-03-07T16:13:17Z,2022-03-07T16:13:17Z,OWNER,"This doesn't seem to work. https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv should be redirecting now that this is deployed - which it is, because https://latest.datasette.io/-/versions shows 644d25d1de78a36b105cca479e7b3e4375a6eadc - but I'm not getting that redirect.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1650#issuecomment-1060864823,https://api.github.com/repos/simonw/datasette/issues/1650,1060864823,IC_kwDOBm6k_c4_O4M3,9599,simonw,2022-03-07T16:14:33Z,2022-03-07T16:14:33Z,OWNER,Same problem here: https://fivethirtyeight.datasettes.com/fivethirtyeight/ahca-2Dpolls%2Fahca_polls should redirect but doesn't.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1439#issuecomment-1060870237,https://api.github.com/repos/simonw/datasette/issues/1439,1060870237,IC_kwDOBm6k_c4_O5hd,9599,simonw,2022-03-07T16:19:22Z,2022-03-07T16:19:22Z,OWNER,"I didn't need to do any of the fancy regular expression routing stuff after all, since the new dash encoding format avoids using `/` so a simple `[^/]+` can capture the correct segments from the URL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1650#issuecomment-1060836262,https://api.github.com/repos/simonw/datasette/issues/1650,1060836262,IC_kwDOBm6k_c4_OxOm,9599,simonw,2022-03-07T15:52:09Z,2022-03-07T15:52:09Z,OWNER,"This is a bit tricky. I tried this, sending a redirect only if a 404 happens: ```diff diff --git a/datasette/app.py b/datasette/app.py index 8c5480c..420664c 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1211,6 +1211,10 @@ class DatasetteRouter: return await self.handle_404(request, send) async def handle_404(self, request, send, exception=None): + # If path contains % encoding, redirect to dash encoding + if '%' in request.scope[""path""]: + await asgi_send_redirect(send, request.scope[""path""].replace(""%"", ""-"")) + return # If URL has a trailing slash, redirect to URL without it path = request.scope.get( ""raw_path"", request.scope[""path""].encode(""utf8"") ``` But this URL didn't work: - http://127.0.0.1:8001/fivethirtyeight/twitter-ratio%2Fsenators I was expecting that to redirect to this page: - http://127.0.0.1:8001/fivethirtyeight/twitter-2Dratio-2Fsenators But instead it took me to another 404: - http://127.0.0.1:8001/fivethirtyeight/twitter-ratio%2Fsenators This is because that URL contains both a %-escaped `/` AND a plain `-` - which was not escaped in the old system but is escaped in the new system.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1653#issuecomment-1061150672,https://api.github.com/repos/simonw/datasette/issues/1653,1061150672,IC_kwDOBm6k_c4_P9_Q,9599,simonw,2022-03-07T21:23:39Z,2022-03-07T21:23:39Z,OWNER,"There may be a short-term fix for this: table view could start accepting a `?_sort_sql=SQLfragment` parameter, similar to the `?_where=` parameter described here: https://docs.datasette.io/en/stable/json_api.html#special-table-arguments That fragment could then be pre-populated in `metadata`. Makes me think maybe that `?_where=` should be optionally settable in metadata too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161937073,Mechanism to default a table to sorting by multiple columns, https://github.com/simonw/datasette/issues/1653#issuecomment-1061148807,https://api.github.com/repos/simonw/datasette/issues/1653,1061148807,IC_kwDOBm6k_c4_P9iH,9599,simonw,2022-03-07T21:21:23Z,2022-03-07T21:21:23Z,OWNER,"This is currently blocked on the fact that Datasette doesn't have a mechanism for sorting by more than one column: - #197","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161937073,Mechanism to default a table to sorting by multiple columns, https://github.com/simonw/datasette/issues/1650#issuecomment-1061038414,https://api.github.com/repos/simonw/datasette/issues/1650,1061038414,IC_kwDOBm6k_c4_PilO,9599,simonw,2022-03-07T19:14:04Z,2022-03-07T19:14:04Z,OWNER,"The problem seems to be that `http://127.0.0.1:8002/fixtures/table%2Fwith%2Fslashes.csv` doesn't result in a 404 at all. If it did, it would be redirected.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1650#issuecomment-1061041034,https://api.github.com/repos/simonw/datasette/issues/1650,1061041034,IC_kwDOBm6k_c4_PjOK,9599,simonw,2022-03-07T19:16:51Z,2022-03-07T19:16:51Z,OWNER,"Here's the problem: https://github.com/simonw/datasette/blob/020effe47bf89f35182960a9645f2383a42ebd54/datasette/utils/__init__.py#L1173-L1175 Which is called here: https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L469-L473 So `table%2Fwith%2Fslashes` ends up decoded as if it was using dash encoding.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160750713,Implement redirects from old % encoding to new dash encoding, https://github.com/simonw/datasette/issues/1651#issuecomment-1061053094,https://api.github.com/repos/simonw/datasette/issues/1651,1061053094,IC_kwDOBm6k_c4_PmKm,9599,simonw,2022-03-07T19:29:01Z,2022-03-07T19:29:01Z,OWNER,"I found an obscure bug in #1650 which I can fix with this too. The following test should pass: ```python @pytest.mark.parametrize( ""path,expected"", ( ( ""/fivethirtyeight/twitter-ratio%2Fsenators"", ""/fivethirtyeight/twitter-2Dratio-2Fsenators"", ), ( ""/fixtures/table%2Fwith%2Fslashes.csv"", ""/fixtures/table-2Fwith-2Fslashes-2Ecsv"", ), # query string should be preserved (""/foo/bar%2Fbaz?id=5"", ""/foo/bar-2Fbaz?id=5""), ), ) def test_redirect_percent_encoding_to_dash_encoding(app_client, path, expected): response = app_client.get(path) assert response.status == 302 assert response.headers[""location""] == expected ``` It currently fails like this: ``` > assert response.headers[""location""] == expected E AssertionError: assert '/fixtures/table-2Fwith-2Fslashes.csv?_nofacet=1&_nocount=1' == '/fixtures/table-2Fwith-2Fslashes-2Ecsv' E - /fixtures/table-2Fwith-2Fslashes-2Ecsv E + /fixtures/table-2Fwith-2Fslashes.csv?_nofacet=1&_nocount=1 ``` Because the logic in that `get_format()` function notices that the table exists, and then weird things happen here: https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L288-L303 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1651#issuecomment-1061170897,https://api.github.com/repos/simonw/datasette/issues/1651,1061170897,IC_kwDOBm6k_c4_QC7R,9599,simonw,2022-03-07T21:48:35Z,2022-03-07T21:48:35Z,OWNER,"My attempts to simplify `get_format()` keep resulting in errors like this one: ``` File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 474, in view_get response_or_template_contexts = await self.data( TypeError: TableView.data() missing 1 required positional argument: 'table' ``` I really need to clean this up.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1651#issuecomment-1061169528,https://api.github.com/repos/simonw/datasette/issues/1651,1061169528,IC_kwDOBm6k_c4_QCl4,9599,simonw,2022-03-07T21:47:01Z,2022-03-07T21:47:01Z,OWNER,"Wow, this code is difficult to follow! Look at this bit inside the `get_format()` method: https://github.com/simonw/datasette/blob/bb499942c15c4e2cfa4b6afab8f8debe5948c009/datasette/views/base.py#L469-L478 That's modifying the arguments that were extracted from the path by the routing regular expressions to have `table` as ` dash-decoded value! So calling `.get_format()` has the side effect of decoding the table names for you. Nasty.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1654#issuecomment-1061181089,https://api.github.com/repos/simonw/datasette/issues/1654,1061181089,IC_kwDOBm6k_c4_QFah,9599,simonw,2022-03-07T22:01:38Z,2022-03-07T22:01:38Z,OWNER,"I'm going to use the [widely adopted](https://www.contributor-covenant.org/adopters/) Contributor Covenant: https://www.contributor-covenant.org/version/1/4/code-of-conduct/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1654#issuecomment-1061181530,https://api.github.com/repos/simonw/datasette/issues/1654,1061181530,IC_kwDOBm6k_c4_QFha,9599,simonw,2022-03-07T22:02:06Z,2022-03-07T22:02:06Z,OWNER,https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/adding-a-code-of-conduct-to-your-project says this should be called `CODE_OF_CONDUCT.md` in order for GitHub to pick it up.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1654#issuecomment-1061182132,https://api.github.com/repos/simonw/datasette/issues/1654,1061182132,IC_kwDOBm6k_c4_QFq0,9599,simonw,2022-03-07T22:02:43Z,2022-03-07T22:02:43Z,OWNER,"Neat, GitHub have a template for this https://github.com/simonw/datasette/community/code-of-conduct/new?template=contributor-covenant","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1654#issuecomment-1061184206,https://api.github.com/repos/simonw/datasette/issues/1654,1061184206,IC_kwDOBm6k_c4_QGLO,9599,simonw,2022-03-07T22:04:51Z,2022-03-07T22:04:51Z,OWNER,I'm going to add this to the main Datasette repo (done) and the `datasette.io` website too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1654#issuecomment-1061197133,https://api.github.com/repos/simonw/datasette/issues/1654,1061197133,IC_kwDOBm6k_c4_QJVN,9599,simonw,2022-03-07T22:19:35Z,2022-03-07T22:19:35Z,OWNER,"Also now live on https://datasette.io ![CleanShot 2022-03-07 at 14 18 30@2x](https://user-images.githubusercontent.com/9599/157127424-805b3166-f0a8-4fac-be87-c055740af580.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161969891,Adopt a code of conduct, https://github.com/simonw/datasette/issues/1651#issuecomment-1061223822,https://api.github.com/repos/simonw/datasette/issues/1651,1061223822,IC_kwDOBm6k_c4_QP2O,9599,simonw,2022-03-07T22:54:54Z,2022-03-07T22:54:54Z,OWNER,"I'm going to do a review of how URL routing works at the moment for the various views. I edited down [the full list](https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/app.py#L997-L1107) a bit - these are the most relevant: ```python add_route(IndexView.as_view(self), r""/(?P<as_format>(\.jsono?)?$)"") add_route( DatabaseView.as_view(self), r""/(?P<db_name>[^/]+?)(?P<as_format>"" + renderer_regex + r""|.jsono|\.csv)?$"", ) add_route( TableView.as_view(self), r""/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)"", ) add_route( RowView.as_view(self), r""/(?P<db_name>[^/]+)/(?P<table>[^/]+?)/(?P<pk_path>[^/]+?)(?P<as_format>"" + renderer_regex + r"")?$"", ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/647#issuecomment-1061226942,https://api.github.com/repos/simonw/datasette/issues/647,1061226942,IC_kwDOBm6k_c4_QQm-,9599,simonw,2022-03-07T23:00:06Z,2022-03-07T23:00:06Z,OWNER,"This needs to take into account the changes made here: - #1439 In the new encoding scheme, `-` has a special meaning in a table name: https://docs.datasette.io/en/latest/internals.html#dash-encoding I think `~` is the right character to use to separate a database name from its hash. `~` should be a URL safe character according to Python's implementation of percent-encoding, see comment here: https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/utils/__init__.py#L1146-L1152 So the plugin could check for `dbname~hash` and react based on that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061267615,https://api.github.com/repos/simonw/datasette/issues/647,1061267615,IC_kwDOBm6k_c4_Qaif,9599,simonw,2022-03-08T00:05:43Z,2022-03-08T00:05:43Z,OWNER,"Built a prototype of that plugin: ```python from datasette import hookimpl from functools import wraps @hookimpl def asgi_wrapper(datasette): def wrap_with_hashed_urls(app): @wraps(app) async def hashed_urls(scope, receive, send): # Only triggers on pages with a path not starting in /-/ # and where the first page component matches a database name if scope.get(""type"") != ""http"": await app(scope, receive, send) return path = scope[""path""].lstrip(""/"") if not path or path.startswith(""-/""): await app(scope, receive, send) return potential_database = path.split(""/"")[0] # It may or may not be already dbname~hash if ""~"" in potential_database: db_name, hash = potential_database.split(""~"", 1) else: db_name = potential_database hash = """" # Is db_name a database we have a hash for? try: db = datasette.get_database(db_name) except KeyError: await app(scope, receive, send) return if db.hash is not None: # TODO: make sure db.hash is documented if db.hash[:7] != hash: # Send a redirect path_bits = path.split(""/"") new_path = ""/"" + ""/"".join([""{}-{}"".format(db_name, db.hash[:7])] + path_bits[1:]) if scope.get(""query_string""): new_path += ""?"" + scope[""query_string""].decode(""latin-1"") await send({ ""type"": ""http.response.start"", ""status"": 302, ""headers"": [ [b""location"", new_path.encode(""latin1"")] ], }) await send({""type"": ""http.response.body"", ""body"": b""""}) return else: # Add a far-future cache header async def wrapped_send(event): if event[""type""] == ""http.response.start"": original_headers = event.get(""headers"") or [] event = { ""type"": event[""type""], ""status"": event[""status""], ""headers"": original_headers + [ [b""Cache-Control"", b""max-age=31536000""] ], } await send(event) await app(scope, receive, wrapped_send) return await app(scope, receive, send) return hashed_urls return wrap_with_hashed_urls ``` One catch: it doesn't affect the way URLs are generated - so every internal link within Datasette links to the non-hash version and then triggers a 302 redirect to the hashed version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061272544,https://api.github.com/repos/simonw/datasette/issues/647,1061272544,IC_kwDOBm6k_c4_Qbvg,9599,simonw,2022-03-08T00:14:42Z,2022-03-08T00:14:42Z,OWNER,Maybe the plugin should interfere with `datasette.databases` on startup and change the registered name for each one?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061276399,https://api.github.com/repos/simonw/datasette/issues/647,1061276399,IC_kwDOBm6k_c4_Qcrv,9599,simonw,2022-03-08T00:21:47Z,2022-03-08T00:21:47Z,OWNER,"This seems to do the job: ```python @hookimpl def startup(datasette): for name, database in datasette.databases.items(): if database.hash: new_name = ""{}_{}"".format(name, database.hash[:7]) del datasette.databases[name] datasette.databases[new_name] = database ``` Would have to teach the rest of the plugin to split on `_` and to only redirect if the user seems to be hitting the URL for an old hash after which Datasette has been restarted with an updated database.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061276646,https://api.github.com/repos/simonw/datasette/issues/647,1061276646,IC_kwDOBm6k_c4_Qcvm,9599,simonw,2022-03-08T00:22:11Z,2022-03-08T00:22:11Z,OWNER,I'm now convinced this is feasible enough that it's worth doing in time for Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1061282743,https://api.github.com/repos/simonw/datasette/issues/647,1061282743,IC_kwDOBm6k_c4_QeO3,9599,simonw,2022-03-08T00:32:34Z,2022-03-08T00:32:47Z,OWNER,It would be neat if the plugin could spot old-style hyphen hash URLs (maybe on 404) and redirect those too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/1645#issuecomment-1061355871,https://api.github.com/repos/simonw/datasette/issues/1645,1061355871,IC_kwDOBm6k_c4_QwFf,9599,simonw,2022-03-08T02:59:28Z,2022-03-08T02:59:28Z,OWNER,"Hah, found a TODO about this: https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/app.py#L997-L999","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1154399841,"Sensible `cache-control` headers for static assets, including those served by plugins", https://github.com/simonw/datasette/issues/1651#issuecomment-1061359915,https://api.github.com/repos/simonw/datasette/issues/1651,1061359915,IC_kwDOBm6k_c4_QxEr,9599,simonw,2022-03-08T03:08:14Z,2022-03-08T03:09:24Z,OWNER,"A lot of the code complexity here is caused by `DataView` ([here](https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/views/base.py#L182-L669)), which has the logic for CSV streaming and plugin formats such that it can be shared between tables and custom queries. It would be good to get rid of that subclassed shared code, figure out how to do it via a utility function instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/932#issuecomment-1061891851,https://api.github.com/repos/simonw/datasette/issues/932,1061891851,IC_kwDOBm6k_c4_Sy8L,9599,simonw,2022-03-08T15:20:48Z,2022-03-08T15:20:48Z,OWNER,Made a start on this here: https://datasette.io/tutorials ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation, https://github.com/simonw/datasette/issues/1384#issuecomment-1062124485,https://api.github.com/repos/simonw/datasette/issues/1384,1062124485,IC_kwDOBm6k_c4_TrvF,167160,khusmann,2022-03-08T19:26:32Z,2022-03-08T19:26:32Z,NONE,"Looks like I'm late to the party here, but wanted to join the convo if there's still time before this interface is solidified in v1.0. My plugin use case is for education / social science data, which is meta-data heavy in the documentation of measurement scales, instruments, collection procedures, etc. that I want to connect to columns, tables, and dbs (and render in static pages, but looks like I can do that with the jinja plugin hook). I'm still digging in and I think @brandonrobertz 's approach will work for me at least for now, but I want to bump this thread in the meantime -- are there still plans for an async metadata hook at some point in the future? (or are you considering other directions?)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1655#issuecomment-1062445113,https://api.github.com/repos/simonw/datasette/issues/1655,1062445113,IC_kwDOBm6k_c4_U6A5,9599,simonw,2022-03-09T01:01:24Z,2022-03-09T01:01:24Z,OWNER,"https://labordata.bunkum.us/-/settings shows `max_returned_rows` had been increased to 5,000 for that instance - the default of 1,000 would help a bit here. Any thoughts on how Datasette could handle this kind of thing better?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data, https://github.com/simonw/datasette/issues/1655#issuecomment-1062450649,https://api.github.com/repos/simonw/datasette/issues/1655,1062450649,IC_kwDOBm6k_c4_U7XZ,536941,fgregg,2022-03-09T01:10:46Z,2022-03-09T01:10:46Z,CONTRIBUTOR,"i increased the max_returned_row, because I have some scripts that get CSVs from this site, and this makes doing pagination of CSVs less annoying for many cases. i know that's streaming csvs is something you are hoping to address in 1.0. let me know if there's anything i can do to help with that. as for what if anything can be done about the size of the dom, I don't have any ideas right now, but i'll poke around.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data, https://github.com/simonw/datasette/issues/1634#issuecomment-1065334891,https://api.github.com/repos/simonw/datasette/issues/1634,1065334891,IC_kwDOBm6k_c4_f7hr,208018,dholth,2022-03-11T17:38:08Z,2022-03-11T17:38:08Z,NONE,I noticed the image was large when using fly. Is it possible to use a -slim base?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1131295060,Update Dockerfile generated by `datasette publish`, https://github.com/simonw/datasette/issues/1384#issuecomment-1065929510,https://api.github.com/repos/simonw/datasette/issues/1384,1065929510,IC_kwDOBm6k_c4_iMsm,167160,khusmann,2022-03-12T17:49:59Z,2022-03-12T17:49:59Z,NONE,"Ok, I'm taking a slightly different approach, which I think is sort of close to the in-memory _metadata table idea. I'm using a startup hook to load metadata / other info from the database, which I store in the datasette object for later: ``` @hookimpl def startup(datasette): async def inner(): datasette._mypluginmetadata = # await db query return inner ``` Then, I can use this in other plugins: ``` @hookimpl def render_cell(value, column, table, database, datasette): # use datasette._mypluginmetadata ``` For my app I don't need anything to update dynamically so it's fine to pre-populate everything on startup. It's also good to have things precached especially for a hook like render_cell, which would otherwise require a ton of redundant db queries. Makes me wonder if we could take a sort of similar caching approach with the internal _metadata table. Like have a little watchdog that could query all of the attached dbs for their _metadata tables every 5min or so, which then could be merged into the in memory _metadata table which then could be accessed sync by the plugins, or something like that. For most the use cases I can think of, live updates don't need to take into effect immediately; refreshing a cache every 5min or on some other trigger (adjustable w a config setting) would be just fine. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-1065940779,https://api.github.com/repos/simonw/datasette/issues/1384,1065940779,IC_kwDOBm6k_c4_iPcr,2670795,brandonrobertz,2022-03-12T18:49:29Z,2022-03-12T18:50:07Z,CONTRIBUTOR,"Hello! Just wanted to chime in and note that there's a plugin to have Datasette [watch for updates to an external metadata.yaml/json and update the internal settings accordingly](https://datasette.io/plugins/datasette-remote-metadata), so I think the cache/poll use case is already covered. @khusmann If you don't need truly dynamic metadata then what you've come up with or the plugin ought to work fine. Making the get_metadata async won't improve the situation by itself as only some of the code paths accessing metadata use that hook. The other paths use the internal metadata dict. Trying to force all paths through a async hook would have performance ramifications and making everything use the internal meta will cause problems for users that need changes to take effect immediately. This is why I came to the non-async solution as it was the path of least change within Datasette. As always, open to new ideas, etc!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-1065951744,https://api.github.com/repos/simonw/datasette/issues/1384,1065951744,IC_kwDOBm6k_c4_iSIA,167160,khusmann,2022-03-12T19:47:17Z,2022-03-12T19:47:17Z,NONE,"Awesome, thanks @brandonrobertz ! The plugin is close, but looks like it only grabs remote metadata, is that right? Instead what I'm wanting is to grab metadata embedded in the attached databases. Rather than extending that plugin, at this point I've realized I need a lot more flexibility in metadata for my data model (esp around formatting cell values and custom file exports) so rather than extending that I'll continue working on a plugin specific to my app. If I'm understanding your plugin code correctly, you query the db using the sync handle every time `get_metdata` is called, right? Won't this become a pretty big bottleneck if a hook into `render_cell` is trying to read metadata / plugin config? > Making the get_metadata async won't improve the situation by itself as only some of the code paths accessing metadata use that hook. The other paths use the internal metadata dict. I agree -- because things like `render_cell` will potentially want to read metadata/config, `get_metadata` should really remain sync and lightweight, which we can do with something like the remote-metadata plugin that could also poll metadata tables in attached databases. That leaves your app, where it sounds like you want changes made by the user in the browser in to be immediately reflected, rather than have to wait for the next metadata refresh. In this case I wonder if you could have your app make a sync write to the datasette object so the change would have the immediate effect, but then have a separate async polling mechanism to eventually write that change out to the database for long-term persistence. Then you'd have the best of both worlds, I think? But probably not worth the trouble if your use cases are small (and/or you're not reading metadata/config from tight loops like render_cell).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1439#issuecomment-1065987808,https://api.github.com/repos/simonw/datasette/issues/1439,1065987808,IC_kwDOBm6k_c4_ia7g,9599,simonw,2022-03-13T00:02:32Z,2022-03-13T00:02:32Z,OWNER,"OK, this has broken a lot more than I expected it would. Turns out `-` is a very common character in existing Datasette database names! https://datasette.io/-/databases for example has two: ```json [ { ""name"": ""docs-index"", ""path"": ""docs-index.db"", ""size"": 1007616, ""is_mutable"": false, ""is_memory"": false, ""hash"": ""0ac6c3de2762fcd174fd249fed8a8fa6046ea345173d22c2766186bf336462b2"" }, { ""name"": ""dogsheep-index"", ""path"": ""dogsheep-index.db"", ""size"": 5496832, ""is_mutable"": false, ""is_memory"": false, ""hash"": ""d1ea238d204e5b9ae783c86e4af5bcdf21267c1f391de3e468d9665494ee012a"" } ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1439#issuecomment-1065988403,https://api.github.com/repos/simonw/datasette/issues/1439,1065988403,IC_kwDOBm6k_c4_ibEz,9599,simonw,2022-03-13T00:06:38Z,2022-03-13T00:07:19Z,OWNER,"If I want to reserve `-` as a character that CAN be used in URLs, the only remaining character that might make sense for escape sequences is `~` - based on this last line of characters that are escape from percentage encoding: ```python _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-~') ``` So I'd add both `-` and `_` back to the safe list, but use `~` to escape `.` and `/` and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1384#issuecomment-1066006292,https://api.github.com/repos/simonw/datasette/issues/1384,1066006292,IC_kwDOBm6k_c4_ifcU,2670795,brandonrobertz,2022-03-13T02:09:44Z,2022-03-13T02:09:44Z,CONTRIBUTOR,"> If I'm understanding your plugin code correctly, you query the db using the sync handle every time `get_metdata` is called, right? Won't this become a pretty big bottleneck if a hook into `render_cell` is trying to read metadata / plugin config? Reading from sqlite DBs is pretty quick and I didn't notice significant performance issues when I was benchmarking. I tested on very large Datasette deployments (hundreds of DBs, millions of rows). See [""Many small queries are efficient in sqlite""](https://sqlite.org/np1queryprob.html) for more information on the rationale here. Also note that in the [datasette-live-config](https://github.com/next-LI/datasette-live-config) reference plugin, the DB connection is cached, so that eliminated most of the performance worries we had. If you need to ensure fresh metadata is being read inside of a `render_cell` hook specifically, you don't need to do anything further! `get_metadata` gets called before `render_cell` every request, so it already has access to the synced meta. There shouldn't be a need to call `get_metadata(...)` or `metadata(...)` inside `render_cell`, you can just use `datasette._metadata_local` if you're really worried about performance. > The plugin is close, but looks like it only grabs remote metadata, is that right? Instead what I'm wanting is to grab metadata embedded in the attached databases. Yes correct, the datadette-remote-metadata plugin doesn't do that. But the datasette-live-config plugin does. [It supports a `__metadata` table](https://github.com/next-LI/datasette-live-config/blob/main/datasette_live_config/__init__.py#L107-L138) that, when it exists on an attached DB, gets pulled into the Datasette internal `_metadata` and is also accessible via `get_metadata`. Updating is instantaneous so there's no gotchas for users or security issues for users relying on the metadata-based permissions. Simon talked about eventually making something like this a standard feature of Datasette, but I'm not sure what the status is on that! Good luck!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-1066143991,https://api.github.com/repos/simonw/datasette/issues/1384,1066143991,IC_kwDOBm6k_c4_jBD3,167160,khusmann,2022-03-13T17:13:09Z,2022-03-13T17:13:09Z,NONE,"Thanks for taking the time to reply @brandonrobertz , this is really helpful info. > See ""Many small queries are efficient in sqlite"" for more information on the rationale here. Also note that in the datasette-live-config reference plugin, the DB connection is cached, so that eliminated most of the performance worries we had. Ah, that's nifty! Yeah, then caching on the python side is likely a waste :) I'm new to working with sqlite so this is super good to know the many-small-queries is a common pattern > I tested on very large Datasette deployments (hundreds of DBs, millions of rows). For my reference, did you include a `render_cell` plugin calling `get_metadata` in those tests? I'm less concerned now that I know a little more about sqlite's caching, but that special situation will jump you to a few orders of magnitude above what the sqlite article describes (e.g. 200 vs 20,000 queries+metadata merges for a page displaying 100 rows of a 200 column table). It wouldn't scale with db size as much as # of visible cells being rendered on the page, although they would be identical queries I suppose so will cache well. (If you didn't test this specific situation, no worries -- I'm just trying to calibrate my intuition on this and can do my own benchmarks at some point.) > Simon talked about eventually making something like this a standard feature of Datasette Yeah, getting metadata (and static pages as well for that matter) from internal tables definitely has my vote for including as a standard feature! Its really nice to be able to distribute a single *.db with all the metadata and static pages bundled. My metadata are sufficiently complex/domain specific that it makes sense to continue on my own plugin for now, but I'll be thinking about more general parts I can spin off as possible contributions to liveconfig (if you're open to them) or other plugins in this ecosystem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-1066169718,https://api.github.com/repos/simonw/datasette/issues/1384,1066169718,IC_kwDOBm6k_c4_jHV2,2670795,brandonrobertz,2022-03-13T19:48:49Z,2022-03-13T19:48:49Z,CONTRIBUTOR,"> For my reference, did you include a `render_cell` plugin calling `get_metadata` in those tests? You shouldn't need to do this, as I mentioned previously. The code inside `render_cell` hook already has access to the most recently sync'd metadata via `datasette._metadata_local`. Refreshing the metadata for every cell seems ... excessive.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-1066194130,https://api.github.com/repos/simonw/datasette/issues/1384,1066194130,IC_kwDOBm6k_c4_jNTS,167160,khusmann,2022-03-13T22:23:04Z,2022-03-13T22:23:04Z,NONE,"Ah, sorry, I didn't get what you were saying you the first time. Using _metadata_local in that way makes total sense -- I agree, refreshing metadata each cell was seeming quite excessive. Now I'm on the same page! :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1384#issuecomment-1066222323,https://api.github.com/repos/simonw/datasette/issues/1384,1066222323,IC_kwDOBm6k_c4_jULz,2670795,brandonrobertz,2022-03-14T00:36:42Z,2022-03-14T00:36:42Z,CONTRIBUTOR,"> Ah, sorry, I didn't get what you were saying you the first time. Using _metadata_local in that way makes total sense -- I agree, refreshing metadata each cell was seeming quite excessive. Now I'm on the same page! :) All good. Report back any issues you find with this stuff. Metadata/dynamic config hasn't been tested widely outside of what I've done AFAIK. If you find a strong use case for async meta, it's going to be better to know sooner rather than later!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata, https://github.com/simonw/datasette/issues/1657#issuecomment-1067413691,https://api.github.com/repos/simonw/datasette/issues/1657,1067413691,IC_kwDOBm6k_c4_n3C7,9599,simonw,2022-03-14T23:37:42Z,2022-03-14T23:37:42Z,OWNER,"Relevant: https://datatracker.ietf.org/doc/html/rfc3986#section-2.1 ``` reserved = gen-delims / sub-delims gen-delims = "":"" / ""/"" / ""?"" / ""#"" / ""["" / ""]"" / ""@"" sub-delims = ""!"" / ""$"" / ""&"" / ""'"" / ""("" / "")"" / ""*"" / ""+"" / "","" / "";"" / ""="" ``` Notably `~` is not in either of those lists.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1067414156,https://api.github.com/repos/simonw/datasette/issues/1657,1067414156,IC_kwDOBm6k_c4_n3KM,9599,simonw,2022-03-14T23:38:41Z,2022-03-14T23:38:41Z,OWNER,"And in https://datatracker.ietf.org/doc/html/rfc3986#section-2.3 ""Unreserved Characters"": unreserved = ALPHA / DIGIT / ""-"" / ""."" / ""_"" / ""~""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1067423720,https://api.github.com/repos/simonw/datasette/issues/1657,1067423720,IC_kwDOBm6k_c4_n5fo,9599,simonw,2022-03-14T23:59:56Z,2022-03-14T23:59:56Z,OWNER,"Updated test: ```python @pytest.mark.parametrize( ""original,expected"", ( (""abc"", ""abc""), (""/foo/bar"", ""~2Ffoo~2Fbar""), (""/-/bar"", ""~2F-~2Fbar""), (""-/db-/table.csv"", ""-~2Fdb-~2Ftable~2Ecsv""), (r""%~-/"", ""~25~7E-~2F""), (""~25~7E~2D~2F"", ""~7E25~7E7E~7E2D~7E2F""), ), ) def test_tilde_encoding(original, expected): actual = utils.tilde_encode(original) assert actual == expected # And test round-trip assert original == utils.tilde_decode(actual) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1067381556,https://api.github.com/repos/simonw/datasette/issues/1657,1067381556,IC_kwDOBm6k_c4_nvM0,9599,simonw,2022-03-14T22:57:27Z,2022-03-14T22:57:45Z,OWNER,"The problem with the [dash encoding mechanism](https://simonwillison.net/2022/Mar/5/dash-encoding/) is that it turns out dashes are used in a LOT of existing Datasette instances - much of https://fivethirtyeight.datasettes.com/fivethirtyeight for example, and even https://datasette.io/ itself: https://datasette.io/dogsheep-index It's pretty ugly to force all of those to change to their dash-encoded equivalent - and in fact it broke https://datasette.io/ in a subtle way: - https://github.com/simonw/datasette.io/issues/94 I'm going to try using `~` instead and see if that works as well and causes less breakage to existing sites.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1067382232,https://api.github.com/repos/simonw/datasette/issues/1657,1067382232,IC_kwDOBm6k_c4_nvXY,9599,simonw,2022-03-14T22:58:47Z,2022-03-14T22:58:47Z,OWNER,"Asked about this [on Twitter](https://twitter.com/simonw/status/1503499169775849473): > Anyone ever seen a proxy or other URL handling system do anything surprising with the tilde ""~"" character? > > I'm considering it as an escaping character, in place of ""-"" as described in Replies so far seem like it should be OK - Apache has supported this for home directories for a couple of decades now without any problems.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1651#issuecomment-1067382442,https://api.github.com/repos/simonw/datasette/issues/1651,1067382442,IC_kwDOBm6k_c4_nvaq,9599,simonw,2022-03-14T22:59:10Z,2022-03-14T22:59:10Z,OWNER,"This work is now blocked on: - https://github.com/simonw/datasette/issues/1657","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/pull/1659#issuecomment-1068193035,https://api.github.com/repos/simonw/datasette/issues/1659,1068193035,IC_kwDOBm6k_c4_q1UL,22429695,codecov[bot],2022-03-15T16:28:25Z,2022-03-15T17:56:09Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1659](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (85dde28) into [main](https://codecov.io/gh/simonw/datasette/commit/c10cd48baf106659bf3f129ad7bfb2226be73821?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c10cd48) will **increase** coverage by `0.03%`. > The diff coverage is `100.00%`. > :exclamation: Current head 85dde28 differs from pull request most recent head 99b8263. Consider uploading reports for the commit 99b8263 to get more accurate results ```diff @@ Coverage Diff @@ ## main #1659 +/- ## ========================================== + Coverage 92.06% 92.10% +0.03% ========================================== Files 34 34 Lines 4576 4584 +8 ========================================== + Hits 4213 4222 +9 + Misses 363 362 -1 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.36% <100.00%> (ø)` | | | [datasette/url\_builder.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3VybF9idWlsZGVyLnB5) | `100.00% <100.00%> (ø)` | | | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.84% <100.00%> (-0.13%)` | :arrow_down: | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `96.07% <100.00%> (+0.58%)` | :arrow_up: | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.21% <100.00%> (+0.01%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c10cd48...99b8263](https://codecov.io/gh/simonw/datasette/pull/1659?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1169895600,Tilde encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068125636,https://api.github.com/repos/simonw/datasette/issues/1657,1068125636,IC_kwDOBm6k_c4_qk3E,9599,simonw,2022-03-15T15:30:54Z,2022-03-15T15:30:54Z,OWNER,I've made a real mess of this. I'm going to revert Datasette`main` back to the last commit that passed the tests and try this again in a branch.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068126821,https://api.github.com/repos/simonw/datasette/issues/1657,1068126821,IC_kwDOBm6k_c4_qlJl,9599,simonw,2022-03-15T15:31:54Z,2022-03-15T15:31:54Z,OWNER,The state I had got to prior to that revert is in https://github.com/simonw/datasette/tree/issue-1657-wip,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1658#issuecomment-1068138578,https://api.github.com/repos/simonw/datasette/issues/1658,1068138578,IC_kwDOBm6k_c4_qoBS,9599,simonw,2022-03-15T15:42:49Z,2022-03-15T15:42:49Z,OWNER,"Easiest way to do this was with three reverts, then cherry-pick back the code of conduct.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1169840669,Revert main to version that passes tests, https://github.com/simonw/datasette/issues/1657#issuecomment-1068148013,https://api.github.com/repos/simonw/datasette/issues/1657,1068148013,IC_kwDOBm6k_c4_qqUt,9599,simonw,2022-03-15T15:50:15Z,2022-03-15T15:50:15Z,OWNER,"The thing that broke everything was this change: <img width=""569"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/158417416-285018c1-af45-49bc-9687-829c0c17ff05.png""> I'm going to bring back the horrible `get_format()` method for the moment, with its weird mutations of the `args` object, then try and get rid of it again later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/pull/1656#issuecomment-1068154183,https://api.github.com/repos/simonw/datasette/issues/1656,1068154183,IC_kwDOBm6k_c4_qr1H,22429695,codecov[bot],2022-03-15T15:55:34Z,2022-03-15T15:55:34Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1656](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (5d9883f) into [main](https://codecov.io/gh/simonw/datasette/commit/c10cd48baf106659bf3f129ad7bfb2226be73821?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c10cd48) will **not change** coverage. > The diff coverage is `n/a`. ```diff @@ Coverage Diff @@ ## main #1656 +/- ## ======================================= Coverage 92.06% 92.06% ======================================= Files 34 34 Lines 4576 4576 ======================================= Hits 4213 4213 Misses 363 363 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c10cd48...5d9883f](https://codecov.io/gh/simonw/datasette/pull/1656?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168357113,"Update pytest requirement from <7.1.0,>=5.2.2 to >=5.2.2,<7.2.0", https://github.com/simonw/datasette/issues/1657#issuecomment-1068181623,https://api.github.com/repos/simonw/datasette/issues/1657,1068181623,IC_kwDOBm6k_c4_qyh3,9599,simonw,2022-03-15T16:18:23Z,2022-03-15T16:18:23Z,OWNER,Moving this to a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/857#issuecomment-1068450483,https://api.github.com/repos/simonw/datasette/issues/857,1068450483,IC_kwDOBm6k_c4_r0Kz,9599,simonw,2022-03-15T20:43:55Z,2022-03-15T20:43:55Z,OWNER,Dupe of #1510.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642297505,Comprehensive documentation for variables made available to templates, https://github.com/simonw/datasette/issues/1439#issuecomment-1068461449,https://api.github.com/repos/simonw/datasette/issues/1439,1068461449,IC_kwDOBm6k_c4_r22J,9599,simonw,2022-03-15T20:51:26Z,2022-03-15T20:51:26Z,OWNER,I'm happy with this now that I've landed Tilde encoding in #1657.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0, https://github.com/simonw/datasette/issues/1657#issuecomment-1068296042,https://api.github.com/repos/simonw/datasette/issues/1657,1068296042,IC_kwDOBm6k_c4_rOdq,9599,simonw,2022-03-15T18:05:54Z,2022-03-15T18:05:54Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#tilde-encoding,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068306916,https://api.github.com/repos/simonw/datasette/issues/1657,1068306916,IC_kwDOBm6k_c4_rRHk,9599,simonw,2022-03-15T18:15:11Z,2022-03-15T18:15:11Z,OWNER,Now live here: https://fivethirtyeight.datasettes.com/fivethirtyeight/august-senate-polls~2Faugust_senate_polls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1657#issuecomment-1068318454,https://api.github.com/repos/simonw/datasette/issues/1657,1068318454,IC_kwDOBm6k_c4_rT72,9599,simonw,2022-03-15T18:25:11Z,2022-03-15T18:25:11Z,OWNER,"Demo: - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.csv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1168995756,Tilde encoding: use ~ instead of - for dash-encoding, https://github.com/simonw/datasette/issues/1651#issuecomment-1068319530,https://api.github.com/repos/simonw/datasette/issues/1651,1068319530,IC_kwDOBm6k_c4_rUMq,9599,simonw,2022-03-15T18:25:42Z,2022-03-15T18:25:42Z,OWNER,"Done: - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.csv - https://latest.datasette.io/fixtures/table~2Fwith~2Fslashes~2Ecsv.json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1161584460,Get rid of the no-longer necessary ?_format=json hack for tables called x.json, https://github.com/simonw/datasette/issues/1062#issuecomment-1068327874,https://api.github.com/repos/simonw/datasette/issues/1062,1068327874,IC_kwDOBm6k_c4_rWPC,9599,simonw,2022-03-15T18:33:49Z,2022-03-15T18:33:49Z,OWNER,"I can get regular `.json` to stream too, using the pattern described in this TIL: https://til.simonwillison.net/python/output-json-array-streaming","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows, https://github.com/simonw/datasette/issues/1660#issuecomment-1068415072,https://api.github.com/repos/simonw/datasette/issues/1660,1068415072,IC_kwDOBm6k_c4_rrhg,9599,simonw,2022-03-15T20:02:36Z,2022-03-15T20:02:36Z,OWNER,"This is one of the worst bits - the `get_format()` method on the `DataView` base class actually modifies `args`, including removing keys! Really confusing: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L454-L482 Then `BaseView` has some surprising responsibilities. It has a utility helper for checking multiple permissions at once: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L81-L105 And its own render method that adds extra stuff to the template context and handles the rel: alternate header: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L131-L157 Then `DataView` does all sorts of weird stuff - from handling database hashes (which I want to remove, see #647): https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L206-L219 To streaming CSV responses: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L286-L308 To handling SQLite exceptions: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L514-L526 And a ton more. It' s a big mess.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1660#issuecomment-1068417357,https://api.github.com/repos/simonw/datasette/issues/1660,1068417357,IC_kwDOBm6k_c4_rsFN,9599,simonw,2022-03-15T20:05:08Z,2022-03-15T20:05:08Z,OWNER,"`DataView` is used as the base class for: - `DatabaseView` - `DatabaseDownload` (just so the permissions checks can be called) - `QueryView` - which isn't routed to directly, it's called from `DatabaseView` if `?sql=` is available and `TableView` for canned queries - `RowTableShared` which is the base class for `TableView` and `RowView`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1660#issuecomment-1068418619,https://api.github.com/repos/simonw/datasette/issues/1660,1068418619,IC_kwDOBm6k_c4_rsY7,9599,simonw,2022-03-15T20:06:19Z,2022-03-15T20:06:19Z,OWNER,"Also related: - #878 - #1512 - #1518 - #870 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1509#issuecomment-1068445412,https://api.github.com/repos/simonw/datasette/issues/1509,1068445412,IC_kwDOBm6k_c4_ry7k,9599,simonw,2022-03-15T20:37:50Z,2022-03-15T20:38:56Z,OWNER,"... maybe Datasette itself should include interactive API documentation, in addition to documenting it in the manual? `/dbname/table/-/apidocs` could return documentation about the specific table, taking into account columns and types.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054243511,Datasette 1.0 JSON API (and documentation), https://github.com/simonw/datasette/issues/1510#issuecomment-1068443509,https://api.github.com/repos/simonw/datasette/issues/1510,1068443509,IC_kwDOBm6k_c4_ryd1,9599,simonw,2022-03-15T20:35:29Z,2022-03-15T20:35:29Z,OWNER,If I set a rule that everything available in the template context MUST also be available via the JSON API (maybe through an extras mechanism) I can combine this with API documentation and solve both at once.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054244712,Datasette 1.0 documented template context (maybe via API docs), https://github.com/simonw/datasette/issues/1509#issuecomment-1068444767,https://api.github.com/repos/simonw/datasette/issues/1509,1068444767,IC_kwDOBm6k_c4_ryxf,9599,simonw,2022-03-15T20:37:03Z,2022-03-15T20:37:03Z,OWNER,"Idea: I could add Pydantic https://pydantic-docs.helpmanual.io/usage/schema/ as an optional test dependency and use it to generate JSON schemas and run validation against examples in the API documentation. Maybe generate API documentation from it too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054243511,Datasette 1.0 JSON API (and documentation), https://github.com/simonw/datasette/issues/1661#issuecomment-1068728484,https://api.github.com/repos/simonw/datasette/issues/1661,1068728484,IC_kwDOBm6k_c4_s4Ck,9599,simonw,2022-03-16T04:47:39Z,2022-03-16T04:47:39Z,OWNER,https://datasette.io/plugins/datasette-hashed-urls is released now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1663#issuecomment-1068742624,https://api.github.com/repos/simonw/datasette/issues/1663,1068742624,IC_kwDOBm6k_c4_s7fg,9599,simonw,2022-03-16T05:17:45Z,2022-03-16T05:17:45Z,OWNER,Should be documented here: https://docs.datasette.io/en/stable/internals.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170554975,Document the internals that were used in datasette-hashed-urls, https://github.com/simonw/datasette/issues/647#issuecomment-1068539404,https://api.github.com/repos/simonw/datasette/issues/647,1068539404,IC_kwDOBm6k_c4_sJ4M,9599,simonw,2022-03-15T22:49:01Z,2022-03-15T22:49:01Z,OWNER,"I shipped the first version of this: https://github.com/simonw/datasette-hashed-urls Next step: test it with a live demo: - https://github.com/simonw/datasette-hashed-urls/issues/2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/647#issuecomment-1068552696,https://api.github.com/repos/simonw/datasette/issues/647,1068552696,IC_kwDOBm6k_c4_sNH4,9599,simonw,2022-03-15T23:13:06Z,2022-03-15T23:13:06Z,OWNER,"The plugin works. I'm going to implement one last feature for it: - https://github.com/simonw/datasette-hashed-urls/issues/3 Then I can remove hashed URL mode in a separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin, https://github.com/simonw/datasette/issues/1661#issuecomment-1068553454,https://api.github.com/repos/simonw/datasette/issues/1661,1068553454,IC_kwDOBm6k_c4_sNTu,9599,simonw,2022-03-15T23:14:37Z,2022-03-15T23:14:37Z,OWNER,"This is going to simplify the code in the various view classes substantially: - #1660","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1068554827,https://api.github.com/repos/simonw/datasette/issues/1661,1068554827,IC_kwDOBm6k_c4_sNpL,9599,simonw,2022-03-15T23:16:58Z,2022-03-15T23:18:58Z,OWNER,"If you attempt to use the [old setting](https://docs.datasette.io/en/stable/settings.html#hash-urls): datasette mydatabase.db --setting hash_urls 1 It should error with a message saying that the feature has been moved to a plugin. I'll do this with a `deprecated_settings` mechanism so the error can be detected even though `datasette --help-settings` will no longer return the setting. https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/cli.py#L479-L489","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1068628839,https://api.github.com/repos/simonw/datasette/issues/1661,1068628839,IC_kwDOBm6k_c4_sftn,9599,simonw,2022-03-16T01:21:36Z,2022-03-16T01:21:48Z,OWNER,"From https://docs.datasette.io/en/0.60.2/performance.html#hashed-url-mode > You can enable these hashed URLs in two ways: using the [hash_urls](https://docs.datasette.io/en/0.60.2/settings.html#setting-hash-urls) configuration setting (which affects all requests to Datasette) or via the `?_hash=1` query string parameter (which only applies to the current request). I'm going to drop` ?_hash=1` entirely. I'd actually forgotten that feature existed!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1661#issuecomment-1068630353,https://api.github.com/repos/simonw/datasette/issues/1661,1068630353,IC_kwDOBm6k_c4_sgFR,9599,simonw,2022-03-16T01:24:56Z,2022-03-16T01:25:49Z,OWNER,"Here's the only bit of code that references that `_hash` mechanism: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/datasette/views/base.py#L259-L265 And here's the test: https://github.com/simonw/datasette/blob/77a904fea14f743560af9cc668146339bdbbd0a9/tests/test_api.py#L828-L854 Related issue: - #471","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170355774,Remove Hashed URL mode, https://github.com/simonw/datasette/issues/1532#issuecomment-1069570893,https://api.github.com/repos/simonw/datasette/issues/1532,1069570893,IC_kwDOBm6k_c4_wFtN,9599,simonw,2022-03-16T20:11:41Z,2022-03-16T20:13:34Z,OWNER,"Could also build a CLI Rich/Textual app to exercise the API - which could embed Datasette as a dependency and work using `datasette.client.get(...)` calls. Could be a plugin that adds a `datasette tui` command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065429936,Use datasette-table Web Component to guide the design of the JSON API for 1.0, https://github.com/simonw/datasette/issues/1549#issuecomment-1087428593,https://api.github.com/repos/simonw/datasette/issues/1549,1087428593,IC_kwDOBm6k_c5A0Nfx,536941,fgregg,2022-04-04T11:17:13Z,2022-04-04T11:17:13Z,CONTRIBUTOR,"another way to get the behavior of downloading the file is to use the download attribute of the anchor tag https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#attr-download","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,Redesign CSV export to improve usability, https://github.com/simonw/datasette/issues/1177#issuecomment-1074017633,https://api.github.com/repos/simonw/datasette/issues/1177,1074017633,IC_kwDOBm6k_c5ABDVh,9599,simonw,2022-03-21T15:08:51Z,2022-03-21T15:08:51Z,OWNER,"Related: - #1062 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780153562,Ability to stream all rows as newline-delimited JSON, https://github.com/simonw/datasette/issues/526#issuecomment-1074019047,https://api.github.com/repos/simonw/datasette/issues/526,1074019047,IC_kwDOBm6k_c5ABDrn,9599,simonw,2022-03-21T15:09:56Z,2022-03-21T15:09:56Z,OWNER,I should research how much overhead creating a new connection costs - it may be that an easy way to solve this is to create A dedicated connection for the query and then close that connection at the end.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/1660#issuecomment-1074136176,https://api.github.com/repos/simonw/datasette/issues/1660,1074136176,IC_kwDOBm6k_c5ABgRw,9599,simonw,2022-03-21T16:38:46Z,2022-03-21T16:38:46Z,OWNER,"I'm going to refactor this stuff out and document it so it can be easily used by plugins: https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L69-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,Refactor and simplify Datasette routing and views, https://github.com/simonw/datasette/issues/1675#issuecomment-1074142617,https://api.github.com/repos/simonw/datasette/issues/1675,1074142617,IC_kwDOBm6k_c5ABh2Z,9599,simonw,2022-03-21T16:45:27Z,2022-03-21T16:45:27Z,OWNER,"Though at that point `check_permission` is such a light wrapper around `self.ds.permission_allowed()` that there's little point in it existing at all. So maybe `check_permisions()` becomes `ds.permissions_allowed()`. `permission_allowed()` v.s. `permissions_allowed()` is a bit of a subtle naming difference, but I think it works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074143209,https://api.github.com/repos/simonw/datasette/issues/1675,1074143209,IC_kwDOBm6k_c5ABh_p,9599,simonw,2022-03-21T16:46:05Z,2022-03-21T16:46:05Z,OWNER,"The other difference though is that `ds.permission_allowed(...)` works against an actor, while `check_permission()` works against a request (though just to access `request.actor`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074141457,https://api.github.com/repos/simonw/datasette/issues/1675,1074141457,IC_kwDOBm6k_c5ABhkR,9599,simonw,2022-03-21T16:44:09Z,2022-03-21T16:44:09Z,OWNER,"A slightly odd thing about these methods is that they either fail silently or they raise a `Forbidden` exception. Maybe they should instead return `True` or `False` and the calling code could decide if it wants to raise the exception? That would make them more usable and a little less surprising.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074158890,https://api.github.com/repos/simonw/datasette/issues/1675,1074158890,IC_kwDOBm6k_c5ABl0q,9599,simonw,2022-03-21T16:57:15Z,2022-03-21T16:57:15Z,OWNER,"Idea: `ds.permission_allowed()` continues to just return `True` or `False`. A new `ds.ensure_permissions(...)` method is added which raises a `Forbidden` exception if a check fails (hence the different name)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074156779,https://api.github.com/repos/simonw/datasette/issues/1675,1074156779,IC_kwDOBm6k_c5ABlTr,9599,simonw,2022-03-21T16:55:08Z,2022-03-21T16:56:02Z,OWNER,"One benefit of the current design of `check_permissions` that raises an exception is that the exception includes information on WHICH of the permission checks failed. Returning just `True` or `False` loses that information. I could return an object which evaluates to `False` but also carries extra information? Bit weird, I've never seen anything like that in other Python code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074161523,https://api.github.com/repos/simonw/datasette/issues/1675,1074161523,IC_kwDOBm6k_c5ABmdz,9599,simonw,2022-03-21T16:59:55Z,2022-03-21T17:00:03Z,OWNER,Also calling that function `permissions_allowed()` is confusing because there is a plugin hook with a similar name already: https://docs.datasette.io/en/stable/plugin_hooks.html#permission-allowed-datasette-actor-action-resource,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1675#issuecomment-1074177827,https://api.github.com/repos/simonw/datasette/issues/1675,1074177827,IC_kwDOBm6k_c5ABqcj,9599,simonw,2022-03-21T17:14:31Z,2022-03-21T17:14:31Z,OWNER,"Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions > This method allows multiple permissions to be checked at onced. It raises a `datasette.Forbidden` exception if any of the checks are denied before one of them is explicitly granted. > > This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns `True` or not a single one of them returns `False`: That's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,Extract out `check_permissions()` from `BaseView, https://github.com/simonw/datasette/issues/1676#issuecomment-1074178865,https://api.github.com/repos/simonw/datasette/issues/1676,1074178865,IC_kwDOBm6k_c5ABqsx,9599,simonw,2022-03-21T17:15:27Z,2022-03-21T17:15:27Z,OWNER,This method here: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/app.py#L632-L664,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,"Reconsider ensure_permissions() logic, can it be less confusing?", https://github.com/simonw/datasette/issues/1676#issuecomment-1074180312,https://api.github.com/repos/simonw/datasette/issues/1676,1074180312,IC_kwDOBm6k_c5ABrDY,9599,simonw,2022-03-21T17:16:45Z,2022-03-21T17:16:45Z,OWNER,"When looking at this code earlier I assumed that the following would check each permission in turn and fail if any of them failed: ```python await self.ds.ensure_permissions( request.actor, [ (""view-table"", (database, table)), (""view-database"", database), ""view-instance"", ] ) ``` But it's not quite that simple: if any of them fail, it fails... but if an earlier one returns `True` the whole stack passes even if there would have been a failure later on! If that is indeed the right abstraction, I need to work to make the documentation as clear as possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,"Reconsider ensure_permissions() logic, can it be less confusing?", https://github.com/simonw/datasette/issues/1677#issuecomment-1074184240,https://api.github.com/repos/simonw/datasette/issues/1677,1074184240,IC_kwDOBm6k_c5ABsAw,9599,simonw,2022-03-21T17:20:17Z,2022-03-21T17:20:17Z,OWNER,"https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/views/base.py#L69-L77 This is weirdly different from how `check_permissions()` used to work, in that it doesn't differentiate between `None` and `False`. https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L79-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175694248,Remove `check_permission()` from `BaseView`,