html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/datasette/issues/2102#issuecomment-1640064620,https://api.github.com/repos/simonw/datasette/issues/2102,1640064620,IC_kwDOBm6k_c5hwWZs,9599,simonw,2023-07-18T11:47:21Z,2023-07-18T11:47:21Z,OWNER,"I think I've figured out the problem here. The question being asked is ""can this actor access this resource, which is within this database within this instance"". The answer to this question needs to consider the full set of questions at once - yes they can access within this instance IF they have access to the specified table and that's the table being asked about. But the questions are currently being asked independently, which means the plugin hook acting on `view-instance` can't see that the answer here should be yes because it's actually about a table that the actor has explicit permission to view. So I think I may need to redesign the plugin hook to always see the full hierarchy of checks, not just a single check at a time. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638926655,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1638926655,IC_kwDOCGYnMM5hsAk_,9599,simonw,2023-07-17T21:42:37Z,2023-07-17T21:42:37Z,OWNER,"I really like this. I'm also interested in: - Plugins that make new custom SQL functions available - similar to this Datasette hook: https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette - Plugins that register functions that can be used as recipes for `sqlite-utils convert` https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes The upload-data-to-Datasette problem is planned to be solved by a future version of https://github.com/simonw/dclient ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/datasette/issues/2102#issuecomment-1638567228,https://api.github.com/repos/simonw/datasette/issues/2102,1638567228,IC_kwDOBm6k_c5hqo08,9599,simonw,2023-07-17T17:24:19Z,2023-07-17T17:25:12Z,OWNER,"Confirmed that this is an issue with regular Datasette signed tokens as well. I created one on https://latest.datasette.io/-/create-token with these details: ```json { ""_r"": { ""r"": { ""fixtures"": { ""sortable"": [ ""vt"" ] } } }, ""a"": ""root"", ""d"": 3600, ""t"": 1689614483 } ``` Run like this: ``` curl -H 'Authorization: Bearer dstok_eyJhIjoicm9vdCIsInQiOjE2ODk2MTQ0ODMsImQiOjM2MDAsIl9yIjp7InIiOnsiZml4dHVyZXMiOnsic29ydGFibGUiOlsidnQiXX19fX0.n-VGxxawz1Q0WK7sqLfhXUgcvY0' \ https://latest.datasette.io/fixtures/sortable.json ``` Returned an HTML Forbidden page: ```html Forbidden ... ``` Same token againts `/-/actor.json` returns: ```json { ""actor"": { ""id"": ""root"", ""token"": ""dstok"", ""_r"": { ""r"": { ""fixtures"": { ""sortable"": [ ""vt"" ] } } }, ""token_expires"": 1689618083 } } ``` Reminder - `""_r""` means restrict, `""r""` means resource.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2104#issuecomment-1638552567,https://api.github.com/repos/simonw/datasette/issues/2104,1638552567,IC_kwDOBm6k_c5hqlP3,9599,simonw,2023-07-17T17:14:20Z,2023-07-17T17:14:20Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/database.py#L391-L451,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1808215339,Tables starting with an underscore should be treated as hidden, https://github.com/simonw/datasette/issues/2102#issuecomment-1636093730,https://api.github.com/repos/simonw/datasette/issues/2102,1636093730,IC_kwDOBm6k_c5hhM8i,9599,simonw,2023-07-14T16:26:27Z,2023-07-14T16:32:49Z,OWNER,"Here's that crucial comment: > If _r is defined then we use those to further restrict the actor. > >Crucially, we only use this to say NO (return False) - we never use it to return YES (True) because that might over-ride other restrictions placed on this actor So that's why I implemented it like this. The goal here is to be able to issue a token which can't do anything _more_ than the actor it is associated with, but CAN be configured to do less. So I think the solution here is for the `_r` checking code to perhaps implement its own view cascade logic - it notices if you have `view-table` and consequently fails to block `view-table` and `view-instance`. I'm not sure that's going to work though - would that mean that granting `view-table` grants `view-database` in a surprising and harmful way? Maybe that's OK: if you have `view-database` but permission checks fail for individual tables and queries you shouldn't be able to see a thing that you shouldn't. Need to verify that though. Also, do `Permission` instances have enough information to implement this kind of cascade without hard-coding anything? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1636053060,https://api.github.com/repos/simonw/datasette/issues/2102,1636053060,IC_kwDOBm6k_c5hhDBE,9599,simonw,2023-07-14T15:51:36Z,2023-07-14T16:14:05Z,OWNER,"This might only be an issue with the code that checks `_r` on actors. https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/default_permissions.py#L185-L222 Added in https://github.com/simonw/datasette/commit/bcc781f4c50a8870e3389c4e60acb625c34b0317 - refs: - #1855 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1636042066,https://api.github.com/repos/simonw/datasette/issues/2102,1636042066,IC_kwDOBm6k_c5hhAVS,9599,simonw,2023-07-14T15:41:54Z,2023-07-14T15:42:32Z,OWNER,"I tried some code spelunking and came across https://github.com/simonw/datasette/commit/d6e03b04302a0852e7133dc030eab50177c37be7 which says: > - If you have table permission but not database permission you can now view the table page Refs: - #832 Which suggests that my initial design decision wasn't what appears to be implemented today. Needs more investigation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1636040164,https://api.github.com/repos/simonw/datasette/issues/2102,1636040164,IC_kwDOBm6k_c5hg_3k,9599,simonw,2023-07-14T15:40:21Z,2023-07-14T15:40:21Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/app.py#L822-L855","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1636036312,https://api.github.com/repos/simonw/datasette/issues/2102,1636036312,IC_kwDOBm6k_c5hg-7Y,9599,simonw,2023-07-14T15:37:14Z,2023-07-14T15:37:14Z,OWNER,"I think I made this decision because I was thinking about default deny: obviously if a user has been denied access to a database. It doesn't make sense that they could access tables within it. But now that I am spending more time with authentication tokens, which default to denying everything, except for the things that you have explicitly listed, this policy, no longer makes as much sense. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2101#issuecomment-1634443907,https://api.github.com/repos/simonw/datasette/issues/2101,1634443907,IC_kwDOBm6k_c5ha6KD,9599,simonw,2023-07-13T15:24:17Z,2023-07-13T15:24:17Z,OWNER,https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/views/table.py#L486-L506,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1803264272,alter: true support for JSON write API, https://github.com/simonw/datasette/pull/2052#issuecomment-1629337927,https://api.github.com/repos/simonw/datasette/issues/2052,1629337927,IC_kwDOBm6k_c5hHblH,9599,simonw,2023-07-10T16:43:38Z,2023-07-10T16:44:23Z,OWNER,"I tried running this locally just now. I made one edit: ```diff diff --git a/demos/plugins/example_js_manager_plugins.py b/demos/plugins/example_js_manager_plugins.py index 7bdb9f3f..f9dfa8e6 100644 --- a/demos/plugins/example_js_manager_plugins.py +++ b/demos/plugins/example_js_manager_plugins.py @@ -15,6 +15,6 @@ def extra_js_urls(view_name): if view_name in PERMITTED_VIEWS: return [ { - ""url"": f""/-/demos/plugins/static/table-example-plugins.js"", + ""url"": f""/static/table-example-plugins.js"", } ] ``` And then started it running like this: ```bash wget https://datasette.io/content.db ``` ```bash datasette content.db \ --plugins-dir demos/plugins \ --static static:datasette/demos/plugins/static ``` It didn't quite work for me - I got this error on a table page: And this error on a query page: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627598570,https://api.github.com/repos/simonw/sqlite-utils/issues/566,1627598570,IC_kwDOCGYnMM5hAy7q,9599,simonw,2023-07-09T04:13:34Z,2023-07-09T04:13:34Z,OWNER,On consulting https://pypi.org/project/tabulate/ it looks like most of those formats don't actually makes sense without headers - so the right thing here might be to raise an error if `--fmt` and `--no-headers` are used at the same time.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795219865,`--no-headers` doesn't work on most formats, https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627597872,https://api.github.com/repos/simonw/sqlite-utils/issues/566,1627597872,IC_kwDOCGYnMM5hAyww,9599,simonw,2023-07-09T04:09:56Z,2023-07-09T04:09:56Z,OWNER,"Thanks, looks like a bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795219865,`--no-headers` doesn't work on most formats, https://github.com/simonw/datasette/issues/1153#issuecomment-1627480353,https://api.github.com/repos/simonw/datasette/issues/1153,1627480353,IC_kwDOBm6k_c5hAWEh,9599,simonw,2023-07-08T20:09:48Z,2023-07-08T20:09:48Z,OWNER,https://docs.datasette.io/en/latest/writing_plugins.html#writing-plugins-that-accept-configuration is fixed now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/datasette/issues/1153#issuecomment-1627478910,https://api.github.com/repos/simonw/datasette/issues/1153,1627478910,IC_kwDOBm6k_c5hAVt-,9599,simonw,2023-07-08T20:01:19Z,2023-07-08T20:01:19Z,OWNER,"Some examples: - https://docs.datasette.io/en/latest/sql_queries.html#canned-queries - https://docs.datasette.io/en/latest/sql_queries.html#canned-query-parameters - https://docs.datasette.io/en/latest/authentication.html#access-to-an-instance - https://docs.datasette.io/en/latest/facets.html#facets-in-metadata - https://docs.datasette.io/en/latest/full_text_search.html#configuring-full-text-search-for-a-table-or-view - https://docs.datasette.io/en/latest/metadata.html - https://docs.datasette.io/en/latest/custom_templates.html#custom-css-and-javascript - https://docs.datasette.io/en/latest/plugins.html#plugin-configuration I need to fix this section: https://docs.datasette.io/en/latest/writing_plugins.html#writing-plugins-that-accept-configuration","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/datasette/issues/1153#issuecomment-1627455892,https://api.github.com/repos/simonw/datasette/issues/1153,1627455892,IC_kwDOBm6k_c5hAQGU,9599,simonw,2023-07-08T18:39:19Z,2023-07-08T18:39:19Z,OWNER,"``` ERROR: Could not find a version that satisfies the requirement Sphinx==6.1.3; extra == ""docs"" (from datasette[docs,test]) (from versions: 0.1.61611, 0.1.61798, 0.1.61843, 0.1.61945, 0.1.61950, 0.2, 0.3, 0.4, 0.4.1, 0.4.2, 0.4.3, 0.5, 0.5.1, 0.5.2b1, 0.5.2, 0.6b1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.6.7, 1.0b1, 1.0b2, 1.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.0.8, 1.1, 1.1.1, 1.1.2, 1.1.3, 1.2b1, 1.2b2, 1.2b3, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.3b1, 1.3b2, 1.3b3, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.4a1, 1.4b1, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.5a1, 1.5a2, 1.5b1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.6b1, 1.6b2, 1.6b3, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5, 1.6.6, 1.6.7, 1.7.0b1, 1.7.0b2, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6, 1.7.7, 1.7.8, 1.7.9, 1.8.0b1, 1.8.0, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.8.5, 1.8.6, 2.0.0b1, 2.0.0b2, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 3.0.0b1, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.2.1, 3.3.0, 3.3.1, 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.5.0, 3.5.1, 3.5.2, 3.5.3, 3.5.4, 4.0.0b1, 4.0.0b2, 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.1.0, 4.1.1, 4.1.2, 4.2.0, 4.3.0, 4.3.1, 4.3.2, 4.4.0, 4.5.0, 5.0.0b1, 5.0.0, 5.0.1, 5.0.2, 5.1.0, 5.1.1, 5.2.0, 5.2.0.post0, 5.2.1, 5.2.2, 5.2.3, 5.3.0) ERROR: No matching distribution found for Sphinx==6.1.3; extra == ""docs"" ``` I'm going to drop Python 3.7.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/datasette/issues/1153#issuecomment-1627451646,https://api.github.com/repos/simonw/datasette/issues/1153,1627451646,IC_kwDOBm6k_c5hAPD-,9599,simonw,2023-07-08T18:21:24Z,2023-07-08T18:21:24Z,OWNER,"This one was tricky: I wanted complete control over the YAML example here, so I could ensure it used multi-line strings correctly. I ended up changing my cog helper function to this: ```python import json import textwrap from yaml import safe_dump from ruamel.yaml import round_trip_load def metadata_example(cog, data=None, yaml=None): assert data or yaml, ""Must provide data= or yaml="" assert not (data and yaml), ""Cannot use data= and yaml="" output_yaml = None if yaml: # dedent it first yaml = textwrap.dedent(yaml).strip() # round_trip_load to preserve key order: data = round_trip_load(yaml) output_yaml = yaml else: output_yaml = safe_dump(data, sort_keys=False) cog.out(""\n.. tab:: YAML\n\n"") cog.out("" .. code-block:: yaml\n\n"") cog.out(textwrap.indent(output_yaml, "" "")) cog.out(""\n\n.. tab:: JSON\n\n"") cog.out("" .. code-block:: json\n\n"") cog.out(textwrap.indent(json.dumps(data, indent=2), "" "")) cog.out(""\n"") ``` This allows me to call it ith YAML in some places: ``` .. [[[cog metadata_example(cog, yaml="""""" databases: fixtures: queries: neighborhood_search: fragment: fragment-goes-here hide_sql: true sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood; """""") .. ]]] ``` I had to introduce https://pypi.org/project/ruamel.yaml/ as a dependency here in order to load YAML from disk while maintaining key order. I'm still using `safe_dump(data, sort_keys=False)` from PyYAML as I couldn't get the result I wanted for outputting YAML from an input of JSON using PyYAML.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/datasette/issues/1153#issuecomment-1627450852,https://api.github.com/repos/simonw/datasette/issues/1153,1627450852,IC_kwDOBm6k_c5hAO3k,9599,simonw,2023-07-08T18:17:35Z,2023-07-08T18:17:35Z,OWNER,"I figured out a workaround: ```python extensions = [ ""sphinx.ext.extlinks"", ""sphinx.ext.autodoc"", ""sphinx_copybutton"", ] if not os.environ.get(""DISABLE_SPHINX_INLINE_TABS""): extensions += [""sphinx_inline_tabs""] ``` That way I can run `sphinx-build -b xml . _build` successfully if I set that environment variable. I get some noisy warnings, but it runs OK. And the resulting `docs.db` file has rows like this, which I think are fine: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/datasette/issues/1153#issuecomment-1627448542,https://api.github.com/repos/simonw/datasette/issues/1153,1627448542,IC_kwDOBm6k_c5hAOTe,9599,simonw,2023-07-08T18:05:44Z,2023-07-08T18:05:44Z,OWNER,"Running with `-P` opens a debugger when it hits the error: ```bash sphinx-build -P -b xml . _build ``` ``` (Pdb) list 2023 2024 Raise an exception unless overridden. 2025 """""" 2026 if (self.document.settings.strict_visitor 2027 or node.__class__.__name__ not in self.optional): 2028 -> raise NotImplementedError( 2029 '%s visiting unknown node type: %s' 2030 % (self.__class__, node.__class__.__name__)) 2031 2032 def unknown_departure(self, node): 2033 """""" (Pdb) self.optional ('meta',) (Pdb) node.__class__.__name__ 'TabContainer' (Pdb) self.document.settings.strict_visitor (Pdb) type(self.document.settings.strict_visitor) ``` So if I can get `TabContainer` into that `self.optional` list I'll have fixed this problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/datasette/issues/1153#issuecomment-1627448180,https://api.github.com/repos/simonw/datasette/issues/1153,1627448180,IC_kwDOBm6k_c5hAON0,9599,simonw,2023-07-08T18:03:31Z,2023-07-08T18:03:31Z,OWNER,"Relevant code: https://github.com/docutils/docutils/blob/3b53ded52bc439d8068b6ecb20ea0a761247e479/docutils/docutils/nodes.py#L2021-L2031 ```python def unknown_visit(self, node): """""" Called when entering unknown `Node` types. Raise an exception unless overridden. """""" if (self.document.settings.strict_visitor or node.__class__.__name__ not in self.optional): raise NotImplementedError( '%s visiting unknown node type: %s' % (self.__class__, node.__class__.__name__)) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/datasette/issues/1153#issuecomment-1627447750,https://api.github.com/repos/simonw/datasette/issues/1153,1627447750,IC_kwDOBm6k_c5hAOHG,9599,simonw,2023-07-08T18:00:56Z,2023-07-08T18:00:56Z,OWNER,"Actually no it's in `sphinx-build`: ``` % sphinx-build -b xml . _build Running Sphinx v6.1.3 building [mo]: targets for 0 po files that are out of date writing output... building [xml]: targets for 28 source files that are out of date updating environment: [new config] 28 added, 0 changed, 0 removed reading sources... [100%] writing_plugins looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [ 3%] authentication Exception occurred: File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/docutils/nodes.py"", line 2028, in unknown_visit raise NotImplementedError( NotImplementedError: visiting unknown node type: TabContainer The full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1wkxmkji.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. A bug report can be filed in the tracker at . Thanks! ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/datasette/issues/1153#issuecomment-1627447478,https://api.github.com/repos/simonw/datasette/issues/1153,1627447478,IC_kwDOBm6k_c5hAOC2,9599,simonw,2023-07-08T17:59:25Z,2023-07-08T17:59:25Z,OWNER,"Hit a problem: ``` Exception occurred: File ""/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/docutils/nodes.py"", line 2028, in unknown_visit raise NotImplementedError( NotImplementedError: visiting unknown node type: TabContainer The full traceback has been saved in /tmp/sphinx-err-tfujyw1h.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. A bug report can be filed in the tracker at . Thanks! ``` That's happening here: https://github.com/simonw/datasette/blob/0183e1a72d4d93b1d9a9363f4d47fcc0b5d5849c/.github/workflows/deploy-latest.yml#L42-L48 My https://github.com/simonw/sphinx-to-sqlite tool can't handle the new `TabContainer` elements introduced by `sphinx-inline-tabs`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/datasette/issues/1153#issuecomment-1627396658,https://api.github.com/repos/simonw/datasette/issues/1153,1627396658,IC_kwDOBm6k_c5hABoy,9599,simonw,2023-07-08T16:40:07Z,2023-07-08T16:40:07Z,OWNER,"https://docs.datasette.io/en/latest/metadata.html ![inline-tabs](https://github.com/simonw/datasette/assets/9599/975bdff5-74ac-451e-92c3-a7dd05d4b862) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/datasette/issues/1153#issuecomment-1627396450,https://api.github.com/repos/simonw/datasette/issues/1153,1627396450,IC_kwDOBm6k_c5hABli,9599,simonw,2023-07-08T16:38:58Z,2023-07-08T16:38:58Z,OWNER,"I'm using `cog` and this utility function to generate the YAML/JSON tabs: https://github.com/simonw/datasette/blob/3b336d8071fb5707bd006de1d614f701d20246a3/docs/metadata_doc.py#L1-L13 Example usage: https://github.com/simonw/datasette/blob/3b336d8071fb5707bd006de1d614f701d20246a3/docs/metadata.rst?plain=1#L17-L53","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/datasette/issues/1153#issuecomment-1627395947,https://api.github.com/repos/simonw/datasette/issues/1153,1627395947,IC_kwDOBm6k_c5hABdr,9599,simonw,2023-07-08T16:35:45Z,2023-07-08T16:35:45Z,OWNER,I was inspired to finally address this after seeing `sphinx-inline-tabs` at work in https://webcolors.readthedocs.io/en/latest/install.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON", https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1618380888,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1618380888,IC_kwDOCGYnMM5gdohY,9599,simonw,2023-07-03T14:09:11Z,2023-07-03T14:09:31Z,OWNER,"For the CLI: ```bash sqlite-utils rename-table data.db old_table_name new_table_name ``` For the Python code, should it go on Table or on Database? ```python db[""foo""].rename_table(""bar"") db.rename_table(""foo"", ""bar"") ``` I think I like the second better, it's slightly more clear. Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming utilities, https://github.com/simonw/sqlite-utils/issues/563#issuecomment-1617395444,https://api.github.com/repos/simonw/sqlite-utils/issues/563,1617395444,IC_kwDOCGYnMM5gZ370,9599,simonw,2023-07-03T05:44:43Z,2023-07-03T05:44:43Z,OWNER,Documentation at the bottom of this section: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-csv-or-tsv-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1785360409,`--empty-null` option when importing CSV, https://github.com/simonw/sqlite-utils/issues/562#issuecomment-1616782404,https://api.github.com/repos/simonw/sqlite-utils/issues/562,1616782404,IC_kwDOCGYnMM5gXiRE,9599,simonw,2023-07-02T19:24:14Z,2023-07-02T19:26:39Z,OWNER,"[Dataclasses](https://docs.python.org/3/library/dataclasses.html) were added in Python 3.7 and `sqlite-utils` was originally written for Python 3.6 - but both 3.6 and 3.7 are EOL now. The thing that makes Dataclasses particularly interesting is the potential to use type annotations with them to help specify the types of the related SQLite columns. Example for https://datasette.io/content/users ```sql CREATE TABLE [users] ( [login] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [avatar_url] TEXT, [gravatar_id] TEXT, [html_url] TEXT, [type] TEXT, [site_admin] INTEGER, [name] TEXT ); ``` And the dataclass: ```python from dataclasses import dataclass @dataclass class User: id: int login: str node_id: str avatar_url: str gravatar_id: str html_url: str type: str site_admin: int name: str ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1784794489,Explore the intersection between sqlite-utils and dataclasses, https://github.com/simonw/datasette/issues/2093#issuecomment-1614652001,https://api.github.com/repos/simonw/datasette/issues/2093,1614652001,IC_kwDOBm6k_c5gPaJh,9599,simonw,2023-06-30T13:27:13Z,2023-06-30T13:27:13Z,OWNER,"I agree, settings in the DB doesn't make sense but metadata does. On the JSON v YAML v TOML issue I just spotted Caddy has a concept of config adapters which they use to resolve exactly that problem: https://caddyserver.com/docs/config-adapters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.toml` File", https://github.com/simonw/datasette/issues/2093#issuecomment-1613889979,https://api.github.com/repos/simonw/datasette/issues/2093,1613889979,IC_kwDOBm6k_c5gMgG7,9599,simonw,2023-06-29T22:44:08Z,2023-06-30T13:25:39Z,OWNER,"I do like also being able to set options using command line options though - for things like SQL time limits I'd much rather be able to throw on `--setting sql_time_limit_ms 10000` than have to save a config file to disk. So I'd want to support both. Which maybe means also having a way to set plugin options with CLI options. `datasette publish` kind of has that ability already: ``` datasette publish heroku my_database.db \ --name my-heroku-app-demo \ --install=datasette-auth-github \ --plugin-secret datasette-auth-github client_id your_client_id \ --plugin-secret datasette-auth-github client_secret your_client_secret ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.toml` File", https://github.com/simonw/datasette/issues/2093#issuecomment-1613887492,https://api.github.com/repos/simonw/datasette/issues/2093,1613887492,IC_kwDOBm6k_c5gMfgE,9599,simonw,2023-06-29T22:40:25Z,2023-06-29T22:40:25Z,OWNER,"I'm strongly in favour of combining settings, configuration and plugin configuration. I'm not keen on mixing in metadata as well - that feels like a different concept to me, and I'm unhappy with how that's already had things like plugin settings leak into it. I'm not yet sold on TOML - I actually find it less intuitive than YAML, surprisingly. They all have their warts I guess. Datasette already has the ability to consume JSON or YAML for metadata - maybe it could grow TOML support too? That way users could have a `datasette.json` or `datasette.yaml` or `datasette.toml` file depending on their preference. In terms of metadata: since that's means to be driven by a plugin hook anyway, maybe one of the potential sources of metadata is a `metadata` nested object in that `datasette.*` configuration file. Or you can have it in a separate `metadata.json` or bundled into the SQLite database or some other plugin-driven mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.toml` File", https://github.com/simonw/datasette/issues/2092#issuecomment-1613381990,https://api.github.com/repos/simonw/datasette/issues/2092,1613381990,IC_kwDOBm6k_c5gKkFm,9599,simonw,2023-06-29T15:23:04Z,2023-06-29T15:26:28Z,OWNER,"Felt lazy: ```bash symbex test_homepage -f tests/test_api.py | \ llm -m 4 --system 'Change all of the == some integer tests in this code to isinstance(thing, int) instead' ``` Output: ```python # File: tests/test_api.py Line: 26 @pytest.mark.asyncio async def test_homepage(ds_client): response = await ds_client.get(""/.json"") assert response.status_code == 200 assert ""application/json; charset=utf-8"" == response.headers[""content-type""] data = response.json() assert isinstance(data.keys(), int) d = data[""fixtures""] assert d[""name""] == ""fixtures"" assert isinstance(d[""tables_count""], int) assert isinstance(len(d[""tables_and_views_truncated""]), int) assert d[""tables_and_views_more""] is True # 4 hidden FTS tables + no_primary_key (hidden in metadata) assert isinstance(d[""hidden_tables_count""], int) # 201 in no_primary_key, plus 6 in other hidden tables: assert isinstance(d[""hidden_table_rows_sum""], int), data assert isinstance(d[""views_count""], int) ``` I'll use most of that and delete the obsoleted comments.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781047747,test_homepage intermittent failure, https://github.com/simonw/datasette/issues/2092#issuecomment-1613375407,https://api.github.com/repos/simonw/datasette/issues/2092,1613375407,IC_kwDOBm6k_c5gKiev,9599,simonw,2023-06-29T15:20:52Z,2023-06-29T15:21:05Z,OWNER,I'm going to remove this assertion entirely. The homepage JSON needs a refactor anyway.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781047747,test_homepage intermittent failure, https://github.com/simonw/datasette/issues/2091#issuecomment-1613369355,https://api.github.com/repos/simonw/datasette/issues/2091,1613369355,IC_kwDOBm6k_c5gKhAL,9599,simonw,2023-06-29T15:18:34Z,2023-06-29T15:18:34Z,OWNER,Posted on the Glitch feedback forum about this here: https://support.glitch.com/t/upgrade-python-version-from-3-7-which-is-now-eol-to-something-more-recent/63011,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781022369,Drop support for Python 3.7, https://github.com/simonw/datasette/issues/2091#issuecomment-1613360413,https://api.github.com/repos/simonw/datasette/issues/2091,1613360413,IC_kwDOBm6k_c5gKe0d,9599,simonw,2023-06-29T15:13:04Z,2023-06-29T15:13:04Z,OWNER,"One problem: https://glitch.com/ still provides 3.7: ``` $ python3 --version Python 3.7.10 ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781022369,Drop support for Python 3.7, https://github.com/simonw/datasette/issues/2090#issuecomment-1613346412,https://api.github.com/repos/simonw/datasette/issues/2090,1613346412,IC_kwDOBm6k_c5gKbZs,9599,simonw,2023-06-29T15:05:04Z,2023-06-29T15:05:04Z,OWNER,"Decided to fix just those ""Ambiguous variable name"" ones: ```bash ruff check . | grep E741 ``` Then iterated through and fixed them all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781005740,Adopt ruff for linting, https://github.com/simonw/datasette/issues/2090#issuecomment-1613339404,https://api.github.com/repos/simonw/datasette/issues/2090,1613339404,IC_kwDOBm6k_c5gKZsM,9599,simonw,2023-06-29T15:01:01Z,2023-06-29T15:01:20Z,OWNER,"I tried it just now and got some interesting results. I dropped in a `ruff.toml` file: ```toml line-length = 160 ``` Because the default line length limit of 88 was causing a lot of noisy errors. Then run: ```bash pip install ruff ruff check . ``` Plenty of warnings about unused imports - running `ruff check . --fix` fixed those automatically, but I think I still need to manually review them as some might be imports which are deliberate and should be in `__all__` to ensure they are visible from that module as well. Some lines in tests are longer than even 160 chars, e.g.: https://github.com/simonw/datasette/blob/99ba05118891db9dc30f1dca22ad6709775560de/tests/test_html.py#L673-L681 These can have ` # noqa: E501` added to the end of those lines to skip the check for them. That got it down to: ``` % ruff check . datasette/views/table.py:23:5: F811 Redefinition of unused `format_bytes` from line 19 run_tests.py:2:5: E401 Multiple imports on one line tests/test_api.py:591:40: F811 Redefinition of unused `app_client_no_files` from line 7 tests/test_api.py:629:35: F811 Redefinition of unused `app_client_no_files` from line 7 tests/test_api.py:635:54: F811 Redefinition of unused `app_client_with_dot` from line 8 tests/test_api.py:661:25: F811 Redefinition of unused `app_client_shorter_time_limit` from line 9 tests/test_api.py:759:25: F811 Redefinition of unused `app_client_two_attached_databases_one_immutable` from line 10 tests/test_api.py:892:28: F811 Redefinition of unused `app_client_larger_cache_size` from line 11 tests/test_api.py:928:5: F811 Redefinition of unused `app_client_with_cors` from line 12 tests/test_api.py:929:5: F811 Redefinition of unused `app_client_two_attached_databases_one_immutable` from line 10 tests/test_api.py:969:38: F811 Redefinition of unused `app_client_two_attached_databases` from line 13 tests/test_api.py:976:39: F811 Redefinition of unused `app_client_conflicting_database_names` from line 14 tests/test_api.py:987:38: F811 Redefinition of unused `app_client_immutable_and_inspect_file` from line 15 tests/test_api.py:1002:24: F811 Redefinition of unused `app_client` from line 6 tests/test_csv.py:67:33: F811 Redefinition of unused `app_client_with_cors` from line 6 tests/test_csv.py:157:21: F811 Redefinition of unused `app_client_csv_max_mb_one` from line 5 tests/test_csv.py:198:20: F811 Redefinition of unused `app_client_with_trace` from line 7 tests/test_csv.py:209:53: F811 Redefinition of unused `app_client_with_trace` from line 7 tests/test_csv.py:215:53: F811 Redefinition of unused `app_client_with_trace` from line 7 tests/test_filters.py:102:11: F811 Redefinition of unused `test_through_filters_from_request` from line 81 tests/test_html.py:19:19: F811 Redefinition of unused `app_client_two_attached_databases` from line 7 tests/test_html.py:175:25: F811 Redefinition of unused `app_client_shorter_time_limit` from line 6 tests/test_html.py:469:51: F811 Redefinition of unused `app_client` from line 4 tests/test_html.py:797:26: F811 Redefinition of unused `app_client_base_url_prefix` from line 5 tests/test_html.py:840:44: F811 Redefinition of unused `app_client_base_url_prefix` from line 5 tests/test_html.py:850:51: F811 Redefinition of unused `app_client_base_url_prefix` from line 5 tests/test_pagination.py:50:43: F821 Undefined name `parse_next` tests/test_pagination.py:82:7: F811 Redefinition of unused `KeysetPaginator` from line 36 tests/test_plugins.py:115:15: E741 Ambiguous variable name: `l` tests/test_plugins.py:482:161: E501 Line too long (170 > 160 characters) tests/test_plugins.py:543:29: E741 Ambiguous variable name: `l` tests/test_plugins.py:563:161: E501 Line too long (170 > 160 characters) tests/test_plugins.py:940:62: E741 Ambiguous variable name: `l` tests/test_table_api.py:739:5: F811 Redefinition of unused `app_client_returned_rows_matches_page_size` from line 6 tests/test_table_api.py:1066:45: F811 Redefinition of unused `app_client_with_trace` from line 5 tests/test_table_html.py:484:29: E741 Ambiguous variable name: `l` tests/test_table_html.py:524:29: E741 Ambiguous variable name: `l` tests/test_table_html.py:675:161: E501 Line too long (165 > 160 characters) tests/test_table_html.py:897:161: E501 Line too long (164 > 160 characters) tests/test_table_html.py:902:161: E501 Line too long (164 > 160 characters) tests/test_utils.py:141:161: E501 Line too long (176 > 160 characters) Found 41 errors. ``` Those ""Redefinition of unused `app_client_two_attached_databases`"" lines are caused because of the fixtures pattern I'm using here: https://github.com/simonw/datasette/blob/99ba05118891db9dc30f1dca22ad6709775560de/tests/test_html.py#L3-L20 I could fix that by getting rid of `fixtures.py` and moving those into `conftest.py`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781005740,Adopt ruff for linting, https://github.com/simonw/datasette/issues/2089#issuecomment-1613316722,https://api.github.com/repos/simonw/datasette/issues/2089,1613316722,IC_kwDOBm6k_c5gKUJy,9599,simonw,2023-06-29T14:48:10Z,2023-06-29T14:48:10Z,OWNER,Spell check is passing now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure, https://github.com/simonw/datasette/issues/2089#issuecomment-1613315851,https://api.github.com/repos/simonw/datasette/issues/2089,1613315851,IC_kwDOBm6k_c5gKT8L,9599,simonw,2023-06-29T14:47:38Z,2023-06-29T14:47:38Z,OWNER,"Confirmed, this was a 2.2.5 change: https://github.com/codespell-project/codespell/releases/tag/v2.2.5 > - Add displaing->displaying by [@peternewman](https://github.com/peternewman) in [#2808](https://github.com/codespell-project/codespell/pull/2808)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure, https://github.com/simonw/datasette/issues/2089#issuecomment-1613307716,https://api.github.com/repos/simonw/datasette/issues/2089,1613307716,IC_kwDOBm6k_c5gKR9E,9599,simonw,2023-06-29T14:42:23Z,2023-06-29T14:42:23Z,OWNER,"Yes, upgrading locally got me the correct version and the test failure: ``` % pip install -U codespell Requirement already satisfied: codespell in /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages (2.2.2) Collecting codespell Downloading codespell-2.2.5-py3-none-any.whl (242 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 242.7/242.7 kB 4.9 MB/s eta 0:00:00 Installing collected packages: codespell Attempting uninstall: codespell Found existing installation: codespell 2.2.2 Uninstalling codespell-2.2.2: Successfully uninstalled codespell-2.2.2 Successfully installed codespell-2.2.5 % codespell docs/metadata.rst docs/metadata.rst:192: displaing ==> displaying ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure, https://github.com/simonw/datasette/issues/2089#issuecomment-1613306787,https://api.github.com/repos/simonw/datasette/issues/2089,1613306787,IC_kwDOBm6k_c5gKRuj,9599,simonw,2023-06-29T14:41:47Z,2023-06-29T14:41:47Z,OWNER,"Looks like in CI it's running 2.2.5: ``` Collecting codespell (from datasette==1.0a2) Downloading codespell-2.2.5-py3-none-any.whl (242 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 242.7/242.7 kB 31.1 MB/s eta 0:00:00 ``` But on my laptop it's 2.2.2: ``` % codespell --version 2.2.2 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure, https://github.com/simonw/datasette/issues/2089#issuecomment-1613305070,https://api.github.com/repos/simonw/datasette/issues/2089,1613305070,IC_kwDOBm6k_c5gKRTu,9599,simonw,2023-06-29T14:40:44Z,2023-06-29T14:40:44Z,OWNER,"I'm not sure why I can't duplicate this failure in my local development environment: ``` % codespell docs/metadata.rst ``` It finds no errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure, https://github.com/simonw/datasette/pull/2077#issuecomment-1613290899,https://api.github.com/repos/simonw/datasette/issues/2077,1613290899,IC_kwDOBm6k_c5gKN2T,9599,simonw,2023-06-29T14:32:16Z,2023-06-29T14:32:16Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1719759468,Bump furo from 2023.3.27 to 2023.5.20, https://github.com/simonw/datasette/issues/1510#issuecomment-1610512875,https://api.github.com/repos/simonw/datasette/issues/1510,1610512875,IC_kwDOBm6k_c5f_nnr,9599,simonw,2023-06-28T02:02:10Z,2023-06-28T02:05:21Z,OWNER,"I prototyped an approach to this using dataclasses and a `cog` mechanism for turning those into rendered tables in Sphinx. Here's what that prototype looks like: See https://github.com/simonw/datasette/commit/68223784167fdec4e7ebfca56002a6548ba7b423 for how it works. Here's the class that documented: https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/datasette/context.py#L54-L68 And the code that generates the rST: https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/datasette/context.py#L19-L45 And the bit that cog executes: https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/docs/template_context.rst?plain=1#L9-L12","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054244712,Datasette 1.0 documented template context (maybe via API docs), https://github.com/simonw/sqlite-utils/issues/561#issuecomment-1610040517,https://api.github.com/repos/simonw/sqlite-utils/issues/561,1610040517,IC_kwDOCGYnMM5f90TF,9599,simonw,2023-06-27T18:44:31Z,2023-06-27T18:44:38Z,OWNER,"Got this working: ```bash sqlite-utils insert /tmp/playground.db Playground_Submission_Data \ ~/Downloads/Playground_Submission_Data.csv --csv --stop-after 2000 ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1777548699,`--stop-after` option for `insert` and `upsert` commands, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606415188,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1606415188,IC_kwDOCGYnMM5fv_NU,9599,simonw,2023-06-26T01:46:47Z,2023-06-26T01:47:01Z,OWNER,"I just tested this in a brand new virtual environment using the macOS Python 3: ```bash pipenv shell --python /Applications/Xcode.app/Contents/Developer/usr/bin/python3 ``` Then in that virtual environment I ran: ```bash pip install sqlite-utils # Confirm the right one is on the path: which sqlite-utils curl ""https://data.nasa.gov/resource/y77d-th95.json"" | \ sqlite-utils insert meteorites.db meteorites - --pk=id sqlite-utils extract meteorites.db meteorites recclass ``` This threw the same error reported above. Then I did this: ```bash rm meteorites.db pip install sqlean.py curl ""https://data.nasa.gov/resource/y77d-th95.json"" | \ sqlite-utils insert meteorites.db meteorites - --pk=id sqlite-utils extract meteorites.db meteorites recclass ``` And that second time it worked correctly.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606411508,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1606411508,IC_kwDOCGYnMM5fv-T0,9599,simonw,2023-06-26T01:42:10Z,2023-06-26T01:42:22Z,OWNER,https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-33 - upgrading to `sqlite-utils>=3.33` and then installing both `sqlean.py` and `sqlite-dump` in the same virtual environment as `sqlite-utils` should fix this issue.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606315321,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606315321,IC_kwDOCGYnMM5fvm05,9599,simonw,2023-06-25T23:18:33Z,2023-06-25T23:18:33Z,OWNER,Documentation preview: https://sqlite-utils--560.org.readthedocs.build/en/560/installation.html#alternatives-to-sqlite3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606310630,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606310630,IC_kwDOCGYnMM5fvlrm,9599,simonw,2023-06-25T23:06:07Z,2023-06-25T23:06:07Z,OWNER,"Filed an issue about the above with `pysqlite3` (which `sqlean.py` is based on) here: - https://github.com/coleifer/pysqlite3/issues/58","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606297356,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606297356,IC_kwDOCGYnMM5fvicM,9599,simonw,2023-06-25T22:42:41Z,2023-06-25T22:42:41Z,OWNER,"Yes that does seem to do the trick: ```pycon >>> import sqlean >>> db = sqlean.connect(""/tmp/4.db"") >>> db.execute('PRAGMA journal_mode;').fetchall() [('delete',)] >>> db.isolation_level '' >>> db.execute('PRAGMA journal_mode=wal;') Traceback (most recent call last): File """", line 1, in sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction >>> db.isolation_level = None >>> db.isolation_level >>> db.execute('PRAGMA journal_mode=wal;') ``` Weird how `isolation_level` of empty string causes the error, but setting that to `None` fixes the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606294627,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606294627,IC_kwDOCGYnMM5fvhxj,9599,simonw,2023-06-25T22:40:10Z,2023-06-25T22:40:10Z,OWNER,I suspect this has something to do with `autocommit` mode in `sqlite3` - which I may be able to turn off by setting `con.isolation_level = None`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606293382,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606293382,IC_kwDOCGYnMM5fvheG,9599,simonw,2023-06-25T22:34:47Z,2023-06-25T22:34:47Z,OWNER,"```pycon >>> import sqlite3 >>> db = sqlite3.connect(""/tmp/1.db"") >>> db.execute('PRAGMA journal_mode=wal;') >>> import sqlean >>> db2 = sqlean.connect(""/tmp/2.db"") >>> db2.execute('PRAGMA journal_mode=wal;') Traceback (most recent call last): File """", line 1, in sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606290917,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606290917,IC_kwDOCGYnMM5fvg3l,9599,simonw,2023-06-25T22:32:28Z,2023-06-25T22:32:28Z,OWNER,"I've fixed most of the test failures, but I still need to fix this one: > cannot change into wal mode from within a transaction","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606273005,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606273005,IC_kwDOCGYnMM5fvcft,9599,simonw,2023-06-25T21:47:47Z,2023-06-25T21:47:47Z,OWNER,I can use https://github.com/simonw/sqlite-dump as an optional dependency to handle the missing `.iterdump()` method.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270887,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606270887,IC_kwDOCGYnMM5fvb-n,9599,simonw,2023-06-25T21:37:12Z,2023-06-26T08:21:00Z,OWNER,"On my own laptop I got a crash running the tests - details here: - https://github.com/nalgeon/sqlean.py/issues/3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270055,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606270055,IC_kwDOCGYnMM5fvbxn,9599,simonw,2023-06-25T21:31:56Z,2023-06-25T21:31:56Z,OWNER,"Lots of failures now that I'm trying to run the tests against `sqlean.py` on macOS and Python 3.10: https://github.com/simonw/sqlite-utils/actions/runs/5371800108/jobs/9744802953 A bunch of these, because `pysqlite3` chooses not to implement `.iterdump()`: ``` @pytest.fixture def db_to_analyze_path(db_to_analyze, tmpdir): path = str(tmpdir / ""test.db"") db = sqlite3.connect(path) > db.executescript(""\n"".join(db_to_analyze.conn.iterdump())) E AttributeError: 'sqlean.dbapi2.Connection' object has no attribute 'iterdump' ``` Also some of these: ``` def test_analyze_whole_database(db): assert set(db.table_names()) == {""one_index"", ""two_indexes""} db.analyze() > assert set(db.table_names()) == {""one_index"", ""two_indexes"", ""sqlite_stat1""} E AssertionError: assert {'one_index',...'two_indexes'} == {'one_index',...'two_indexes'} E Extra items in the left set: E 'sqlite_stat4' E Full diff: E - {'two_indexes', 'sqlite_stat1', 'one_index'} E + {'two_indexes', 'sqlite_stat1', 'sqlite_stat4', 'one_index'} E ? ++++++++++++++++ ``` Apparently `sqlean.py` adds a `sqlite_stat4` table that the tests are not expecting. Plus some errors that look like this: ``` def test_enable_wal(): runner = CliRunner() dbs = [""test.db"", ""test2.db""] with runner.isolated_filesystem(): for dbname in dbs: db = Database(dbname) db[""t""].create({""pk"": int}, pk=""pk"") assert db.journal_mode == ""delete"" result = runner.invoke(cli.cli, [""enable-wal""] + dbs) > assert 0 == result.exit_code E AssertionError: assert 0 == 1 E + where 1 = .exit_code ``` Test summary: ``` ============ 13 failed, 909 passed, 16 skipped, 2 errors in 19.29s ============= ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1604379952,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1604379952,IC_kwDOCGYnMM5foOUw,9599,simonw,2023-06-23T14:39:55Z,2023-06-23T15:39:32Z,OWNER,"Ideally a workaround for this right now would be to install `pysqlite3` in the same virtual environment: sqlite-utils install pysqlite3-binary But `pysqlite3-binary` doesn't yet ship a wheel for macOS so this probably won't work for most people. The ""easiest"" fix at the moment is to use Python from Homebrew - so `brew install sqlite-utils` for example won't suffer from this problem. Not a great solution for people who aren't using Homebrew though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/datasette/issues/260#issuecomment-1600778057,https://api.github.com/repos/simonw/datasette/issues/260,1600778057,IC_kwDOBm6k_c5fae9J,9599,simonw,2023-06-21T12:51:22Z,2023-06-21T12:51:22Z,OWNER,"Another example of confusion from this today: https://discord.com/channels/823971286308356157/823971286941302908/1121042411238457374 See also https://gist.github.com/BinomeDeNewton/651ac8b50dd5420f8e54d1682eee5fed?permalink_comment_id=4605982#gistcomment-4605982","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323223872,Validate metadata.json on startup, https://github.com/simonw/datasette/pull/2053#issuecomment-1565058994,https://api.github.com/repos/simonw/datasette/issues/2053,1565058994,IC_kwDOBm6k_c5dSOey,9599,simonw,2023-05-26T23:13:02Z,2023-05-26T23:13:02Z,OWNER,"I should have an extra called `extra_html_context` which bundles together all of the weird extra stuff needed by the HTML template, and is then passed as the root context when the template is rendered (with the other stuff from extras patched into it).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1563793781,https://api.github.com/repos/simonw/datasette/issues/2053,1563793781,IC_kwDOBm6k_c5dNZl1,9599,simonw,2023-05-26T04:27:55Z,2023-05-26T04:27:55Z,OWNER,"I should split out a `canned_query.html` template too, as something that extends the `query.html` template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1563667574,https://api.github.com/repos/simonw/datasette/issues/2053,1563667574,IC_kwDOBm6k_c5dM6x2,9599,simonw,2023-05-26T00:40:22Z,2023-05-26T00:40:22Z,OWNER,"Or maybe... - `BaseQueryView(View)` - knows how to render the results of a SQL query - `QueryView(BaseQueryView)` - renders from `?sql=` - `CannedQueryView(BaseQueryView)` - renders for a named canned query And then later perhaps: - `RowQueryView(BaseQueryView)` - renders the `select * from t where pk = ?` - `TableQueryView(BaseQueryView)` - replaces the super complex existing `TableView`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1563663925,https://api.github.com/repos/simonw/datasette/issues/2053,1563663925,IC_kwDOBm6k_c5dM541,9599,simonw,2023-05-26T00:32:47Z,2023-05-26T00:35:47Z,OWNER,"I'm going to entirely split canned queries off from `?sql=` queries - they share a bunch of code right now which is just making everything much harder to follow. I'll refactor their shared bits into functions that they both call. Or _maybe_ I'll try having `CannedQueryView` as a subclass of `QueryView`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1563663616,https://api.github.com/repos/simonw/datasette/issues/2053,1563663616,IC_kwDOBm6k_c5dM50A,9599,simonw,2023-05-26T00:32:08Z,2023-05-26T00:32:08Z,OWNER,"Now that I have the new `View` subclass from #2078 I want to use it to simplify this code. Challenge: there are several things to consider here: - The `/db` page without `?sql=` displays a list of tables in that database - With `?sql=` it shows the query results for that query (or an error) - If it's a `/db/name-of-canned-query` it works a bit like the query page, but executes a canned query instead of the `?sql=` query - POST `/db/name-of-canned-query` is support for writable canned queries","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2080#issuecomment-1563650990,https://api.github.com/repos/simonw/datasette/issues/2080,1563650990,IC_kwDOBm6k_c5dM2uu,9599,simonw,2023-05-26T00:08:59Z,2023-05-26T00:08:59Z,OWNER,"I'm not going to document this yet, I want to let it bake for a bit longer first.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726603778,New View base class, https://github.com/simonw/datasette/pull/2080#issuecomment-1563626231,https://api.github.com/repos/simonw/datasette/issues/2080,1563626231,IC_kwDOBm6k_c5dMwr3,9599,simonw,2023-05-25T23:25:17Z,2023-05-25T23:25:17Z,OWNER,I'm going to try using this for the `/-/patterns` page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726603778,New View base class, https://github.com/simonw/datasette/issues/2078#issuecomment-1563625093,https://api.github.com/repos/simonw/datasette/issues/2078,1563625093,IC_kwDOBm6k_c5dMwaF,9599,simonw,2023-05-25T23:23:15Z,2023-05-25T23:23:15Z,OWNER,"Rest of the work on this will happen in the PR: - #2080","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2079#issuecomment-1563607291,https://api.github.com/repos/simonw/datasette/issues/2079,1563607291,IC_kwDOBm6k_c5dMsD7,9599,simonw,2023-05-25T22:56:28Z,2023-05-25T22:56:28Z,OWNER,Wrote this up as a TIL: https://til.simonwillison.net/http/testing-cors-max-age,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563597589,https://api.github.com/repos/simonw/datasette/issues/2079,1563597589,IC_kwDOBm6k_c5dMpsV,9599,simonw,2023-05-25T22:42:07Z,2023-05-25T22:42:07Z,OWNER,"Mystery solved as to why I wasn't seeing this work: I had ""Disable Cache"" checked! I ran this experiment after un-checking that box: ```javascript fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ test: 'test' }) }); // And run it again fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ test: 'test' }) }); // Now try a thing that doesn't serve that max-age header yet: fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ test: 'test' }) }); // And a second time but within 5s fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ test: 'test' }) }); // Third time after waiting longer than 5s fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ test: 'test' }) }); // Try that original one again - still within the 1hr cache time fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ test: 'test' }) }); ``` The results show that the cache of 1hr was being obeyed for `latest.datasette.io` while the `latest-with-plugins.datasette.io` default cache of 5s was being obeyed too. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563588199,https://api.github.com/repos/simonw/datasette/issues/2079,1563588199,IC_kwDOBm6k_c5dMnZn,9599,simonw,2023-05-25T22:29:47Z,2023-05-25T22:30:12Z,OWNER,"https://fetch.spec.whatwg.org/#http-access-control-max-age says: > Indicates the number of seconds (5 by default) the information provided by the [Access-Control-Allow-Methods](https://fetch.spec.whatwg.org/#http-access-control-allow-methods) and [Access-Control-Allow-Headers](https://fetch.spec.whatwg.org/#http-access-control-allow-headers) [headers](https://fetch.spec.whatwg.org/#concept-header) can be cached. So there was already a 5s cache anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563587230,https://api.github.com/repos/simonw/datasette/issues/2079,1563587230,IC_kwDOBm6k_c5dMnKe,9599,simonw,2023-05-25T22:28:20Z,2023-05-25T22:28:20Z,OWNER,"Weird... after the deploy went out: But the request did indeed get the new header: So I'm not sure why it's making multiple `POST` requests like that. Maybe it's because the attempted `POST` failed with a 404?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563565407,https://api.github.com/repos/simonw/datasette/issues/2079,1563565407,IC_kwDOBm6k_c5dMh1f,9599,simonw,2023-05-25T22:09:53Z,2023-05-25T22:09:53Z,OWNER,Updated docs: https://docs.datasette.io/en/latest/json_api.html#enabling-cors,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563563438,https://api.github.com/repos/simonw/datasette/issues/2079,1563563438,IC_kwDOBm6k_c5dMhWu,9599,simonw,2023-05-25T22:08:28Z,2023-05-25T22:08:28Z,OWNER,"I ran this on https://www.example.com/ twice using the console: ```javascript fetch( `https://latest.datasette.io/ephemeral/foo/1/-/update`, { method: ""POST"", mode: ""cors"", headers: { Authorization: `Bearer tok`, ""Content-Type"": ""application/json"", }, body: JSON.stringify({update: {blah: 1}}), } ) .then((r) => r.json()) .then((data) => { console.log(data); }); ``` And got this in the network pane: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563558915,https://api.github.com/repos/simonw/datasette/issues/2079,1563558915,IC_kwDOBm6k_c5dMgQD,9599,simonw,2023-05-25T22:04:41Z,2023-05-25T22:04:41Z,OWNER,I'm going with 3600 for 1 hour instead of 600 for 10 minutes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563547097,https://api.github.com/repos/simonw/datasette/issues/2079,1563547097,IC_kwDOBm6k_c5dMdXZ,9599,simonw,2023-05-25T21:51:38Z,2023-05-25T21:51:38Z,OWNER,"Also need to update this documentation: https://github.com/simonw/datasette/blob/9584879534ff0556e04e4c420262972884cac87b/docs/json_api.rst?plain=1#L453-L465 Or maybe make that automated via `cog`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2078#issuecomment-1563522011,https://api.github.com/repos/simonw/datasette/issues/2078,1563522011,IC_kwDOBm6k_c5dMXPb,9599,simonw,2023-05-25T21:22:30Z,2023-05-25T21:22:30Z,OWNER,"This is bad: ```python async def __call__(self, request, datasette): try: handler = getattr(self, request.method.lower()) return await handler(request, datasette) except AttributeError: return await self.method_not_allowed(request) ``` Because it hides any `AttributeError` exceptions that might occur in the view code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563511171,https://api.github.com/repos/simonw/datasette/issues/2078,1563511171,IC_kwDOBm6k_c5dMUmD,9599,simonw,2023-05-25T21:11:20Z,2023-05-25T21:13:05Z,OWNER,I'm going to call this `VerbView` for the moment. Might even rename it to `View` later.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563498048,https://api.github.com/repos/simonw/datasette/issues/2078,1563498048,IC_kwDOBm6k_c5dMRZA,9599,simonw,2023-05-25T20:57:52Z,2023-05-25T20:58:13Z,OWNER,"Here's a new `BaseView` class that automatically populates `OPTIONS` based on available methods: ```python class BaseView: async def head(self, *args, **kwargs): try: response = await self.get(*args, **kwargs) response.body = b"""" return response except AttributeError: raise async def method_not_allowed(self, request): if ( request.path.endswith("".json"") or request.headers.get(""content-type"") == ""application/json"" ): response = Response.json( {""ok"": False, ""error"": ""Method not allowed""}, status=405 ) else: response = Response.text(""Method not allowed"", status=405) return response async def options(self, request, *args, **kwargs): response = Response.text(""ok"") response.headers[""allow""] = "", "".join( method.upper() for method in (""head"", ""get"", ""post"", ""put"", ""patch"", ""delete"") if hasattr(self, method) ) return response async def __call__(self, request, datasette): try: handler = getattr(self, request.method.lower()) return await handler(request, datasette) except AttributeError: return await self.method_not_allowed(request) class DemoView(BaseView): async def get(self, datasette, request): return Response.text(""Hello there! {} - {}"".format(datasette, request)) post = get ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563488929,https://api.github.com/repos/simonw/datasette/issues/2078,1563488929,IC_kwDOBm6k_c5dMPKh,9599,simonw,2023-05-25T20:48:12Z,2023-05-25T20:48:39Z,OWNER,"Actually no need for that extra level of parameter detection: `BaseView.__call__` should _always_ take `datasette, request` - `scope` and `receive` are both available on `request`, and `send` is only needed if you're not planning on returning a `Response` object. So the `get` and `post` and suchlike methods should take `datasette` and `request` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563444296,https://api.github.com/repos/simonw/datasette/issues/2078,1563444296,IC_kwDOBm6k_c5dMERI,9599,simonw,2023-05-25T20:06:08Z,2023-05-25T20:06:08Z,OWNER,"This prototype seems to work well: ```diff diff --git a/datasette/app.py b/datasette/app.py index d7dace67..ed0edf28 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -17,6 +17,7 @@ import secrets import sys import threading import time +import types import urllib.parse from concurrent import futures from pathlib import Path @@ -1266,6 +1267,8 @@ class Datasette: # TODO: /favicon.ico and /-/static/ deserve far-future cache expires add_route(favicon, ""/favicon.ico"") + add_route(wrap_view(DemoView, self), '/demo') + add_route( asgi_static(app_root / ""datasette"" / ""static""), r""/-/static/(?P.*)$"" ) @@ -1673,8 +1676,46 @@ def _cleaner_task_str(task): return _cleaner_task_str_re.sub("""", s) -def wrap_view(view_fn, datasette): - @functools.wraps(view_fn) +class DemoView: + async def __call__(self, datasette, request): + return Response.text(""Hello there! {} - {}"".format(datasette, request)) + +def wrap_view(view_fn_or_class, datasette): + is_function = isinstance(view_fn_or_class, types.FunctionType) + if is_function: + return wrap_view_function(view_fn_or_class, datasette) + else: + if not isinstance(view_fn_or_class, type): + raise ValueError(""view_fn_or_class must be a function or a class"") + return wrap_view_class(view_fn_or_class, datasette) + + +def wrap_view_class(view_class, datasette): + async def async_view_for_class(request, send): + instance = view_class() + if inspect.iscoroutinefunction(instance.__call__): + return await async_call_with_supported_arguments( + instance.__call__, + scope=request.scope, + receive=request.receive, + send=send, + request=request, + datasette=datasette, + ) + else: + return call_with_supported_arguments( + instance.__call__, + scope=request.scope, + receive=request.receive, + send=send, + request=request, + datasette=datasette, + ) + + return async_view_for_class + + +def wrap_view_function(view_fn, datasette): async def async_view_fn(request, send): if inspect.iscoroutinefunction(view_fn): response = await async_call_with_supported_arguments( ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563419066,https://api.github.com/repos/simonw/datasette/issues/2078,1563419066,IC_kwDOBm6k_c5dL-G6,9599,simonw,2023-05-25T19:42:16Z,2023-05-25T19:43:08Z,OWNER,"Maybe what I want here is the ability to register classes with the router - and have the router know that if it's a class it should instantiate it via its constructor and then await `__call__` it. The neat thing about it is that it can reduce the risk of having a class instance that accidentally shares state between requests. It also encourages that each class only responds based on the `datasette, request, ...` objects that are passed to its methods.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563359114,https://api.github.com/repos/simonw/datasette/issues/2078,1563359114,IC_kwDOBm6k_c5dLveK,9599,simonw,2023-05-25T18:47:57Z,2023-05-25T18:47:57Z,OWNER,"Oops, that broke everything: ``` @documented async def await_me_maybe(value: typing.Any) -> typing.Any: ""If value is callable, call it. If awaitable, await it. Otherwise return it."" > if callable(value): E TypeError: 'module' object is not callable ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563329245,https://api.github.com/repos/simonw/datasette/issues/2078,1563329245,IC_kwDOBm6k_c5dLoLd,9599,simonw,2023-05-25T18:26:47Z,2023-05-25T18:28:08Z,OWNER,"With type hints and a namedtuple: ```python import asyncio import types from typing import NamedTuple, Any class CallableStatus(NamedTuple): is_callable: bool is_async_callable: bool def check_callable(obj: Any) -> CallableStatus: if not callable(obj): return CallableStatus(False, False) if isinstance(obj, type): # It's a class return CallableStatus(True, False) if isinstance(obj, types.FunctionType): return CallableStatus(True, asyncio.iscoroutinefunction(obj)) if hasattr(obj, ""__call__""): return CallableStatus(True, asyncio.iscoroutinefunction(obj.__call__)) assert False, ""obj {} is somehow callable with no __call__ method"".format(repr(obj)) ``` ```python for thing in ( async_func, non_async_func, AsyncClass(), NotAsyncClass(), ClassNoCall(), AsyncClass, NotAsyncClass, ClassNoCall, ): print(thing, check_callable(thing)) ``` ``` CallableStatus(is_callable=True, is_async_callable=True) CallableStatus(is_callable=True, is_async_callable=False) <__main__.AsyncClass object at 0x106ba7490> CallableStatus(is_callable=True, is_async_callable=True) <__main__.NotAsyncClass object at 0x106740150> CallableStatus(is_callable=True, is_async_callable=False) <__main__.ClassNoCall object at 0x10676d910> CallableStatus(is_callable=False, is_async_callable=False) CallableStatus(is_callable=True, is_async_callable=False) CallableStatus(is_callable=True, is_async_callable=False) CallableStatus(is_callable=True, is_async_callable=False) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563326000,https://api.github.com/repos/simonw/datasette/issues/2078,1563326000,IC_kwDOBm6k_c5dLnYw,9599,simonw,2023-05-25T18:23:38Z,2023-05-25T18:23:38Z,OWNER,I don't like that `is_callable()` implies a single boolean result but actually returns a pair. I'll call it `check_callable(obj)` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563318598,https://api.github.com/repos/simonw/datasette/issues/2078,1563318598,IC_kwDOBm6k_c5dLllG,9599,simonw,2023-05-25T18:17:03Z,2023-05-25T18:21:25Z,OWNER,"I think I want that to return `(is_callable, is_async)` - so I can both test if the thing can be called AND if it should be awaited in the same operation (without any exceptions). I tried this: ```python def is_callable(obj): ""Returns (is_callable, is_async_callable)"" if not callable(obj): return False, False if isinstance(obj, types.FunctionType): return True, asyncio.iscoroutinefunction(obj) if hasattr(obj, '__call__'): return True, asyncio.iscoroutinefunction(obj.__call__) return False, False ``` ```python for thing in ( async_func, non_async_func, AsyncClass(), NotAsyncClass(), ClassNoCall(), AsyncClass, NotAsyncClass, ClassNoCall ): print(thing, is_callable(thing)) ``` And got: ``` (True, True) (True, False) <__main__.AsyncClass object at 0x106cce490> (True, True) <__main__.NotAsyncClass object at 0x106ccf710> (True, False) <__main__.ClassNoCall object at 0x106ccc810> (False, False) (True, True) (True, False) (True, False) ``` Which is almost right, but I don't like that `AsyncClass` is shown as callable (which it is, since it's a class) and awaitable (which it is not - the `__call__` method may be async but calling the class constructor is not). So I'm going to detect classes using `isinstance(obj, type)`. ```python def is_callable(obj): ""Returns (is_callable, is_async_callable)"" if not callable(obj): return False, False if isinstance(obj, type): # It's a class return True, False if isinstance(obj, types.FunctionType): return True, asyncio.iscoroutinefunction(obj) if hasattr(obj, '__call__'): return True, asyncio.iscoroutinefunction(obj.__call__) assert False, ""obj {} somehow is callable with no __call__ method"".format(obj) ``` I am reasonably confident the `AssertionError` can never be raised. And now: ``` (True, True) (True, False) <__main__.AsyncClass object at 0x106ccfa50> (True, True) <__main__.NotAsyncClass object at 0x106ccc8d0> (True, False) <__main__.ClassNoCall object at 0x106cd7690> (False, False) (True, False) (True, False) (True, False) ``` Which is what I wanted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563308919,https://api.github.com/repos/simonw/datasette/issues/2078,1563308919,IC_kwDOBm6k_c5dLjN3,9599,simonw,2023-05-25T18:08:34Z,2023-05-25T18:08:34Z,OWNER,"After much fiddling this seems to work: ```python import asyncio, types def is_async_callable(obj): if not callable(obj): raise ValueError(""Object is not callable"") if isinstance(obj, types.FunctionType): return asyncio.iscoroutinefunction(obj) if hasattr(obj, '__call__'): return asyncio.iscoroutinefunction(obj.__call__) raise ValueError(""Not a function and has no __call__ attribute"") ``` Tested like so: ```python class AsyncClass: async def __call__(self): pass class NotAsyncClass: def __call__(self): pass class ClassNoCall: pass async def async_func(): pass def non_async_func(): pass for thing in (AsyncClass(), NotAsyncClass(), ClassNoCall(), async_func, non_async_func): try: print(thing, is_async_callable(thing)) except Exception as ex: print(thing, ex) ``` ``` <__main__.AsyncClass object at 0x106c32150> True <__main__.NotAsyncClass object at 0x106c32390> False <__main__.ClassNoCall object at 0x106c32750> Object is not callable True False ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563294669,https://api.github.com/repos/simonw/datasette/issues/2078,1563294669,IC_kwDOBm6k_c5dLfvN,9599,simonw,2023-05-25T17:57:06Z,2023-05-25T17:57:06Z,OWNER,"I may need to be able to detect if a class instance has an `async def __call__` method - I think I can do that like so: ```python def iscoroutinefunction(obj): if inspect.iscoroutinefunction(obj): return True if hasattr(obj, '__call__') and inspect.iscoroutinefunction(obj.__call__): return True return False ``` From https://github.com/encode/starlette/issues/886#issuecomment-606585152","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563292373,https://api.github.com/repos/simonw/datasette/issues/2078,1563292373,IC_kwDOBm6k_c5dLfLV,9599,simonw,2023-05-25T17:55:12Z,2023-05-25T17:55:30Z,OWNER,"So I think subclasses of `BaseView` need to offer a callable which accepts all five of the DI arguments - `datasette`, `request`, `scope`, `send`, `receive` - and then makes a decision based on the HTTP verb as to which method of the class to call. Those methods themselves can accept a subset of those parameters and will only be sent on to them. Having two layers of parameter detection feels a little bit untidy, but I think it will work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/pull/2053#issuecomment-1563285150,https://api.github.com/repos/simonw/datasette/issues/2053,1563285150,IC_kwDOBm6k_c5dLdae,9599,simonw,2023-05-25T17:48:50Z,2023-05-25T17:49:52Z,OWNER,"Uncommitted experimental code: ```diff diff --git a/datasette/views/database.py b/datasette/views/database.py index 455ebd1f..85775433 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -909,12 +909,13 @@ async def query_view( elif format_ in datasette.renderers.keys(): # Dispatch request to the correct output format renderer # (CSV is not handled here due to streaming) + print(data) result = call_with_supported_arguments( datasette.renderers[format_][0], datasette=datasette, - columns=columns, - rows=rows, - sql=sql, + columns=data[""rows""][0].keys(), + rows=data[""rows""], + sql='', query_name=None, database=db.name, table=None, @@ -923,7 +924,7 @@ async def query_view( # These will be deprecated in Datasette 1.0: args=request.args, data={ - ""rows"": rows, + ""rows"": data[""rows""], }, # TODO what should this be? ) result = await await_me_maybe(result) diff --git a/docs/index.rst b/docs/index.rst index 5a9cc7ed..254ed3da 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -57,6 +57,7 @@ Contents settings introspection custom_templates + template_context plugins writing_plugins plugin_hooks ``` Where `docs/template_context.rst` looked like this: ```rst .. _template_context: Template context ================ .. currentmodule:: datasette.context This page describes the variables made available to templates used by Datasette to render different pages of the application. .. autoclass:: QueryContext :members: ``` And `datasette/context.py` had this: ```python from dataclasses import dataclass @dataclass class QueryContext: """""" Used by the ``/database`` page when showing the results of a SQL query """""" id: int ""Id is a thing"" rows: list[dict] ""Name is another thing"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/issues/2078#issuecomment-1563283939,https://api.github.com/repos/simonw/datasette/issues/2078,1563283939,IC_kwDOBm6k_c5dLdHj,9599,simonw,2023-05-25T17:47:38Z,2023-05-25T17:47:38Z,OWNER,"The idea behind `wrap_view()` is dependency injection - it's mainly used by plugins: https://docs.datasette.io/en/0.64.3/plugin_hooks.html#register-routes-datasette But I like the pattern so I started using it for some of Datasette's own features. I should use it for _all_ of Datasette's own features. But I still like the way `BaseView` helps with running different code for GET/POST/etc verbs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563282327,https://api.github.com/repos/simonw/datasette/issues/2078,1563282327,IC_kwDOBm6k_c5dLcuX,9599,simonw,2023-05-25T17:46:05Z,2023-05-25T17:46:05Z,OWNER,"Here's what `wrap_view()` does: https://github.com/simonw/datasette/blob/49184c569cd70efbda4f3f062afef3a34401d8d5/datasette/app.py#L1676-L1700 It's used e.g. here: https://github.com/simonw/datasette/blob/49184c569cd70efbda4f3f062afef3a34401d8d5/datasette/app.py#L1371-L1375 The `BaseView` thing meanwhile works like this: https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/views/base.py#L56-L157","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/sqlite-utils/issues/552#issuecomment-1556292204,https://api.github.com/repos/simonw/sqlite-utils/issues/552,1556292204,IC_kwDOCGYnMM5cwyJs,9599,simonw,2023-05-21T21:05:15Z,2023-05-21T21:05:15Z,OWNER,Now live at https://sqlite-utils.datasette.io/en/latest/installation.html#setting-up-shell-completion,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718612569,Document how to setup shell auto-completion, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556291915,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556291915,IC_kwDOCGYnMM5cwyFL,9599,simonw,2023-05-21T21:04:03Z,2023-05-21T21:04:03Z,OWNER,Now live at https://sqlite-utils.datasette.io/en/latest/cli.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556288300,https://api.github.com/repos/simonw/sqlite-utils/issues/553,1556288300,IC_kwDOCGYnMM5cwxMs,9599,simonw,2023-05-21T20:48:01Z,2023-05-21T20:48:01Z,OWNER,If https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries looks good I can merge this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718635018,Reformatted CLI examples in docs, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556288270,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556288270,IC_kwDOCGYnMM5cwxMO,9599,simonw,2023-05-21T20:47:51Z,2023-05-21T20:47:51Z,OWNER,This page has all of the changes: https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556287599,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556287599,IC_kwDOCGYnMM5cwxBv,9599,simonw,2023-05-21T20:44:55Z,2023-05-21T20:44:55Z,OWNER,"Put this in a PR so I can preview it: - #553 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556269616,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556269616,IC_kwDOCGYnMM5cwsow,9599,simonw,2023-05-21T19:33:13Z,2023-05-21T19:33:13Z,OWNER,Now released: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-32,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556265772,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556265772,IC_kwDOCGYnMM5cwrss,9599,simonw,2023-05-21T19:16:15Z,2023-05-21T19:16:15Z,OWNER,"Another option: That's using this markup: ``` Newline-delimited JSON ~~~~~~~~~~~~~~~~~~~~~~ Use ``--nl`` to get back newline-delimited JSON objects: .. code-block:: bash sqlite-utils dogs.db ""select * from dogs"" --nl .. code-block:: output {""id"": 1, ""age"": 4, ""name"": ""Cleo""} {""id"": 2, ""age"": 2, ""name"": ""Pancakes""} ``` And this extra CSS: ```css .highlight-output .highlight { border-left: 9px solid #30c94f; } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556263182,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556263182,IC_kwDOCGYnMM5cwrEO,9599,simonw,2023-05-21T19:06:48Z,2023-05-21T19:06:48Z,OWNER,"I could split them up into two blocks like this: I do miss the visual indication that one of these is the command and one is the output though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556262574,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556262574,IC_kwDOCGYnMM5cwq6u,9599,simonw,2023-05-21T19:04:59Z,2023-05-21T19:04:59Z,OWNER,"I wrote the docs like this because early examples include both the command and its output: https://sqlite-utils.datasette.io/en/stable/cli.html#returning-json ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556255309,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556255309,IC_kwDOCGYnMM5cwpJN,9599,simonw,2023-05-21T18:42:25Z,2023-05-21T18:42:25Z,OWNER,Tests passed here: https://github.com/simonw/sqlite-utils/actions/runs/5039119716,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7, https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556250236,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556250236,IC_kwDOCGYnMM5cwn58,9599,simonw,2023-05-21T18:25:26Z,2023-05-21T18:25:26Z,OWNER,"Relevant issues: - https://github.com/python/importlib_metadata/issues/406 - https://github.com/PyCQA/flake8/issues/1701 It looks to me like this is only a problem for `flake8` on Python 3.7 - 3.8 and higher work OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7, https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556249984,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556249984,IC_kwDOCGYnMM5cwn2A,9599,simonw,2023-05-21T18:24:48Z,2023-05-21T18:24:48Z,OWNER,"This is blocking: - #549","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556247818,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556247818,IC_kwDOCGYnMM5cwnUK,9599,simonw,2023-05-21T18:17:46Z,2023-05-21T18:17:46Z,OWNER,Draft documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556242262,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556242262,IC_kwDOCGYnMM5cwl9W,9599,simonw,2023-05-21T18:00:05Z,2023-05-21T18:00:05Z,OWNER,"Failing `mypy` test: https://github.com/simonw/sqlite-utils/actions/runs/5038983349/jobs/9036828465 ``` sqlite_utils/cli.py:37: error: Skipping analyzing ""trogon"": module is installed, but missing library stubs or py.typed marker [import] sqlite_utils/cli.py:37: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 52 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon, https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556241812,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556241812,IC_kwDOCGYnMM5cwl2U,9599,simonw,2023-05-21T17:58:25Z,2023-05-21T17:58:25Z,OWNER,Documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon, https://github.com/simonw/sqlite-utils/issues/548#issuecomment-1556231832,https://api.github.com/repos/simonw/sqlite-utils/issues/548,1556231832,IC_kwDOCGYnMM5cwjaY,9599,simonw,2023-05-21T17:24:13Z,2023-05-21T17:24:13Z,OWNER,"Oh, I see why that is now: https://github.com/simonw/sqlite-utils/blob/6027f3ea6939a399aeef2578fca17efec0e539df/sqlite_utils/cli.py#L2670-L2679 This is because of the following command: sqlite-utils analyze-tables table1 table2 --column x Since you can pass multiple tables AND multiple columns, the tool currently assumes that the column(s) you specify may be available on a subset of the provided tables. I'm going to change this so if the column is not on ANY of those tables you get an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718576761,analyze-tables should validate provide --column names, https://github.com/simonw/sqlite-utils/issues/547#issuecomment-1556228395,https://api.github.com/repos/simonw/sqlite-utils/issues/547,1556228395,IC_kwDOCGYnMM5cwikr,9599,simonw,2023-05-21T17:11:15Z,2023-05-21T17:11:15Z,OWNER,"This will be a cosmetic change to the CLI output only - the options to save data to the database and the Python API function will continue to return `[(None, 158)]`. I can add an optimization though to avoid running the SQL count query if we know that it's all `null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718572201,No need to show common values if everything is null, https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556225788,https://api.github.com/repos/simonw/sqlite-utils/issues/544,1556225788,IC_kwDOCGYnMM5cwh78,9599,simonw,2023-05-21T17:02:05Z,2023-05-21T17:02:05Z,OWNER,"New docs: - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-analyze-tables - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#analyze-tables - https://sqlite-utils.datasette.io/en/latest/python-api.html#analyzing-a-column - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.analyze_column New help output: ``` % sqlite-utils analyze-tables --help Usage: sqlite-utils analyze-tables [OPTIONS] PATH [TABLES]... Analyze the columns in one or more tables Example: sqlite-utils analyze-tables data.db trees Options: -c, --column TEXT Specific columns to analyze --save Save results to _analyze_tables table --common-limit INTEGER How many common values --no-most Skip most common values --no-least Skip least common values --load-extension TEXT Path to SQLite extension, with optional :entrypoint -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718515590,New options for analyze-tables --common-limit --no-most and --no-least, https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213396,https://api.github.com/repos/simonw/sqlite-utils/issues/546,1556213396,IC_kwDOCGYnMM5cwe6U,9599,simonw,2023-05-21T15:58:12Z,2023-05-21T16:18:46Z,OWNER,"Documentation preview: - https://sqlite-utils--546.org.readthedocs.build/en/546/cli.html#cli-analyze-tables - https://sqlite-utils--546.org.readthedocs.build/en/546/cli-reference.html#analyze-tables - https://sqlite-utils--546.org.readthedocs.build/en/546/python-api.html#analyzing-a-column - https://sqlite-utils--546.org.readthedocs.build/en/546/reference.html#sqlite_utils.db.Table.analyze_column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718550688,"Analyze tables options: --common-limit, --no-most, --no-least", https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556211643,https://api.github.com/repos/simonw/sqlite-utils/issues/544,1556211643,IC_kwDOCGYnMM5cwee7,9599,simonw,2023-05-21T15:48:17Z,2023-05-21T15:48:17Z,OWNER,I generated the commit message in https://github.com/simonw/sqlite-utils/commit/1c1991b447a1ddd3d61d9d4a8a1d6a9da47ced20 using `git diff | llm --system 'describe this change'`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718515590,New options for analyze-tables --common-limit --no-most and --no-least, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556210844,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556210844,IC_kwDOCGYnMM5cweSc,9599,simonw,2023-05-21T15:44:10Z,2023-05-21T15:44:10Z,OWNER,"It looks like `nargs=-1` on a positional argument isn't yet supported - opened an issue here: - https://github.com/Textualize/trogon/issues/4","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556191894,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556191894,IC_kwDOCGYnMM5cwZqW,9599,simonw,2023-05-21T14:20:14Z,2023-05-21T14:20:14Z,OWNER,"Opened a feature request for customizing the help and command name: - https://github.com/Textualize/trogon/issues/2","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556190531,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556190531,IC_kwDOCGYnMM5cwZVD,9599,simonw,2023-05-21T14:13:43Z,2023-05-21T14:13:43Z,OWNER,"OK, this works! ![trogon](https://github.com/simonw/sqlite-utils/assets/9599/2ae194c5-ec82-471a-9d1b-b01b3f2632f3) To try it out, install that branch from GitHub: pip install https://github.com/simonw/sqlite-utils/archive/refs/heads/trogon.zip Then run this: sqlite-utils install trogon And this: sqlite-utils tui ","{""total_count"": 5, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 3, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556189823,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556189823,IC_kwDOCGYnMM5cwZJ_,9599,simonw,2023-05-21T14:09:59Z,2023-05-21T14:09:59Z,OWNER,"I don't want to add `trogon` as a default dependency because it's a little heavy - it pulls in all of Rich and Textual as well. People who use `sqlite-utils` just for its Python API won't benefit from this - it's a CLI feature only. But I have a `sqlite-utils install ...` command for helping people to install packages into the same virtual environment as `sqlite-utils` no matter how they installed that tool: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install So I can treat Trogon as an optional dependency and add the `sqlite-utils tui` command only if that package is also installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/datasette/issues/2073#issuecomment-1546119773,https://api.github.com/repos/simonw/datasette/issues/2073,1546119773,IC_kwDOBm6k_c5cJ-pd,9599,simonw,2023-05-12T18:24:07Z,2023-05-12T18:24:07Z,OWNER,"Here's a demo of this breaking in Datasette Lite: https://lite.datasette.io/?sql=https://gist.github.com/simonw/261564c0ca01567df6eeb9b222b8be84&json=https%3A%2F%2Fcdn.jsdelivr.net%2Fnpm%2Fweb-features%2Findex.json#/data/baseline?_filter_column_1=is_baseline&_filter_op_1=exact&_filter_value_1=1&_filter_column_2=&_filter_op_2=notnull__1&_filter_value_2=1&_filter_column=&_filter_op=exact&_filter_value=&_sort=&_facet=is_baseline Here's a SQL query that demonstrates the underlying issue: ```sql select 'working', count(*) from baseline where is_baseline = 1 union all select 'broken', count(*) from baseline where is_baseline = '1' ``` https://lite.datasette.io/?sql=https://gist.github.com/simonw/261564c0ca01567df6eeb9b222b8be84&json=https%3A%2F%2Fcdn.jsdelivr.net%2Fnpm%2Fweb-features%2Findex.json#/data?sql=select+%27working%27%2C+count%28*%29+from+baseline+where+is_baseline+%3D+1%0Aunion+all%0Aselect+%27broken%27%2C+count%28*%29+from+baseline+where+is_baseline+%3D+%271%27 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1708030220,Faceting doesn't work against integer columns in views, https://github.com/simonw/datasette/issues/2073#issuecomment-1546117538,https://api.github.com/repos/simonw/datasette/issues/2073,1546117538,IC_kwDOBm6k_c5cJ-Gi,9599,simonw,2023-05-12T18:21:38Z,2023-05-12T18:21:38Z,OWNER,"https://latest.datasette.io/fixtures doesn't currently have a view with any integer columns in it, making this bug harder to demonstrate there. I can't replicate the bug using https://datasette.io/content/plugins?_facet=stargazers_count&stargazers_count=3 - I would expect that not to work correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1708030220,Faceting doesn't work against integer columns in views, https://github.com/simonw/datasette/issues/2070#issuecomment-1540494121,https://api.github.com/repos/simonw/datasette/issues/2070,1540494121,IC_kwDOBm6k_c5b0hMp,9599,simonw,2023-05-09T16:25:00Z,2023-05-09T16:25:00Z,OWNER,Can now be used here: https://github.com/simonw/datasette/actions/workflows/deploy-branch-preview.yml,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1702354223,Mechanism for deploying a preview of a branch using Vercel, https://github.com/simonw/datasette/issues/2070#issuecomment-1540491751,https://api.github.com/repos/simonw/datasette/issues/2070,1540491751,IC_kwDOBm6k_c5b0gnn,9599,simonw,2023-05-09T16:23:12Z,2023-05-09T16:23:12Z,OWNER,Added a actions `BRANCH_PREVIEW_VERCEL_TOKEN` secret to this repository.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1702354223,Mechanism for deploying a preview of a branch using Vercel, https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539157643,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1539157643,IC_kwDOCGYnMM5bva6L,9599,simonw,2023-05-08T22:45:09Z,2023-05-08T22:45:21Z,OWNER,"Here's an example from the new tests: https://github.com/simonw/sqlite-utils/blob/a75abeb61b91a28650d3b9933e7ec80ad0d92529/tests/test_create.py#L291-L307","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`, https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1539109816,https://api.github.com/repos/simonw/sqlite-utils/issues/448,1539109816,IC_kwDOCGYnMM5bvPO4,9599,simonw,2023-05-08T22:01:00Z,2023-05-08T22:01:00Z,OWNER,"This is being handled in: - #520","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279144769,Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto', https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1539109587,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1539109587,IC_kwDOCGYnMM5bvPLT,9599,simonw,2023-05-08T22:00:46Z,2023-05-08T22:00:46Z,OWNER,"> Hey, isn't this essentially the same issue as #448 ? Yes it is, good catch!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,rows_from_file() raises confusing error if file-like object is not in binary mode, https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1539108140,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1539108140,IC_kwDOCGYnMM5bvO0s,9599,simonw,2023-05-08T21:59:41Z,2023-05-08T21:59:41Z,OWNER,That original example passes against `main` now.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail, https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1539101853,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1539101853,IC_kwDOCGYnMM5bvNSd,9599,simonw,2023-05-08T21:52:44Z,2023-05-08T21:52:44Z,OWNER,I like the `lambda-{uuid}` idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539100300,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539100300,IC_kwDOCGYnMM5bvM6M,9599,simonw,2023-05-08T21:50:51Z,2023-05-08T21:50:51Z,OWNER,Seeing as `sqlite-utils` doesn't currently provide mechanisms for adding `check` constraints like this I'm going to leave this - I'm happy with the fix I put in for the `not null` constraints.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539099703,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539099703,IC_kwDOCGYnMM5bvMw3,9599,simonw,2023-05-08T21:50:06Z,2023-05-08T21:50:06Z,OWNER,"Applying the fix from the PR here doesn't fix the above problem either: - https://github.com/simonw/sqlite-utils/pull/515 So it looks like these kinds of `check` constraints currently aren't compatible with how `upsert()` works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539094287,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539094287,IC_kwDOCGYnMM5bvLcP,9599,simonw,2023-05-08T21:44:11Z,2023-05-08T21:44:11Z,OWNER,"OK, this fails silently: ```python import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute('''CREATE TABLE employees ( id INTEGER PRIMARY KEY, name TEXT, age INTEGER, salary REAL, CHECK (salary is not null and salary > 0) );''') db[""employees""].upsert({""id"": 1, ""name"": ""Bob""}, pk=""id"") list(db[""employees""].rows) ```` It outputs: ```python [] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539079507,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539079507,IC_kwDOCGYnMM5bvH1T,9599,simonw,2023-05-08T21:28:37Z,2023-05-08T21:28:37Z,OWNER,"> This means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted. Huh... on that basis, it's possible my fix in https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304 is incomplete. I only covered the 'not null' case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539078429,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539078429,IC_kwDOCGYnMM5bvHkd,9599,simonw,2023-05-08T21:27:40Z,2023-05-08T21:27:40Z,OWNER,"Dupe of: - #538","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/pull/515#issuecomment-1539077777,https://api.github.com/repos/simonw/sqlite-utils/issues/515,1539077777,IC_kwDOCGYnMM5bvHaR,9599,simonw,2023-05-08T21:27:10Z,2023-05-08T21:27:10Z,OWNER,I should have spotted this PR before I shipped my own fix! https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194930,"upsert new rows with constraints, fixes #514", https://github.com/simonw/sqlite-utils/pull/519#issuecomment-1539058795,https://api.github.com/repos/simonw/sqlite-utils/issues/519,1539058795,IC_kwDOCGYnMM5bvCxr,9599,simonw,2023-05-08T21:12:52Z,2023-05-08T21:12:52Z,OWNER,"This is a really neat fix, thank you.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1505568103,Fixes breaking DEFAULT values, https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539055393,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1539055393,IC_kwDOCGYnMM5bvB8h,9599,simonw,2023-05-08T21:10:06Z,2023-05-08T21:10:06Z,OWNER,Thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`, https://github.com/simonw/sqlite-utils/pull/528#issuecomment-1539053230,https://api.github.com/repos/simonw/sqlite-utils/issues/528,1539053230,IC_kwDOCGYnMM5bvBau,9599,simonw,2023-05-08T21:08:23Z,2023-05-08T21:08:23Z,OWNER,"I fixed this in: - #527 Will fully remove this misfeature in: - #542","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578793661,Enable `Table.convert()` on falsey values, https://github.com/simonw/sqlite-utils/issues/542#issuecomment-1539052467,https://api.github.com/repos/simonw/sqlite-utils/issues/542,1539052467,IC_kwDOCGYnMM5bvBOz,9599,simonw,2023-05-08T21:07:41Z,2023-05-08T21:07:41Z,OWNER,"Relevant commits (will mostly revert these): - https://github.com/simonw/sqlite-utils/commit/455c35b512895c19bf922c2b804d750d27cb8dbd - https://github.com/simonw/sqlite-utils/commit/e0ec4c345129996011951e400388fd74865f65a2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1700936245,Remove `skip_false=True` and `--no-skip-false` in `sqlite-utils` 4.0, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539051724,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1539051724,IC_kwDOCGYnMM5bvBDM,9599,simonw,2023-05-08T21:07:04Z,2023-05-08T21:07:04Z,OWNER,"Updated documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-data-in-columns - https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#convert","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539035838,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1539035838,IC_kwDOCGYnMM5bu9K-,9599,simonw,2023-05-08T20:55:00Z,2023-05-08T20:55:00Z,OWNER,"I'm going to go with `--no-skip-false` as the CLI option. It's ugly, but this whole thing is ugly. I'm going to make a note to remove this misfeature in `sqlite-utils` 4.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539033736,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1539033736,IC_kwDOCGYnMM5bu8qI,9599,simonw,2023-05-08T20:52:51Z,2023-05-08T20:52:51Z,OWNER,"OK, I implemented that at the Python API level. I need to decide how it should work for the `sqlite-utils convert` command too: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#convert","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/sqlite-utils/issues/530#issuecomment-1539018912,https://api.github.com/repos/simonw/sqlite-utils/issues/530,1539018912,IC_kwDOCGYnMM5bu5Cg,9599,simonw,2023-05-08T20:39:00Z,2023-05-08T20:39:00Z,OWNER,"I think the natural place to add these in the Python library API would be https://sqlite-utils.datasette.io/en/stable/python-api.html#adding-foreign-key-constraints - maybe something like this: ```python db[""books""].add_foreign_key(""author_id"", ""authors"", ""id"", on_delete=RESTRICT) ``` Then for the CLI tool could be added to https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key ``` sqlite-utils add-foreign-key my.db books author_id authors id --on-update SET_NULL ``` I wouldn't support these for the other methods of adding foreign keys - `foreign_keys(...)` for the various `.insert()` etc methods and the `add_foreign_keys(...)` bulk menthod.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1595340692,"add ability to configure ""on delete"" and ""on update"" attributes of foreign keys:", https://github.com/simonw/sqlite-utils/issues/530#issuecomment-1539015064,https://api.github.com/repos/simonw/sqlite-utils/issues/530,1539015064,IC_kwDOCGYnMM5bu4GY,9599,simonw,2023-05-08T20:35:07Z,2023-05-08T20:35:07Z,OWNER,"Wow, this is a neat feature I didn't know about. Looks like there are a bunch of options: - NO ACTION (default) - RESTRICT: application is prohibited from deleting a parent key when there exists one or more child keys mapped to it - SET NULL: when a parent key is deleted the child key columns of all rows in the child table that mapped to the parent key are set to contain SQL NULL values - SET DEFAULT: set a specific default - CASCADE: propagates the delete or update operation on the parent key to each dependent child key","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1595340692,"add ability to configure ""on delete"" and ""on update"" attributes of foreign keys:", https://github.com/simonw/sqlite-utils/issues/532#issuecomment-1539009453,https://api.github.com/repos/simonw/sqlite-utils/issues/532,1539009453,IC_kwDOCGYnMM5bu2ut,9599,simonw,2023-05-08T20:30:29Z,2023-05-08T20:30:42Z,OWNER,"Here's an improvement: ``` % sqlite-utils insert /tmp/b.db blah /tmp/blah.txt [####################################] 100% Error: Invalid JSON - use --csv for CSV or --tsv for TSV files JSON error: Expecting value: line 1 column 1 (char 0) ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620254998,Show more information when JSON can't be imported with sqlite-utils insert, https://github.com/simonw/sqlite-utils/issues/532#issuecomment-1539006509,https://api.github.com/repos/simonw/sqlite-utils/issues/532,1539006509,IC_kwDOCGYnMM5bu2At,9599,simonw,2023-05-08T20:28:56Z,2023-05-08T20:28:56Z,OWNER,Was this a newline-delimited JSON file perhaps?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620254998,Show more information when JSON can't be imported with sqlite-utils insert, https://github.com/simonw/sqlite-utils/issues/541#issuecomment-1538963959,https://api.github.com/repos/simonw/sqlite-utils/issues/541,1538963959,IC_kwDOCGYnMM5burn3,9599,simonw,2023-05-08T19:59:34Z,2023-05-08T19:59:34Z,OWNER,"There are 8 failing tests left: ``` ==== short test summary info ==== FAILED tests/test_cli_memory.py::test_memory_csv[False-test] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_csv[False-t] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_csv[False-t1] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_tsv[False] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_dump[extra_args0] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_two_files_with_same_stem - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedate] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: .convert_value at 0x106bcaca0> FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedatetime] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: .convert_value at 0x106bc9620> ERROR tests/test_cli.py::test_upsert_analyze - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> ==== 8 failed, 894 passed, 4 skipped, 1 error in 6.27s ==== ``` Full traceback here: https://gist.github.com/simonw/b40b3e814729d6c02a0302a84ce54d9e","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1700840265,Get tests to pass with `pytest -Werror`, https://github.com/simonw/sqlite-utils/issues/534#issuecomment-1538933540,https://api.github.com/repos/simonw/sqlite-utils/issues/534,1538933540,IC_kwDOCGYnMM5bukMk,9599,simonw,2023-05-08T19:34:37Z,2023-05-08T19:34:37Z,OWNER,"On macOS this shows the same warning: ``` % python -Wdefault $(which sqlite-utils) insert dogs.db dogs dogs.csv --csv [############------------------------] 35% [####################################] 100%/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs.csv' encoding='utf-8-sig'> insert_upsert_implementation( ResourceWarning: Enable tracemalloc to get the object allocation traceback ``` The file itself is a `click.File` which is automatically closed - https://click.palletsprojects.com/en/8.1.x/api/#click.File - but it looks like it's the `_io.TextIOWrapper` which is not being closed: https://github.com/simonw/sqlite-utils/blob/2376c452a56b0c3e75e7ca698273434e32945304/sqlite_utils/cli.py#L949-L956","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1622640374, ResourceWarning: unclosed file, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538921774,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538921774,IC_kwDOCGYnMM5buhUu,9599,simonw,2023-05-08T19:24:41Z,2023-05-08T19:24:41Z,OWNER,That fix seems to work!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538910894,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538910894,IC_kwDOCGYnMM5buequ,9599,simonw,2023-05-08T19:16:52Z,2023-05-08T19:17:00Z,OWNER,"How about if I had logic which checked that all not-null columns were provided in the call to `upsert_all()` - and if they were, modified the `INSERT OR IGNORE INTO` to include a placeholder value for those columns that would then be fixed by the later `UPDATE`? Something like this: ```python [ ('INSERT OR IGNORE INTO [comments]([id], name) VALUES(?, ?);', [1, '']), ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1]) ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538903556,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538903556,IC_kwDOCGYnMM5buc4E,9599,simonw,2023-05-08T19:11:24Z,2023-05-08T19:13:23Z,OWNER,"I could detect if this happens using `cursor.rowcount` - not sure how I would recover from it though. This would also require some major re-engineering, since currently it all works by generating a list of SQL queries in advance and applying them inside a loop in `.insert_chunk()`: https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2839-L2878 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538893329,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538893329,IC_kwDOCGYnMM5buaYR,9599,simonw,2023-05-08T19:04:47Z,2023-05-08T19:04:47Z,OWNER,"This feels like a fundamental flaw in the way upserts are implemented by `sqlite-utils`. One fix would be to switch to using the `UPSERT` feature in SQLite: https://www.sqlite.org/lang_UPSERT.html But... > UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04). I still want to support SQLite versions earlier than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538889482,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538889482,IC_kwDOCGYnMM5buZcK,9599,simonw,2023-05-08T19:02:38Z,2023-05-08T19:02:38Z,OWNER,"Here's the code at fault: https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2774-L2788","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538887361,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538887361,IC_kwDOCGYnMM5buY7B,9599,simonw,2023-05-08T19:01:20Z,2023-05-08T19:01:20Z,OWNER,"Here's the problem: ```python import sqlite3 db = sqlite3.connect("":memory:"") db.execute('create table foo (id integer primary key, name not null)') db.execute('insert into foo (id) values (1)') ``` Produces: ``` IntegrityError: NOT NULL constraint failed: foo.name ``` But this: ```python db.execute('insert or ignore into foo (id) values (1)') ``` Completes without an exception.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538801855,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538801855,IC_kwDOCGYnMM5buEC_,9599,simonw,2023-05-08T18:00:17Z,2023-05-08T18:00:17Z,OWNER,"From time in the debugger, after creating the table it ends up doing this: ``` (Pdb) queries_and_params [ ('INSERT OR IGNORE INTO [comments]([id]) VALUES(?);', [1]), ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1]) ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538793817,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538793817,IC_kwDOCGYnMM5buCFZ,9599,simonw,2023-05-08T17:55:10Z,2023-05-08T17:55:10Z,OWNER,"Confirmed - I added this test and it fails: ```python def test_upsert_all_not_null(fresh_db): # https://github.com/simonw/sqlite-utils/issues/538 fresh_db[""comments""].upsert_all( [{""id"": 1, ""name"": ""Cleo""}], pk=""id"", not_null=[""name""], ) assert list(fresh_db[""comments""].rows) == [{""id"": 1, ""name"": ""Cleo""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537514610,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537514610,IC_kwDOCGYnMM5bpJxy,9599,simonw,2023-05-07T18:43:24Z,2023-05-07T18:43:24Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#returning-raw-data-such-as-binary-content - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#memory","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537514069,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537514069,IC_kwDOCGYnMM5bpJpV,9599,simonw,2023-05-07T18:40:18Z,2023-05-07T18:40:18Z,OWNER,"https://docs.readthedocs.io/en/stable/config-file/v2.html suggests: ```yaml build: os: ubuntu-22.04 tools: python: ""3.11"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513912,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537513912,IC_kwDOCGYnMM5bpJm4,9599,simonw,2023-05-07T18:39:29Z,2023-05-07T18:39:29Z,OWNER,"https://readthedocs.org/projects/sqlite-utils/builds/20513034/ said: > Problem in your project's configuration. Invalid ""python.version"": expected one of (2, 2.7, 3, 3.5, 3.6, 3.7, 3.8, pypy3.5), got 3.11","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513653,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537513653,IC_kwDOCGYnMM5bpJi1,9599,simonw,2023-05-07T18:37:59Z,2023-05-07T18:38:19Z,OWNER,"Useful comment here: - https://github.com/urllib3/urllib3/issues/2168#issuecomment-1537360928 > I faced the same issue. I switched to a different Python kernel (3.11.2) and it worked.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507676,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507676,IC_kwDOCGYnMM5bpIFc,9599,simonw,2023-05-07T18:09:43Z,2023-05-07T18:09:54Z,OWNER,"This worked: ```bash sqlite-utils memory /tmp/books3.json:nl \ 'select name from books3' --raw-lines > titles.txt ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507525,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507525,IC_kwDOCGYnMM5bpIDF,9599,simonw,2023-05-07T18:09:09Z,2023-05-07T18:09:09Z,OWNER,"I'm tempted to upgrade `--raw` to do this instead, but that would be a breaking change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507394,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507394,IC_kwDOCGYnMM5bpIBC,9599,simonw,2023-05-07T18:08:44Z,2023-05-07T18:08:44Z,OWNER,"Prototype: ```diff diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 153e5f9..c830518 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -124,6 +124,7 @@ See :ref:`cli_query`. --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row + --raw-lines Raw output, first column of each row -p, --param ... Named :parameters for SQL query --functions TEXT Python code defining one or more custom SQL functions @@ -192,6 +193,7 @@ See :ref:`cli_memory`. --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row + --raw-lines Raw output, first column of each row -p, --param ... Named :parameters for SQL query --encoding TEXT Character encoding for CSV input, defaults to utf-8 diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index d25b1df..da0e4b6 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -1653,6 +1653,7 @@ def drop_view(path, view, ignore, load_extension): ) @output_options @click.option(""-r"", ""--raw"", is_flag=True, help=""Raw output, first column of first row"") +@click.option(""--raw-lines"", is_flag=True, help=""Raw output, first column of each row"") @click.option( ""-p"", ""--param"", @@ -1677,6 +1678,7 @@ def query( fmt, json_cols, raw, + raw_lines, param, load_extension, functions, @@ -1700,7 +1702,19 @@ def query( _register_functions(db, functions) _execute_query( - db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols + db, + sql, + param, + raw, + raw_lines, + table, + csv, + tsv, + no_headers, + fmt, + nl, + arrays, + json_cols, ) @@ -1728,6 +1742,7 @@ def query( ) @output_options @click.option(""-r"", ""--raw"", is_flag=True, help=""Raw output, first column of first row"") +@click.option(""--raw-lines"", is_flag=True, help=""Raw output, first column of each row"") @click.option( ""-p"", ""--param"", @@ -1773,6 +1788,7 @@ def memory( fmt, json_cols, raw, + raw_lines, param, encoding, no_detect_types, @@ -1879,12 +1895,36 @@ def memory( _register_functions(db, functions) _execute_query( - db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols + db, + sql, + param, + raw, + raw_lines, + table, + csv, + tsv, + no_headers, + fmt, + nl, + arrays, + json_cols, ) def _execute_query( - db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols + db, + sql, + param, + raw, + raw_lines, + table, + csv, + tsv, + no_headers, + fmt, + nl, + arrays, + json_cols, ): with db.conn: try: @@ -1903,6 +1943,13 @@ def _execute_query( sys.stdout.buffer.write(data) else: sys.stdout.write(str(data)) + elif raw_lines: + for row in cursor: + data = row[0] + if isinstance(data, bytes): + sys.stdout.buffer.write(data + b""\n"") + else: + sys.stdout.write(str(data) + ""\n"") elif fmt or table: print( tabulate.tabulate( ``` Needs tests and more documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/datasette/issues/2065#issuecomment-1524709988,https://api.github.com/repos/simonw/datasette/issues/2065,1524709988,IC_kwDOBm6k_c5a4Tpk,9599,simonw,2023-04-27T05:09:36Z,2023-04-27T05:09:36Z,OWNER,"That fixed it - after installing `main.zip` again I ran this and it worked: ~/.rye/tools/main-zip/bin/datasette install datasette-graphql ``` % ~/.rye/tools/main-zip/bin/datasette plugins [ { ""name"": ""datasette-graphql"", ""static"": true, ""templates"": true, ""version"": ""2.2"", ""hooks"": [ ""database_actions"", ""extra_template_vars"", ""menu_links"", ""register_routes"", ""startup"", ""table_actions"" ] } ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686033652,Datasette cannot be installed with Rye, https://github.com/simonw/datasette/issues/2065#issuecomment-1524707628,https://api.github.com/repos/simonw/datasette/issues/2065,1524707628,IC_kwDOBm6k_c5a4TEs,9599,simonw,2023-04-27T05:06:44Z,2023-04-27T05:06:44Z,OWNER,"I need `pip` as a dependency too: ``` % ~/.rye/tools/main-zip/bin/datasette install datasette-graphql Traceback (most recent call last): File ""/Users/simon/.rye/tools/main-zip/bin/datasette"", line 8, in sys.exit(cli()) ^^^^^ File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1130, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1055, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 760, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/datasette/cli.py"", line 365, in install run_module(""pip"", run_name=""__main__"") File """", line 222, in run_module File """", line 142, in _get_module_details ImportError: No module named pip ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686033652,Datasette cannot be installed with Rye, https://github.com/simonw/datasette/issues/2065#issuecomment-1524699863,https://api.github.com/repos/simonw/datasette/issues/2065,1524699863,IC_kwDOBm6k_c5a4RLX,9599,simonw,2023-04-27T04:56:22Z,2023-04-27T04:56:22Z,OWNER,Turned this into a TIL: https://til.simonwillison.net/python/rye,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686033652,Datasette cannot be installed with Rye, https://github.com/simonw/datasette/issues/2066#issuecomment-1524680160,https://api.github.com/repos/simonw/datasette/issues/2066,1524680160,IC_kwDOBm6k_c5a4MXg,9599,simonw,2023-04-27T04:27:50Z,2023-04-27T04:27:50Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524675817,https://api.github.com/repos/simonw/datasette/issues/2066,1524675817,IC_kwDOBm6k_c5a4LTp,9599,simonw,2023-04-27T04:21:03Z,2023-04-27T04:21:03Z,OWNER,I went with this to generate the long string: https://github.com/simonw/datasette/blob/0b0c5cd7a94fe3f151a3e10261b5c84ee64f2f18/tests/test_csv.py#L157-L176,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524669124,https://api.github.com/repos/simonw/datasette/issues/2066,1524669124,IC_kwDOBm6k_c5a4JrE,9599,simonw,2023-04-27T04:10:44Z,2023-04-27T04:10:52Z,OWNER,"I need an alternative way of generating a long string with a shorter URL. ```sql select group_concat('abcabcabc', '') from json_each(json_array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)) ``` https://latest.datasette.io/_memory?sql=select+group_concat%28%27abcabcabc%27%2C+%27%27%29+from+json_each%28json_array%281%2C+2%2C+3%2C+4%2C+5%2C+6%2C+7%2C+8%2C+9%2C+10%29%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524666049,https://api.github.com/repos/simonw/datasette/issues/2066,1524666049,IC_kwDOBm6k_c5a4I7B,9599,simonw,2023-04-27T04:06:18Z,2023-04-27T04:06:18Z,OWNER,"Most recent `httpx` release is 0.24 a couple of weeks ago. Here's what changed in that version: https://github.com/encode/httpx/compare/0.23.3...0.24.0 It looks like that URL limit is new: https://github.com/encode/httpx/commit/57daabf673705954afa94686c0002801c93d31f3#diff-78d8d93b5dd4c77d99c3e2b46b7286ba71a8fd60e92d8bd4eee5fb200b4f87bfR149-R155 ```python def urlparse(url: str = """", **kwargs: typing.Optional[str]) -> ParseResult: # Initial basic checks on allowable URLs. # --------------------------------------- # Hard limit the maximum allowable URL length. if len(url) > MAX_URL_LENGTH: raise InvalidURL(""URL too long"") ``` https://github.com/encode/httpx/blob/32e25497a36e6222cc64a758c98275b450dac28d/httpx/_urlparse.py#L153-L155","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524660603,https://api.github.com/repos/simonw/datasette/issues/2066,1524660603,IC_kwDOBm6k_c5a4Hl7,9599,simonw,2023-04-27T04:02:55Z,2023-04-27T04:02:55Z,OWNER,"In the debugger: ``` (Pdb) MAX_URL_LENGTH 65536 ``` Weird this only seems to be a problem with `httpx` on Python 3.7 though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524659084,https://api.github.com/repos/simonw/datasette/issues/2066,1524659084,IC_kwDOBm6k_c5a4HOM,9599,simonw,2023-04-27T04:02:07Z,2023-04-27T04:02:07Z,OWNER,"This is the failing test: https://github.com/simonw/datasette/blob/249fcf8e3e2a90e763f41b080c1b9ec8017f5005/tests/test_csv.py#L156-L167","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524655203,https://api.github.com/repos/simonw/datasette/issues/2066,1524655203,IC_kwDOBm6k_c5a4GRj,9599,simonw,2023-04-27T03:59:56Z,2023-04-27T03:59:56Z,OWNER,"In a fresh Datasette checkout I ran: pipenv shell --python /Users/simon/.pyenv/versions/3.7.16/bin/python That gave me a virtual environment with 3.7.16 Python. Then I ran: pip install -e '.[test]' Weirdly that gave me a `which pytest` of `/opt/homebrew/bin/pytest` which ran the tests on 3.11. I figured out the location of the virtual environment with `which python` and then ran this: ``` % /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/bin/pytest tests/test_csv.py ============================================================================================== test session starts =============================================================================================== platform darwin -- Python 3.7.16, pytest-7.3.1, pluggy-1.0.0 SQLite: 3.39.5 rootdir: /private/tmp/datasette configfile: pytest.ini plugins: asyncio-0.21.0, timeout-2.1.0, xdist-3.2.1, anyio-3.6.2 asyncio: mode=strict collected 15 items tests/test_csv.py ..........F.... [100%] ==================================================================================================== FAILURES ==================================================================================================== ________________________________________________________________________________________________ test_max_csv_mb _________________________________________________________________________________________________ app_client_csv_max_mb_one = def test_max_csv_mb(app_client_csv_max_mb_one): response = app_client_csv_max_mb_one.get( ( ""/fixtures.csv?sql=select+'{}'+"" ""from+compound_three_primary_keys&_stream=1&_size=max"" > ).format(""abcdefg"" * 10000) ) /private/tmp/datasette/tests/test_csv.py:161: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/asgiref/sync.py:240: in __call__ return call_result.result() /Users/simon/.pyenv/versions/3.7.16/lib/python3.7/concurrent/futures/_base.py:428: in result return self.__get_result() /Users/simon/.pyenv/versions/3.7.16/lib/python3.7/concurrent/futures/_base.py:384: in __get_result raise self._exception /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/asgiref/sync.py:306: in main_wrap result = await self.awaitable(*args, **kwargs) /private/tmp/datasette/datasette/utils/testing.py:76: in get headers=headers, /private/tmp/datasette/datasette/utils/testing.py:167: in _request content=post_body, /private/tmp/datasette/datasette/app.py:1787: in request method, self._fix(path, avoid_path_rewrites), **kwargs /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:1528: in request extensions=extensions, /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:346: in build_request url = self._merge_url(url) /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:376: in _merge_url merge_url = URL(url) /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_urls.py:113: in __init__ self._uri_reference = urlparse(url, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ url = ""http://localhost/fixtures.csv?sql=select+'abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcde...gabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg'+from+compound_three_primary_keys&_stream=1&_size=max"" kwargs = {} def urlparse(url: str = """", **kwargs: typing.Optional[str]) -> ParseResult: # Initial basic checks on allowable URLs. # --------------------------------------- # Hard limit the maximum allowable URL length. if len(url) > MAX_URL_LENGTH: > raise InvalidURL(""URL too long"") E httpx.InvalidURL: URL too long /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_urlparse.py:155: InvalidURL ============================================================================================ short test summary info ============================================================================================= FAILED tests/test_csv.py::test_max_csv_mb - httpx.InvalidURL: URL too long ========================================================================================== 1 failed, 14 passed in 1.83s ========================================================================================== (datasette) simon@Simons-MacBook-Pro datasette % ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524648995,https://api.github.com/repos/simonw/datasette/issues/2066,1524648995,IC_kwDOBm6k_c5a4Ewj,9599,simonw,2023-04-27T03:56:42Z,2023-04-27T03:57:11Z,OWNER,"I don't have 3.7 locally. Trying to install it with `pyenv`. brew install pyenv Then: pyenv install --list | grep 3.7 Installing: pyenv install 3.7.16 Output: Installed Python-3.7.16 to /Users/simon/.pyenv/versions/3.7.16 So the executable is `/Users/simon/.pyenv/versions/3.7.16/bin/python`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524638233,https://api.github.com/repos/simonw/datasette/issues/2066,1524638233,IC_kwDOBm6k_c5a4CIZ,9599,simonw,2023-04-27T03:50:51Z,2023-04-27T03:50:51Z,OWNER,Failure was on 3.7. I'll try that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524637376,https://api.github.com/repos/simonw/datasette/issues/2066,1524637376,IC_kwDOBm6k_c5a4B7A,9599,simonw,2023-04-27T03:50:19Z,2023-04-27T03:50:19Z,OWNER,"Having trouble replicating this on my laptop. I tried a fresh virtual environment with fresh packages (in case this is a `httpx` change) but this passed: pytest tests/test_csv.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2065#issuecomment-1524616740,https://api.github.com/repos/simonw/datasette/issues/2065,1524616740,IC_kwDOBm6k_c5a384k,9599,simonw,2023-04-27T03:38:21Z,2023-04-27T03:38:21Z,OWNER,"Tried this: rye install https://github.com/simonw/datasette/archive/refs/heads/main.zip But got this error: Error: Expected one of `@`, `(`, `<`, `=`, `>`, `~`, `!`, `;`, found `:` I instead downloaded that file and ran: rye install main.zip This worked! And now: ``` ~/.rye/tools/main-zip/bin/datasette --version ``` ``` datasette, version 1.0a2 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686033652,Datasette cannot be installed with Rye, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1506223848,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1506223848,IC_kwDOCGYnMM5Zxybo,9599,simonw,2023-04-13T02:08:16Z,2023-04-13T02:08:16Z,OWNER,"I agree, this is a design flaw. It's technically a breaking change. As such, I would need to bump to v4 to responsibly release this. I'd rather bundle in a few more breaking changes before shipping that version. One thing we could do here is add an extra argument to `.convert()` - something like this: ```python table.convert(col, lambda x: x+1, skip_false=False) ``` This would trigger the new, improved behaviour without breaking existing code that depends on how it works at the moment. Then in `sqlite-utils` 4 we can change the default of that option. What do you think? (I'm open to suggestions for better names for this parameter too)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/datasette/pull/2056#issuecomment-1506179555,https://api.github.com/repos/simonw/datasette/issues/2056,1506179555,IC_kwDOBm6k_c5Zxnnj,9599,simonw,2023-04-13T01:21:05Z,2023-04-13T01:22:08Z,OWNER,"OK, I'm sold - this is a really neat improvement. One thing to change in the PR: right now it runs `pip install --user ruff` on every commit, which hits PyPI to install the package. I prefer to avoid hitting PyPI every time, so I like to use the GitHub Actions cache. My usual pattern for that looks like this: https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/.github/workflows/test-pyodide.yml#L16-L20 Then a separate command that runs `pip install ...` will benefit from that cache. Are you OK to make that change?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff, https://github.com/simonw/datasette/pull/2056#issuecomment-1506177857,https://api.github.com/repos/simonw/datasette/issues/2056,1506177857,IC_kwDOBm6k_c5ZxnNB,9599,simonw,2023-04-13T01:18:18Z,2023-04-13T01:18:18Z,OWNER,"Cool - and now https://github.com/simonw/datasette/pull/2056/files is showing me those inline annotations: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff, https://github.com/simonw/datasette/pull/2056#issuecomment-1506177115,https://api.github.com/repos/simonw/datasette/issues/2056,1506177115,IC_kwDOBm6k_c5ZxnBb,9599,simonw,2023-04-13T01:17:16Z,2023-04-13T01:17:16Z,OWNER,"Here are the failures: https://github.com/simonw/datasette/actions/runs/4684460653/jobs/8300630794?pr=2056 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff, https://github.com/simonw/datasette/pull/2056#issuecomment-1506175208,https://api.github.com/repos/simonw/datasette/issues/2056,1506175208,IC_kwDOBm6k_c5Zxmjo,9599,simonw,2023-04-13T01:14:13Z,2023-04-13T01:14:13Z,OWNER,https://github.com/simonw/datasette/actions/runs/4664796647/jobs/8300596121?pr=2056 it's pretty fast - that finished in 9s.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff, https://github.com/simonw/datasette/issues/2059#issuecomment-1506174353,https://api.github.com/repos/simonw/datasette/issues/2059,1506174353,IC_kwDOBm6k_c5ZxmWR,9599,simonw,2023-04-13T01:13:00Z,2023-04-13T01:13:00Z,OWNER,"Can you provide a URL to an example, and/or a screenshot of this? Is it a browser warning or is it a warning from Heroku itself?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665053646,"""Deceptive site ahead"" alert on Heroku deployment", https://github.com/simonw/datasette/issues/2058#issuecomment-1504426792,https://api.github.com/repos/simonw/datasette/issues/2058,1504426792,IC_kwDOBm6k_c5Zq7so,9599,simonw,2023-04-12T02:02:42Z,2023-04-12T02:02:42Z,OWNER,"I tightened up the benchmark (it was measuring the time taken to create the tables too) and got this: ![image](https://user-images.githubusercontent.com/9599/231328328-85ca35ac-a11b-46f4-b132-dae367103570.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504328395,https://api.github.com/repos/simonw/datasette/issues/2058,1504328395,IC_kwDOBm6k_c5ZqjrL,9599,simonw,2023-04-12T00:28:38Z,2023-04-12T00:28:38Z,OWNER,"Here's a much better chart, which shows that MD5 performance unsurprisingly gets worse as the number of tables increases while `schema_version` remains constant: ![image](https://user-images.githubusercontent.com/9599/231316778-513bd99f-5ea4-495c-b86d-c572a7106369.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504315697,https://api.github.com/repos/simonw/datasette/issues/2058,1504315697,IC_kwDOBm6k_c5Zqgkx,9599,simonw,2023-04-12T00:16:22Z,2023-04-12T00:27:12Z,OWNER,"I got ChatGPT (code execution alpha) to run a micro-benchmark for me. This was the conclusion: > The benchmark using `PRAGMA schema_version` is approximately 1.36 times faster than the benchmark using `hashlib.md5` for the case with 100 tables. For the case with 200 tables, the benchmark using `PRAGMA schema_version` is approximately 2.33 times faster than the benchmark using `hashlib.md5`. Here's the chart it drew me: ![image](https://user-images.githubusercontent.com/9599/231315366-3a12b6d3-08d7-419d-a1fd-36eb24da0d85.png) (It's a pretty rubbish chart though, it only took measurements at 100 and 200 and drew a line between the two, I should have told it to measure every 10 and plot that) And the full transcript: https://gist.github.com/simonw/2fc46effbfbe49e6de0bcfdc9e31b235 The benchmark looks good enough on first glance that I don't feel the need to be more thorough with it. `PRAGMA schema_version` is faster, but not so fast that I feel like the MD5 hack is worth worrying about too much. I'm tempted to add something to the `/-/versions` page that tries to identify if this is a problem or not though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504298448,https://api.github.com/repos/simonw/datasette/issues/2058,1504298448,IC_kwDOBm6k_c5ZqcXQ,9599,simonw,2023-04-12T00:04:01Z,2023-04-12T00:04:01Z,OWNER,"Here's a potential workaround: when I store the schema versions, I could also score an MD5 hash of the full schema (`select group_concat(sql) from sqlite_master`). When I read the schema version with `PRAGMA schema_version` I could catch that exception and, if I see it, I could calculate that MD5 hash again as a fallback and use that to determine if the schema has changed instead. The performance overhead of this needs investigating - how much more expensive is `md5(... that SQL query result)` compared to just `PRAGMA schema_version`, especially on a database with a lot of tables?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504295345,https://api.github.com/repos/simonw/datasette/issues/2058,1504295345,IC_kwDOBm6k_c5Zqbmx,9599,simonw,2023-04-12T00:01:42Z,2023-04-12T00:02:26Z,OWNER,"Here's the relevant code: https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/app.py#L421-L437 This function is called on almost every request (everything that subclasses `BaseView` at least - need to remember that for the refactor in #2053 etc). https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/views/base.py#L101-L103 It uses `PRAGMA schema_version` as a cheap way to determine if the schema has changed, in which case it needs to refresh the internal schema tables. This was already the cause of a subtle bug here: - #1231 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504292145,https://api.github.com/repos/simonw/datasette/issues/2058,1504292145,IC_kwDOBm6k_c5Zqa0x,9599,simonw,2023-04-11T23:58:59Z,2023-04-11T23:58:59Z,OWNER,Asked on the SQLite Forum if anyone has seen this before: https://sqlite.org/forum/forumpost/793a2ed75b,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504291892,https://api.github.com/repos/simonw/datasette/issues/2058,1504291892,IC_kwDOBm6k_c5Zqaw0,9599,simonw,2023-04-11T23:58:45Z,2023-04-11T23:58:45Z,OWNER,"I thought it might relate to the ""defensive mode"" issue described here: - https://github.com/simonw/sqlite-utils/issues/235 But I have since determined that the Datasette official Docker image does NOT run anything in defensive mode, so I don't think it's related to that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1504288134,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1504288134,IC_kwDOCGYnMM5ZqZ2G,9599,simonw,2023-04-11T23:55:06Z,2023-04-12T03:34:32Z,OWNER,"Also checked the official Datasette Docker image - I had to run that in Codespaces because it doesn't currently work on my M2 Mac: ``` codespace@codespaces-112c61:/workspaces/sqlite-utils$ docker pull datasetteproject/datasette Using default tag: latest ... codespace@codespaces-112c61:/workspaces/sqlite-utils$ docker run -it datasetteproject/datasette / bin/bash root@75ba34f501ec:/# python Python 3.11.0 (main, Dec 6 2022, 13:31:55) [GCC 10.2.1 20210110] on linux Type ""help"", ""copyright"", ""credits"" or ""license"" for more information. >>> import sqlite3 .executescript("""""" PRAGMA writable_schema = 1; UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)'; PRAGMA writable_schema = 0; """""")>>> db = sqlite3.connect("":memory:"") >>> db.executescript("""""" ... PRAGMA writable_schema = 1; ... UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)'; ... PRAGMA writable_schema = 0; ... """""") >>> ``` So that confirms that the official image also has a Python with a SQLite that's not in defensive mode.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1504245029,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1504245029,IC_kwDOCGYnMM5ZqPUl,9599,simonw,2023-04-11T23:13:41Z,2023-04-11T23:14:39Z,OWNER,"I also tested this against the current `ubuntu:latest` Docker image (on an M2 Mac), in Python 3.10 and 3.11: ``` docker run -it ubuntu:latest /bin/bash ``` Then in the container: ``` apt-get update apt-get install python3 python3 # pasted in the above recipe apt install software-properties-common add-apt-repository ppa:deadsnakes/ppa apt install python3.11 python3.11 # pasted it in again ``` In both cases the Python code did not raise an exception, which suggests that on Ubuntu those two Python versions do not have the defensive mode set.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/datasette/issues/2057#issuecomment-1503838640,https://api.github.com/repos/simonw/datasette/issues/2057,1503838640,IC_kwDOBm6k_c5ZosGw,9599,simonw,2023-04-11T17:48:23Z,2023-04-11T17:48:23Z,OWNER,"> This looks wrong to me - I would expect something like `is_directory()` not `is_file()` for telling if `static/` is a directory. I was right about that: ```pycon >>> importlib.resources.files('datasette_graphql') PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql') >>> importlib.resources.files('datasette_graphql').joinpath(""static"") PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql/static') >>> p = importlib.resources.files('datasette_graphql').joinpath(""static"") >>> p PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql/static') >>> p.is_ p.is_absolute() p.is_char_device() p.is_fifo() p.is_mount() p.is_reserved() p.is_symlink() p.is_block_device() p.is_dir() p.is_file() p.is_relative_to( p.is_socket() >>> p.is_dir() True >>> p.is_file() False ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1503833906,https://api.github.com/repos/simonw/datasette/issues/2057,1503833906,IC_kwDOBm6k_c5Zoq8y,9599,simonw,2023-04-11T17:44:16Z,2023-04-11T17:45:45Z,OWNER,"Another prompt: > How to fix this: > > `pkg_resources.get_distribution(package).version` Response: ```python import importlib.metadata # Get the version number of the specified package package_version = importlib.metadata.version(package) ``` That seems to work: ```pycon >>> import importlib.metadata >>> importlib.metadata.version(""datasette"") '0.64.2' >>> importlib.metadata.version(""pluggy"") '1.0.0' >>> importlib.metadata.version(""not-a-package"") ... importlib.metadata.PackageNotFoundError: No package metadata was found for not-a-package ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1503832422,https://api.github.com/repos/simonw/datasette/issues/2057,1503832422,IC_kwDOBm6k_c5Zoqlm,9599,simonw,2023-04-11T17:42:57Z,2023-04-11T17:46:42Z,OWNER,"I ran this prompt against ChatGPT with the Browsing alpha: > ```python > if pkg_resources.resource_isdir(plugin.__name__, ""static""): > static_path = pkg_resources.resource_filename( > plugin.__name__, ""static"" > ) > if pkg_resources.resource_isdir(plugin.__name__, ""templates""): > templates_path = pkg_resources.resource_filename( > plugin.__name__, ""templates"" > ) > ``` > This code gives a deprecation warning in Python 3.11 - fix it It looked up the fix for me: And suggested: ```python import importlib.resources # Replace pkg_resources.resource_isdir with importlib.resources.files().is_file() if importlib.resources.files(plugin.__name__).joinpath(""static"").is_file(): static_path = importlib.resources.as_file( importlib.resources.files(plugin.__name__).joinpath(""static"") ) if importlib.resources.files(plugin.__name__).joinpath(""templates"").is_file(): templates_path = importlib.resources.as_file( importlib.resources.files(plugin.__name__).joinpath(""templates"") ) ``` This looks wrong to me - I would expect something like `is_directory()` not `is_file()` for telling if `static/` is a directory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502559442,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1502559442,IC_kwDOCGYnMM5ZjzzS,9599,simonw,2023-04-11T01:32:30Z,2023-04-11T01:33:27Z,OWNER,"This seems to work: ```python import sqlite3 db = sqlite3.connect("":memory:"") db.executescript("""""" PRAGMA writable_schema = 1; UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)'; PRAGMA writable_schema = 0; """""") ``` It succeeds on my Python 3.11 and raises the following exception on my broken Python 3.9: ``` sqlite3.OperationalError: table sqlite_master may not be modified ``` Removing the `PRAGMA writable_schema = 1;` causes the same exception to be raised on both Pythons.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502557629,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1502557629,IC_kwDOCGYnMM5ZjzW9,9599,simonw,2023-04-11T01:30:12Z,2023-04-11T01:30:12Z,OWNER,"I'll ask on the SQLite forum if it's possible to toggle that mode on and off using regular SQL. My hunch is that it isn't. In which case `sqlite-utils` should at least know how to catch this error and display a much more readable error message, maybe with a link to further documentation. A utility function that can detect this mode would be really useful too. I'd probably have to do a test that tries to modify `sqlite_master` on a new in-memory database to catch if it's possible or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502556111,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1502556111,IC_kwDOCGYnMM5Zjy_P,9599,simonw,2023-04-11T01:28:41Z,2023-04-11T01:28:41Z,OWNER,"Investigating this one now. The `sqlite-utils` test suite passes without errors on my Python 3.11.2 installation... but it fails with this error on a Python 3.9.6 installation. In the broken version's virtual environment directory I ran this: ``` cat pyvenv.cfg home = /Applications/Xcode.app/Contents/Developer/usr/bin implementation = CPython version_info = 3.9.6.final.0 virtualenv = 20.17.1 include-system-site-packages = false base-prefix = /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9 base-exec-prefix = /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9 base-executable = /Applications/Xcode.app/Contents/Developer/usr/bin/python3 ``` So it looks like the Xcode `python3` has ""defensive"" mode turned on for SQLite. As far as I can tell there's no way to turn it OFF again in Python. My virtual environment that DOES work has this: ``` home = /opt/homebrew/opt/python@3.11/bin implementation = CPython version_info = 3.11.2.final.0 virtualenv = 20.17.1 include-system-site-packages = false base-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11 base-exec-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11 base-executable = /opt/homebrew/opt/python@3.11/bin/python3.11 ``` So the Python 3.11 I installed through Homebrew doesn't exhibit this bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/datasette/issues/2054#issuecomment-1500608101,https://api.github.com/repos/simonw/datasette/issues/2054,1500608101,IC_kwDOBm6k_c5ZcXZl,9599,simonw,2023-04-07T20:14:38Z,2023-04-07T20:14:38Z,OWNER,"Ooh that one's really interesting - very different from the others: ```ruby # app.rb require ""roda"" class App < Roda route do |r| r.root do ""Home page"" end r.on ""pages"" do r.get "":slug"" do |slug| ""Page: #{slug}"" end end r.on ""news"" do r.get "":yyyy/:mm/:dd"" do |yyyy, mm, dd| ""News for #{yyyy}/#{mm}/#{dd}"" end end end end # config.ru require_relative ""app"" run App.freeze.app ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499604822,https://api.github.com/repos/simonw/datasette/issues/2054,1499604822,IC_kwDOBm6k_c5ZYidW,9599,simonw,2023-04-06T20:48:19Z,2023-04-06T20:48:55Z,OWNER,"I actually quite like that. I could even use `@classmethod` and have utility methods defined on that class that both `get()` and `post()` could call. The crucial rule here is NO INSTANCE STATE - that's what makes routing to classes particularly damaging, and encourages code that's hard to maintain.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499604066,https://api.github.com/repos/simonw/datasette/issues/2054,1499604066,IC_kwDOBm6k_c5ZYiRi,9599,simonw,2023-04-06T20:47:30Z,2023-04-06T20:47:30Z,OWNER,"I'm contemplating a new approach: using a class with static methods. Something like this: ```python class TableView(MethodRouter): @staticmethod async def get(request): return Response.text(""GET"") @staticmethod async def post(request): return Response.text(""POST"") ``` So effectively the class is just there to bundle together verb implementations, and to provide a `route(request)` method which knows how to dispatch them to the right place. It can offer default HEAD and OPTIONS methods too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499596941,https://api.github.com/repos/simonw/datasette/issues/2054,1499596941,IC_kwDOBm6k_c5ZYgiN,9599,simonw,2023-04-06T20:39:35Z,2023-04-06T20:39:35Z,OWNER,"I think there are two main approaches: - Map path to a function, that function then behaves differently on GET/POST/etc - Map method AND path to a function - the `app.get('/example')` v.s. `app.post('/example')` ones The decorator approach is calling out to me a bit now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499530921,https://api.github.com/repos/simonw/datasette/issues/2054,1499530921,IC_kwDOBm6k_c5ZYQap,9599,simonw,2023-04-06T19:41:52Z,2023-04-06T19:42:09Z,OWNER,Turned this into a TIL: https://til.simonwillison.net/gpt3/gpt4-api-design,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499491258,https://api.github.com/repos/simonw/datasette/issues/2054,1499491258,IC_kwDOBm6k_c5ZYGu6,9599,simonw,2023-04-06T19:00:16Z,2023-04-06T20:40:16Z,OWNER,"I asked GPT-4 to compare URL routing based on path and verb approaches in different languages and frameworks and spit out some examples. Full transcript here: https://gist.github.com/simonw/fa2379b97420404a81b0fcdb4db79657 Initial prompt: > Show me code examples of different web frameworks in Python and JavaScript and Go illustrating how HTTP routing works - in particular the problem of mapping an incoming HTTP request to some code based on both the URL path and the HTTP verb > > Don't show installation instructions, just get straight to the code snippets Flask: ```python from flask import Flask app = Flask(__name__) @app.route('/example', methods=['GET']) def get_example(): return ""This is a GET request."" @app.route('/example', methods=['POST']) def post_example(): return ""This is a POST request."" if __name__ == '__main__': app.run() ``` Express: ```javascript const express = require('express'); const app = express(); app.get('/example', (req, res) => { res.send('This is a GET request.'); }); app.post('/example', (req, res) => { res.send('This is a POST request.'); }); app.listen(3000, () => { console.log('Server is running on port 3000'); }); ``` Gin: ```go package main import ( ""github.com/gin-gonic/gin"" ) func main() { r := gin.Default() r.GET(""/example"", func(c *gin.Context) { c.String(200, ""This is a GET request."") }) r.POST(""/example"", func(c *gin.Context) { c.String(200, ""This is a POST request."") }) r.Run() } ``` Actix-web: ```rust use actix_web::{web, App, HttpResponse, HttpServer, Responder}; async fn get_example() -> impl Responder { HttpResponse::Ok().body(""This is a GET request."") } async fn post_example() -> impl Responder { HttpResponse::Ok().body(""This is a POST request."") } #[actix_web::main] async fn main() -> std::io::Result<()> { HttpServer::new(|| { App::new() .route(""/example"", web::get().to(get_example)) .route(""/example"", web::post().to(post_example)) }) .bind(""127.0.0.1:8080"")? .run() .await } ``` Koa: ```javascript const Koa = require('koa'); const Router = require('koa-router'); const app = new Koa(); const router = new Router(); router.get('/example', (ctx) => { ctx.body = 'This is a GET request.'; }); router.post('/example', (ctx) => { ctx.body = 'This is a POST request.'; }); app.use(router.routes()); app.use(router.allowedMethods()); app.listen(3000, () => { console.log('Server is running on port 3000'); }); ``` Django: ```python from django.http import HttpResponse from django.views import View class ExampleView(View): def get(self, request): return HttpResponse(""This is a GET request."") def post(self, request): return HttpResponse(""This is a POST request."") ``` ```python from django.urls import path from .views import ExampleView urlpatterns = [ path('example/', ExampleView.as_view(), name='example'), ] ``` Rocket: ```rust #![feature(proc_macro_hygiene, decl_macro)] #[macro_use] extern crate rocket; use rocket::response::content; #[get(""/example"")] fn get_example() -> content::Plain<&'static str> { content::Plain(""This is a GET request."") } #[post(""/example"")] fn post_example() -> content::Plain<&'static str> { content::Plain(""This is a POST request."") } #[launch] fn rocket() -> _ { rocket::build().mount(""/"", routes![get_example, post_example]) } ``` Elixir/Phoenix: ```elixir defmodule ExampleAppWeb.Router do use ExampleAppWeb, :router pipeline :api do plug :accepts, [""json""] end scope ""/api"", ExampleAppWeb do pipe_through :api get ""/example"", ExampleController, :get_example post ""/example"", ExampleController, :post_example end end ``` Bottle: ```python from bottle import Bottle, run app = Bottle() @app.route('/example', method='GET') def get_example(): return ""This is a GET request."" @app.route('/example', method='POST') def post_example(): return ""This is a POST request."" if __name__ == '__main__': run(app, host='localhost', port=8080) ``` FastAPI: ```python from fastapi import FastAPI app = FastAPI() @app.get(""/example"") async def get_example(): return {""message"": ""This is a GET request.""} @app.post(""/example"") async def post_example(): return {""message"": ""This is a POST request.""} if __name__ == ""__main__"": import uvicorn uvicorn.run(app, host=""127.0.0.1"", port=8000) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499467703,https://api.github.com/repos/simonw/datasette/issues/2054,1499467703,IC_kwDOBm6k_c5ZYA-3,9599,simonw,2023-04-06T18:37:22Z,2023-04-06T18:57:22Z,OWNER,"I've been hoping to move entirely away from class-based views, but now I'm wondering if that's the right decision given the need to support HTTP verbs. It is cleaner to have a class that has methods for each verb, or to have a single function that can behave differently depending on the verb? Or should I have a mechanism for dispatching to separate functions based on the verbs that isn't implemented in a base class? The trouble with using base classes is that I've already shown that they tempt me to put logic in weird places, which makes code harder to modify later on. The thing here is that the ONLY thing I want to use the base class for is as an HTTP verb switcher - which makes me doubt that I should risk taking on the other temptations of having a base class.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499465648,https://api.github.com/repos/simonw/datasette/issues/2054,1499465648,IC_kwDOBm6k_c5ZYAew,9599,simonw,2023-04-06T18:35:03Z,2023-04-06T18:35:03Z,OWNER,"There are actually five classes that subclass `DataView`: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/row.py#L16 https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/database.py#L34 https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/database.py#L172 https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/database.py#L215 https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/table.py#L72 I don't think `DatabaseView` and `DatabaseDownload` should have subclassed that at all, since they don't return a table of data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499462324,https://api.github.com/repos/simonw/datasette/issues/2054,1499462324,IC_kwDOBm6k_c5ZX_q0,9599,simonw,2023-04-06T18:31:56Z,2023-04-06T18:31:56Z,OWNER,"The `DataView` class does a LOT of work - mostly involving CSV responses. https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L160-L544 It has a `redirect()` method with some complex logic and CORS handling: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L163-L172 It uses this method a lot, which has to be over-ridden in the `TableView` and `RowView` and `QueryView` subclasses: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L174-L175 This method: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L180 Is the bulk of the complexity, because it knows how to both turn a list of SQLite rows into a CSV file but also knows how to call `.data()` repeatedly with different pagination arguments in order to stream CSV back for a large table. The `async def get()` method for GET requests is also very complicated. It mainly handles format stuff - knowing how to render HTML v.s. JSON v.s. CSV v.s. other formats specified using this plugin hook: https://docs.datasette.io/en/1.0a2/plugin_hooks.html#register-output-renderer-datasette Plus it catches interrupted queries and returns a special error page for those (and other error messages too): https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L381-L408 It adds the time taken to execute the queries: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L410-L411","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499457291,https://api.github.com/repos/simonw/datasette/issues/2054,1499457291,IC_kwDOBm6k_c5ZX-cL,9599,simonw,2023-04-06T18:26:45Z,2023-04-06T18:26:45Z,OWNER," Here's `BaseView`: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L56-L145 It has methods for the `options`, `get`, `post`, `delete`, `put`, `patch` and `head` HTTP verbs, most defaulting to returinng a 405 Method not allowed message in plain text or JSON, depending on this check: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L71-L81 Also adds CORS headers to anything if CORS mode is on: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L106-L107 And adds the `database_color` (weirdly) and the `select_templates` variables to the template context: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L112-L122 And has special code for setting the `Link: ...; rel=""alternate""` HTTP header: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L124-L136","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499457201,https://api.github.com/repos/simonw/datasette/issues/2054,1499457201,IC_kwDOBm6k_c5ZX-ax,9599,simonw,2023-04-06T18:26:39Z,2023-04-06T18:26:39Z,OWNER,These classes - `TableView` and `RowView` and `QueryView` - all subclass `DataView` which subclasses` BaseView`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499452122,https://api.github.com/repos/simonw/datasette/issues/2054,1499452122,IC_kwDOBm6k_c5ZX9La,9599,simonw,2023-04-06T18:21:51Z,2023-04-06T18:21:51Z,OWNER,"I'm going to make notes against the code in the most recent alpha release, ignoring the recent work I did to refactor `TableView`. https://github.com/simonw/datasette/tree/1.0a2/datasette/views","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/pull/2053#issuecomment-1498279469,https://api.github.com/repos/simonw/datasette/issues/2053,1498279469,IC_kwDOBm6k_c5ZTe4t,9599,simonw,2023-04-05T23:28:53Z,2023-04-05T23:28:53Z,OWNER,"Table errors page currently does this: ```json { ""ok"": false, ""error"": ""no such column: blah"", ""status"": 400, ""title"": ""Invalid SQL"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/issues/2049#issuecomment-1498275621,https://api.github.com/repos/simonw/datasette/issues/2049,1498275621,IC_kwDOBm6k_c5ZTd8l,9599,simonw,2023-04-05T23:23:01Z,2023-04-05T23:23:01Z,OWNER,"The default representation here can be even smaller. For rows it's this: ```json { ""ok"": true, ""next"": ""d,v"", ""rows"": [...] } ``` For SQL queries I'm considering this: ```json { ""ok"": true, ""rows"": [...] } ``` I considered adding `""sql""` and `""params""` too, but on further thought those would be entirely a waste of bytes the majority of the time. If a user wants those they can request them with an `?_extra=query` as seen here: http://localhost:8001/content/releases.json?_size=0&_extra=query ```json { ""ok"": true, ""next"": null, ""query"": { ""sql"": ""select html_url, id, author, node_id, tag_name, target_commitish, name, draft, prerelease, created_at, published_at, body, repo, reactions, mentions_count from releases order by id limit 1"", ""params"": {} }, ""rows"": [] } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1646734246,Custom SQL queries should use new JSON ?_extra= format, https://github.com/simonw/datasette/issues/2050#issuecomment-1492387771,https://api.github.com/repos/simonw/datasette/issues/2050,1492387771,IC_kwDOBm6k_c5Y9Ae7,9599,simonw,2023-03-31T17:59:48Z,2023-03-31T17:59:48Z,OWNER,"Some of the extras from the table view make sense here. A few custom ones make sense too - including this already existing but undocumented one: https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/views/row.py#L86-L89","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1649791661,Row page JSON should use new ?_extra= format, https://github.com/simonw/datasette/issues/2035#issuecomment-1492206593,https://api.github.com/repos/simonw/datasette/issues/2035,1492206593,IC_kwDOBm6k_c5Y8UQB,9599,simonw,2023-03-31T16:09:08Z,2023-03-31T16:09:08Z,OWNER,"I could ship this as part of: - #2049 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2049#issuecomment-1489530555,https://api.github.com/repos/simonw/datasette/issues/2049,1489530555,IC_kwDOBm6k_c5YyG67,9599,simonw,2023-03-30T00:51:27Z,2023-03-30T00:51:27Z,OWNER,"I'd really like to refactor all of the extras functions into a `datasette/extras.py` module. The table ones currently rely a LOT on local variables in scope though, so I would need to rewrite those such that EVERY dependency they take is passed to `asyncinject` explicitly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1646734246,Custom SQL queries should use new JSON ?_extra= format, https://github.com/simonw/datasette/issues/2049#issuecomment-1489530037,https://api.github.com/repos/simonw/datasette/issues/2049,1489530037,IC_kwDOBm6k_c5YyGy1,9599,simonw,2023-03-30T00:50:30Z,2023-03-30T00:50:30Z,OWNER,"Two things to consider here: `_shape=` and `_extra=`. Most of the shapes make sense, with the exception of `?_shape=object` since we don't know which column we would use as a primary key. Looking at the (undocumented) list of extras from the table view, here are the ones I think make sense: - `count` - YES - `facet_results` - no - `facets_timed_out` - no - `suggested_facets` - no - `human_description_en` - no - `next_url` - MAYBE - `columns` - YES - `primary_keys` - no - `display_columns` - YES - `display_rows` - YES - `debug` - YES? - `request` - YES - `query` - YES - `metadata` - YES - `extras` - YES - `database` - YES - `table` - no - `database_color` - no? - `table_actions` - no - `filters` - no - `renderers` - YES - `custom_table_templates` - no - `sorted_facet_results` - no - `table_definition` - no - `view_definition` - no - `is_view` - no - `private` - YES - `expandable_columns` - no - `form_hidden_args` - no Just the YES ones: - `count` - this is new - `columns` - `display_columns` - `display_rows` - `debug` - `request` - `query` - `metadata` - `extras` - `database` - `renderers` - `private` The `count` one is interesting - I think I can provide that by optionally running `select count(*) from (inner query)`. It's a new feature though and not one I want to expose on the HTML view since it could result in poor performance - but having it as an extra that API users can opt into may make sense.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1646734246,Custom SQL queries should use new JSON ?_extra= format, https://github.com/simonw/datasette/issues/2049#issuecomment-1489526501,https://api.github.com/repos/simonw/datasette/issues/2049,1489526501,IC_kwDOBm6k_c5YyF7l,9599,simonw,2023-03-30T00:44:05Z,2023-03-30T00:44:05Z,OWNER,"As part of this I should be able to figure out which bits of the new code I wrote for the table view should actually be shared with the query view. That stuff is mostly going to be from this commit: https://github.com/simonw/datasette/commit/d97e82df3c8a3f2e97038d7080167be9bb74a68d Here's the existing QueryView class I need to replace: https://github.com/simonw/datasette/blob/4c1e277edbd783d06840d3f9b20bf00783478ce4/datasette/views/database.py#L215-L532","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1646734246,Custom SQL queries should use new JSON ?_extra= format, https://github.com/simonw/datasette/issues/262#issuecomment-1488010837,https://api.github.com/repos/simonw/datasette/issues/262,1488010837,IC_kwDOBm6k_c5YsT5V,9599,simonw,2023-03-29T06:22:21Z,2023-03-29T06:22:21Z,OWNER,I need to get the arbitrary query page to return the same format. It likely won't have nearly as many extras.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/pull/2014#issuecomment-1487998788,https://api.github.com/repos/simonw/datasette/issues/2014,1487998788,IC_kwDOBm6k_c5YsQ9E,9599,simonw,2023-03-29T06:08:23Z,2023-03-29T06:08:23Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1566081801,Bump black from 22.12.0 to 23.1.0, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484277416,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1484277416,IC_kwDOCGYnMM5YeEao,9599,simonw,2023-03-26T23:43:12Z,2023-03-26T23:43:12Z,OWNER,Fixed: https://sqlite-utils.datasette.io/en/latest/reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/datasette/issues/1608#issuecomment-1484276946,https://api.github.com/repos/simonw/datasette/issues/1608,1484276946,IC_kwDOBm6k_c5YeETS,9599,simonw,2023-03-26T23:41:11Z,2023-03-26T23:41:11Z,OWNER,It's working again now: https://docs.datasette.io/en/latest/sql_queries.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484276302,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1484276302,IC_kwDOCGYnMM5YeEJO,9599,simonw,2023-03-26T23:39:27Z,2023-03-26T23:39:27Z,OWNER,"Actually final Datasette fix was: ```html+jinja {% block scripts %} {{ super() }} {% endblock %} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/datasette/issues/1608#issuecomment-1484274796,https://api.github.com/repos/simonw/datasette/issues/1608,1484274796,IC_kwDOBm6k_c5YeDxs,9599,simonw,2023-03-26T23:38:18Z,2023-03-26T23:38:18Z,OWNER,"ChatGPT prompt: > ``` > jQuery(function ($) { > // Show banner linking to /stable/ if this is a /latest/ page > if (!/\/latest\//.test(location.pathname)) { > return; > } > var stableUrl = location.pathname.replace(""/latest/"", ""/stable/""); > // Check it's not a 404 > fetch(stableUrl, { method: ""HEAD"" }).then((response) => { > if (response.status == 200) { > var warning = $( > `
>

Note

>

> This documentation covers the development version of Datasette.

>

See this page for the current stable release. >

>
` > ); > warning.find(""a"").attr(""href"", stableUrl); > $(""article[role=main]"").prepend(warning); > } > }); > }); > > Rewrite this to use vanilla JavaScript, no jQuery > ``` Got back: ```javascript document.addEventListener(""DOMContentLoaded"", function() { // Show banner linking to /stable/ if this is a /latest/ page if (!/\/latest\//.test(location.pathname)) { return; } var stableUrl = location.pathname.replace(""/latest/"", ""/stable/""); // Check it's not a 404 fetch(stableUrl, { method: ""HEAD"" }).then((response) => { if (response.status === 200) { var warning = document.createElement(""div""); warning.className = ""admonition warning""; warning.innerHTML = `

Note

This documentation covers the development version of Datasette.

See this page for the current stable release.

`; var mainArticle = document.querySelector(""article[role=main]""); mainArticle.insertBefore(warning, mainArticle.firstChild); } }); }); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1484272313,https://api.github.com/repos/simonw/datasette/issues/1608,1484272313,IC_kwDOBm6k_c5YeDK5,9599,simonw,2023-03-26T23:36:17Z,2023-03-26T23:36:17Z,OWNER,"Still getting this error: Weird because `jQuery` is present in the devtools console.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/1608#issuecomment-1484270067,https://api.github.com/repos/simonw/datasette/issues/1608,1484270067,IC_kwDOBm6k_c5YeCnz,9599,simonw,2023-03-26T23:26:10Z,2023-03-26T23:27:42Z,OWNER,"That didn't work, because `jQuery` isn't yet defined that high up the template. I should use `{% block scripts %}` instead: https://github.com/pradyunsg/furo/blob/05350110cf02d033ebb57147eb54e9371a63545a/src/furo/theme/furo/base.html#L89-L102","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484269326,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1484269326,IC_kwDOCGYnMM5YeCcO,9599,simonw,2023-03-26T23:24:12Z,2023-03-26T23:28:14Z,OWNER,"This broke on the upgrade to Furo. I fixed it in Datasette like this: https://github.com/simonw/datasette/commit/5c1cfa451d78e3935193f5e10eba59bf741241a1 and https://github.com/simonw/datasette/commit/db8cf899e286fbaa0a40f3a9ae8d5aaa1478822e","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/datasette/issues/1608#issuecomment-1484268146,https://api.github.com/repos/simonw/datasette/issues/1608,1484268146,IC_kwDOBm6k_c5YeCJy,9599,simonw,2023-03-26T23:23:22Z,2023-03-26T23:23:22Z,OWNER,This broke when I upgraded to the Furo theme.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/, https://github.com/simonw/datasette/issues/262#issuecomment-1480355670,https://api.github.com/repos/simonw/datasette/issues/262,1480355670,IC_kwDOBm6k_c5YPG9W,9599,simonw,2023-03-22T22:50:30Z,2023-03-22T22:50:30Z,OWNER,"I just landed this PR so this feature is now in `main`: - #1999 Still needs documentation and maybe some extra tests too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/pull/1999#issuecomment-1480343044,https://api.github.com/repos/simonw/datasette/issues/1999,1480343044,IC_kwDOBm6k_c5YPD4E,9599,simonw,2023-03-22T22:33:15Z,2023-03-22T22:33:15Z,OWNER,"This still needs documentation, but now the tests are passing I'm going to merge it into `main`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1480339527,https://api.github.com/repos/simonw/datasette/issues/1999,1480339527,IC_kwDOBm6k_c5YPDBH,9599,simonw,2023-03-22T22:28:54Z,2023-03-22T22:28:54Z,OWNER,"I hacked at the CSV stuff until it worked. I need to clean it up though, but I can do that in this separate task: - #1101","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1480299765,https://api.github.com/repos/simonw/datasette/issues/1999,1480299765,IC_kwDOBm6k_c5YO5T1,9599,simonw,2023-03-22T21:44:03Z,2023-03-22T21:45:10Z,OWNER,"Oh this is a bit tricky. I have a failing test because a plugin that uses the `extra_css_urls` hook can't see the columns for the page. Turns out that bit comes from here: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/app.py#L1203-L1217 Which assumes the context has `""columns""` - but that's only now available if `?_extra=columns` was passed. Actually I think I can cheat here, since it's still getting the HTML context in order to render the template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1480179217,https://api.github.com/repos/simonw/datasette/issues/1999,1480179217,IC_kwDOBm6k_c5YOb4R,9599,simonw,2023-03-22T19:55:31Z,2023-03-22T21:34:02Z,OWNER,"I rebased from `main`. Now: ``` FAILED tests/test_csv.py::test_table_csv - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_cors_headers - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_no_header - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_blob_columns - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_download - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101 FAILED tests/test_csv.py::test_csv_trace - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...} FAILED tests/test_plugins.py::test_hook_render_cell_demo - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_plugins.py::test_hook_render_cell_async[/fixtures/simple_primary_key] - assert b'RENDER_CELL_ASYNC_RESULT' in b'\n\n\n Error 500\n details.open = false);\n});\n\n\n\n\n\n', ...} == {'1+1': 2, 'c... 0xXXX>', ...} FAILED tests/test_permissions.py::test_permissions_checked[/fixtures/simple_primary_key-permissions3] - assert 500 in (200, 403) FAILED tests/test_table_html.py::test_table_csv_json_export_interface - assert 500 == 200 FAILED tests/test_table_html.py::test_table_metadata - assert 500 == 200 FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/simple_primary_key-expected_classes3] - assert 500 == 200 FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert 500 == 200 FAILED tests/test_plugins.py::test_hook_table_actions[simple_view] - AssertionError: assert [] == [{'href': '/'...simple_view'}] ERROR tests/test_internals_database.py::test_execute_write_fn_connection_exception - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_mtime_ns - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_mtime_ns_is_none_for_memory - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_is_mutable - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_database_memory_name - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_in_memory_databases_forbid_writes - Failed: Timeout >1.0s =============== 21 failed, 1275 passed, 2 skipped, 1 xfailed, 6 errors in 59.18s ================ ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1480163485,https://api.github.com/repos/simonw/datasette/issues/1999,1480163485,IC_kwDOBm6k_c5YOYCd,9599,simonw,2023-03-22T19:48:00Z,2023-03-22T19:48:00Z,OWNER,"Getting close now! Only 13 failures left, mostly relating to CSV. ``` FAILED tests/test_csv.py::test_table_csv - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_cors_headers - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_no_header - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_blob_columns - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_download - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101 FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...} FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets' FAILED tests/test_csv.py::test_csv_trace - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...} FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'1+1': 2, 'c... 0xXXX>', ...} == {'1+1': 2, 'c... 0xXXX>', ...} =============== 13 failed, 1287 passed, 2 skipped, 1 xfailed in 61.57s (0:01:01) ================ ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1480154453,https://api.github.com/repos/simonw/datasette/issues/1999,1480154453,IC_kwDOBm6k_c5YOV1V,9599,simonw,2023-03-22T19:41:39Z,2023-03-22T19:43:04Z,OWNER,"To replace this code: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L122 Maybe `datasette.render_template()` should optionally accept a list of templates. https://docs.datasette.io/en/stable/internals.html#await-render-template-template-context-none-request-none - turns out it does already: > If this is a list of template file names then the first one that exists will be loaded and rendered. It doesn't have an easy way to populate that `select_templates` debug template variable though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/issues/2041#issuecomment-1477155602,https://api.github.com/repos/simonw/datasette/issues/2041,1477155602,IC_kwDOBm6k_c5YC5sS,9599,simonw,2023-03-21T01:17:43Z,2023-03-21T01:17:43Z,OWNER,"Removed code in https://github.com/simonw/datasette/commit/538ca9d2e2d41308f5b149c934e2bc43727fd77c - it will merge when I land: - #1999","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1633077183,Remove obsolete table POST code, https://github.com/simonw/datasette/issues/2041#issuecomment-1477144853,https://api.github.com/repos/simonw/datasette/issues/2041,1477144853,IC_kwDOBm6k_c5YC3EV,9599,simonw,2023-03-21T01:02:13Z,2023-03-21T01:02:13Z,OWNER,"This code here: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/table.py#L117-L169","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1633077183,Remove obsolete table POST code, https://github.com/simonw/datasette/pull/1999#issuecomment-1477082852,https://api.github.com/repos/simonw/datasette/issues/1999,1477082852,IC_kwDOBm6k_c5YCn7k,9599,simonw,2023-03-20T23:27:25Z,2023-03-20T23:27:25Z,OWNER,"Urgh getting CSV to work is going to be _so hard_, because the logic for that currently lives in a huge chunk of code in `BaseView` which depends on the old design of the `data()` method: https://github.com/simonw/datasette/blob/4bb49848697e40b8b9a1557be42b8e59eac965b3/datasette/views/base.py#L177-L343","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1476898261,https://api.github.com/repos/simonw/datasette/issues/1999,1476898261,IC_kwDOBm6k_c5YB63V,9599,simonw,2023-03-20T20:37:52Z,2023-03-20T20:37:52Z,OWNER,"Manual testing spotted a bug. `/content/repos.json?owner=9599&_facet_array=topics` - does not return a `facet_results` key. `/content/repos.json?owner=9599&_facet_array=topics&_facet=owner` does.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1476860334,https://api.github.com/repos/simonw/datasette/issues/1999,1476860334,IC_kwDOBm6k_c5YBxmu,9599,simonw,2023-03-20T20:06:56Z,2023-03-20T22:09:03Z,OWNER,"`pytest -n 8` locally says: ``` FAILED tests/test_canned_queries.py::test_canned_query_form_csrf_hidden_field[add_name_specify_id-True] - assert '\n\n\n Error 500\n details.open = false);\n});\n\n\n\n Error 500\n details.open = false);\n});\n' in '\n\n\n Error 500\n details.open = false);\... FAILED tests/test_canned_queries.py::test_insert - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_insert_error - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[False-True] - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[True-True] - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_vary_header - KeyError: 'vary' FAILED tests/test_csv.py::test_csv_trace - AssertionError: assert 'application/...charset=utf-8' == 'text/html; charset=utf-8' FAILED tests/test_csv.py::test_table_csv - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_blob_columns - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_download - AssertionError: assert 'application/...charset=utf-8' == 'text/csv; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_no_header - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101 FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_counts - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_facets - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_csv.py::test_table_csv_with_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_facets.py::test_array_facet_handle_duplicate_tags - KeyError: 'facet_results' FAILED tests/test_facets.py::test_conflicting_facet_names_json - AssertionError: assert {'results', 'timed_out'} == {'created', '...gs', 'tags_2'} FAILED tests/test_facets.py::test_facet_size - KeyError: 'suggested_facets' FAILED tests/test_facets.py::test_json_array_with_blanks_and_nulls - KeyError: 'suggested_facets' FAILED tests/test_facets.py::test_other_types_of_facet_in_metadata - assert 'created (date)\n' in '\n\n\n fixtures: facetable: 15 rows\n \n\n\n\n' in '\n\n\n fixtures: simple_primary_key: 5 rows\n ' in '\n\n\n fixtures: table/with/slashes.csv: 1 row\n ', ...} FAILED tests/test_plugins.py::test_hook_register_output_renderer_no_parameters - assert b'Hello' == b'{""ok"": true... ""n"": null}]}' FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_broken_value - assert 200 == 500 FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_response - assert {'next': None...', ...}, ...]} == {'this_is': 'json'} === 39 failed, 1259 passed, 2 skipped, 1 xfailed in 58.07s === ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1476854645,https://api.github.com/repos/simonw/datasette/issues/1999,1476854645,IC_kwDOBm6k_c5YBwN1,9599,simonw,2023-03-20T20:02:24Z,2023-03-20T20:02:24Z,OWNER,"``` def test_routes(routes, path, expected_class, expected_matches): match, view = resolve_routes(routes, path) if expected_class is None: assert match is None else: > assert view.view_class.__name__ == expected_class E AttributeError: 'function' object has no attribute 'view_class' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1476851525,https://api.github.com/repos/simonw/datasette/issues/1999,1476851525,IC_kwDOBm6k_c5YBvdF,9599,simonw,2023-03-20T19:59:51Z,2023-03-20T20:01:40Z,OWNER,"Three failures in `test_html.py`: ``` FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert '' in '\n\n\n fixtures: simple_primary_key: 5 rows\n ' in '\n\n\n fixtures: table/with/slashes.csv: 1 row\n \xa0']] == [['\xa0']] FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels - assert [['b']] == [['b']] FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['']] == [['']] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/infinity-expected_column_options0] - AssertionError: assert ['- column -'...wid', 'value'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/primary_key_multiple_columns-expected_column_options1] - AssertionError: assert ['- column -'...', 'content2'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/compound_primary_key-expected_column_options2] - AssertionError: assert ['- column -'...2', 'content'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns - AssertionError: assert ['- column -'] == ['- column -'...nulls_2', ...] FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references - assert [['']] == [['']] FAILED tests/test_table_html.py::test_view_html - assert None is not None FAILED tests/test_table_html.py::test_extra_where_clauses - assert [('_where', ""...'_city_id=1')] == [] FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10-expected_hidden0] - AssertionError: assert [] == [('_size', '10')] FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10&_ignore=1&_ignore=2-expected_hidden1] - AssertionError: assert [] == [('_size', '1...ignore', '2')] FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated[/fixtures/searchable?_sort=text2&_where=1-expected_hidden3] - AssertionError: assert [] == [('_where', '1')] FAILED tests/test_table_html.py::test_binary_data_display_in_table - assert [['\xa0']] == [['\xa0']] FAILED tests/test_table_html.py::test_metadata_sort - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_metadata_sort_desc - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood-2-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 2 FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood&_facet_size=50-5-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 5 FAILED tests/test_table_html.py::test_facet_total - assert 500 == 200 ``` Deduped that's 30 tests: ``` FAILED tests/test_table_html.py::test_add_filter_redirects FAILED tests/test_table_html.py::test_binary_data_display_in_table FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys FAILED tests/test_table_html.py::test_empty_search_parameter_gets_removed FAILED tests/test_table_html.py::test_existing_filter_redirects FAILED tests/test_table_html.py::test_extra_where_clauses FAILED tests/test_table_html.py::test_facet_more_links FAILED tests/test_table_html.py::test_facet_total FAILED tests/test_table_html.py::test_facets_persist_through_filter_form FAILED tests/test_table_html.py::test_metadata_sort FAILED tests/test_table_html.py::test_metadata_sort_desc FAILED tests/test_table_html.py::test_next_does_not_persist_in_hidden_field FAILED tests/test_table_html.py::test_other_hidden_form_fields FAILED tests/test_table_html.py::test_reflected_hidden_form_fields FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated FAILED tests/test_table_html.py::test_searchable_view_persists_fts_table FAILED tests/test_table_html.py::test_sort_by_desc_redirects FAILED tests/test_table_html.py::test_sort_links FAILED tests/test_table_html.py::test_table_csv_json_export_interface FAILED tests/test_table_html.py::test_table_html_compound_primary_key FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels FAILED tests/test_table_html.py::test_table_html_filter_form_column_options FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column FAILED tests/test_table_html.py::test_table_html_foreign_key_links FAILED tests/test_table_html.py::test_table_html_no_primary_key FAILED tests/test_table_html.py::test_table_html_simple_primary_key FAILED tests/test_table_html.py::test_view_html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1475074025,https://api.github.com/repos/simonw/datasette/issues/1999,1475074025,IC_kwDOBm6k_c5X69fp,9599,simonw,2023-03-19T02:14:28Z,2023-03-19T02:14:51Z,OWNER,"I had to replicate quite a bit of this logic from `base.py`: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L526-L544 I should refactor this when I move the canned / arbitrary query views away from that base class too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1475016834,https://api.github.com/repos/simonw/datasette/issues/1999,1475016834,IC_kwDOBm6k_c5X6viC,9599,simonw,2023-03-18T22:30:31Z,2023-03-18T22:30:31Z,OWNER,"`test_paginate_using_link_header` will be tricky. The reason the tests are failing is that `json_renderer()` attempts to populate the `link` header using `data[""next_url""]` - but that's not present unless `?_extra=next_url` has been passed: https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/renderer.py#L101-L102 But I can only rely on `next` being present, not `next_url`. I thought I could maybe assemble the `link` header using `next`, by turning that into a `next_url` link - but there's some custom logic which kicks in for pagination against SQL views (offset/limit pagination) to calculate the `next_url` which isn't easily replicable at the `json_renderer()` layer, in the `_next_value_and_url()` utility function: https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/views/table.py#L2275-L2282 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1475003292,https://api.github.com/repos/simonw/datasette/issues/1999,1475003292,IC_kwDOBm6k_c5X6sOc,9599,simonw,2023-03-18T21:46:20Z,2023-03-18T21:46:20Z,OWNER,"Now 25 failures in `test_table_api.py`: ``` FAILED tests/test_table_api.py::test_expand_labels - assert {'2': {'pk': 2, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', '_city_id': 1, '_neighborhood': 'Dogpatch', 'tags': '[""tag1"", ""tag3""]', 'complex_array'... FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_co... FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1, 'value': inf}, {'rowid': 2, 'value': -inf}, {'rowid': 3, 'value': 1.5}] == [{'rowid': 1, 'value': None}, {'rowid': 2, 'value': None}, {'rowid': 3, 'value'... FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_compoun... FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'rowid': 1, 'data': ""b'\\x15\\x1c\\x02\\xc7\\xad\\x05\\xfe'""}, {'rowid': 2, 'data': ""b'\\x15\\x1c\\x03\\xc7\\xad\\x05\\xfe'""}, {'rowid': 3, 'data': None}] == [{'rowid': 1, 'data': {'... FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{""rowid"": 1, ""data"": {""$base64"": true, ""encoded"": ""FRwCx60F/g==""}}\n{""rowid"": 2, ""data"": {""$base64"": true, ""encoded"": ""FRwDx60F/g==""}}\n{""rowid"": 3, ""data"": null}] - assert '{""ok"": false, ""error"": ""Object of type bytes is not JSON serializable"", ""status"": 500, ""title"": null}' == '{""rowid"": 1, ""data"": {""$base64"": true, ""encoded"": ""FRwCx60F/g==""}}\n{""rowid""... FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21 FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns' ============================================================================= 25 failed, 86 passed, 1 xfailed in 7.18s ============================================================================= ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1474704790,https://api.github.com/repos/simonw/datasette/issues/1999,1474704790,IC_kwDOBm6k_c5X5jWW,9599,simonw,2023-03-18T04:52:59Z,2023-03-18T04:52:59Z,OWNER,"Here are the next set of tests to get passing: ``` % pytest tests/test_table_api.py ``` ``` FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id-expected_facet_results0] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id&state=MI-expected_facet_results1] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int-expected_facet_results2] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int&planet_int=1-expected_facet_results3] - KeyError: 'name' FAILED tests/test_table_api.py::test_suggested_facets - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_allow_facet_off - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_suggest_facets_off - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nofacet[True] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nofacet[False] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nosuggest[True] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nosuggest[False] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nocount[True-None] - KeyError: 'count' FAILED tests/test_table_api.py::test_nocount[False-15] - KeyError: 'count' FAILED tests/test_table_api.py::test_expand_labels - AssertionError: assert {'13': {'_cit...:00:00', ...}} == {'13': {'_cit...:00:00', ...}} FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'forei...l': '1', ...}} == {'1': {'forei...': '1'}, ...}} FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1,...'value': 1.5}] == [{'rowid': 1,...'value': 1.5}] FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'foreign_ke...': None, ...}] == [{'foreign_ke...': None, ...}] FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'data': ""b'..., 'rowid': 3}] == [{'data': {'$..., 'rowid': 3}] FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{""rowid"": 1, ""data"": {""$base64"": true, ""encoded"": ""FRwCx60F/g==""}}\n{""rowid"": 2, ""data"": {""$base64"": true, ""encoded"": ""FRwDx60F/g==""}}\n{""rowid"": 3, ""data"": null}] - assert '{""ok"": false...title"": null}' == '{""rowid"": 1,...""data"": null}' FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21 FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns' ============================================================================= 38 failed, 73 passed, 1 xfailed in 7.25s ============================================================================= ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465303378,https://api.github.com/repos/simonw/sqlite-utils/issues/533,1465303378,IC_kwDOCGYnMM5XVsFS,9599,simonw,2023-03-12T21:24:04Z,2023-03-12T21:24:04Z,OWNER,"Upgraded to Sphinx 6 locally and got the same error: ``` % just docs Cogging README.md Cogging docs/changelog.rst Cogging docs/cli-reference.rst Cogging docs/cli.rst Cogging docs/contributing.rst Cogging docs/index.rst Cogging docs/installation.rst Cogging docs/python-api.rst Cogging docs/reference.rst sphinx-autobuild -a -b html ""."" ""_build"" --watch ../sqlite_utils [sphinx-autobuild] > sphinx-build -b html -a /Users/simon/Dropbox/Development/sqlite-utils/docs /Users/simon/Dropbox/Development/sqlite-utils/docs/_build Running Sphinx v6.1.3 loading pickled environment... failed failed: Can't get attribute '_stable_repr_object' on building [mo]: all of 0 po files writing output... building [html]: all source files updating environment: [new config] 8 added, 0 changed, 0 removed reading sources... [ 12%] changelog Exception occurred: File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/sphinx/ext/extlinks.py"", line 103, in role title = caption % part ~~~~~~~~^~~~~~ TypeError: not all arguments converted during string formatting The full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1ey36c1n.log, if you want to report the issue to the developers. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620516340,ReadTheDocs error: not all arguments converted during string formatting, https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465302936,https://api.github.com/repos/simonw/sqlite-utils/issues/533,1465302936,IC_kwDOCGYnMM5XVr-Y,9599,simonw,2023-03-12T21:22:09Z,2023-03-12T21:22:09Z,OWNER,"Could be the same problem as: - https://github.com/simonw/datasette/issues/1972 Which I fixed in https://github.com/simonw/datasette/commit/3af313e165215696af899e772f47bf7c27873ae3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620516340,ReadTheDocs error: not all arguments converted during string formatting, https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465302343,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1465302343,IC_kwDOCGYnMM5XVr1H,9599,simonw,2023-03-12T21:19:13Z,2023-03-12T21:19:13Z,OWNER,"Aah, I think I see why you wrote it like that. The problem is that `init_spatialite()` does other stuff too: https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L1161-L1171 So it needs to be able to load the SpatiaLite extension from the correct place, and THEN run `select InitSpatialMetadata()` to configure the database, if needed. So the problem you're trying to solve here is to let people optionally pass in the path to SpatiaLite if it's not one of the ones that are searched by default. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon, https://github.com/simonw/datasette/pull/1999#issuecomment-1463113856,https://api.github.com/repos/simonw/datasette/issues/1999,1463113856,IC_kwDOBm6k_c5XNViA,9599,simonw,2023-03-10T02:13:15Z,2023-03-10T02:13:15Z,OWNER,Idea for if this change ends up making a bunch of breaking changes to the templates (which I think it should) - I can generate a GitHub diff link between the old and new templates and include that link in the 1.0 upgrade documentation to help people who wrote custom templates see what they might need to change - with minimal effort from myself to document those changes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/issues/2038#issuecomment-1463112173,https://api.github.com/repos/simonw/datasette/issues/2038,1463112173,IC_kwDOBm6k_c5XNVHt,9599,simonw,2023-03-10T02:11:06Z,2023-03-10T02:11:06Z,OWNER,"Here's an example of something that would break: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/templates/database.html#L12-L15 Because `metadata` is an empty dictionary sometimes, so `{{ metadata.title or database }}` would raise an error and need to be replaced by `{{ metadata.get(""title"") or database }}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1618249044,Consider a `strict_templates` setting, https://github.com/simonw/datasette/issues/2038#issuecomment-1463110978,https://api.github.com/repos/simonw/datasette/issues/2038,1463110978,IC_kwDOBm6k_c5XNU1C,9599,simonw,2023-03-10T02:09:41Z,2023-03-10T02:09:41Z,OWNER,"I'm torn on this. It's useful for me right now for refactoring, but I feel like it should be a permanent thing, not a setting - or it should default to on and people should turn it off, but who would ever do that?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1618249044,Consider a `strict_templates` setting, https://github.com/simonw/datasette/pull/1999#issuecomment-1463024951,https://api.github.com/repos/simonw/datasette/issues/1999,1463024951,IC_kwDOBm6k_c5XM_03,9599,simonw,2023-03-10T00:17:58Z,2023-03-10T00:17:58Z,OWNER,"Renderers have an impact on three different pages: query results, table page and row page. The row page feature is incomplete though: https://congress-legislators.datasettes.com/legislators/social_media/A000055 Why is there no `.csv` link there?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1463023674,https://api.github.com/repos/simonw/datasette/issues/1999,1463023674,IC_kwDOBm6k_c5XM_g6,9599,simonw,2023-03-10T00:16:03Z,2023-03-10T00:16:03Z,OWNER,"I also need to figure out the `renderers` stuff, so I can link to the right URLs for CSV and JSON and other formats: https://github.com/simonw/datasette/blob/6d07a7da1531cd749844fc6827d9a1e57009b2ea/datasette/views/base.py#L474-L518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1463022397,https://api.github.com/repos/simonw/datasette/issues/1999,1463022397,IC_kwDOBm6k_c5XM_M9,9599,simonw,2023-03-10T00:14:21Z,2023-03-10T00:14:21Z,OWNER,"Ironically the thing I most need right now is comprehensive documentation of what variables are being passed to the templates! One big challenge is that I need to untangle the template context that happens in `BaseView` - I'm hacking that together at the moment, but I need a real answer for how that should work in a world in which view functions aren't using that base class. https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L145","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1463021383,https://api.github.com/repos/simonw/datasette/issues/1999,1463021383,IC_kwDOBm6k_c5XM-9H,9599,simonw,2023-03-10T00:12:50Z,2023-03-10T00:12:50Z,OWNER,"Now at 34 failed, 34 passed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1463005744,https://api.github.com/repos/simonw/datasette/issues/1999,1463005744,IC_kwDOBm6k_c5XM7Iw,9599,simonw,2023-03-09T23:52:15Z,2023-03-09T23:52:23Z,OWNER,"I need to figure out what to do about `extra_context_from_filters` - which was previously passed straight to the HTML context. https://github.com/simonw/datasette/blob/11f7feb7a3f7166c71389786880863d60ed3d165/datasette/views/table.py#L406-L422","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1462997800,https://api.github.com/repos/simonw/datasette/issues/1999,1462997800,IC_kwDOBm6k_c5XM5Mo,9599,simonw,2023-03-09T23:39:47Z,2023-03-09T23:39:47Z,OWNER,"Found a neat trick: ```diff diff --git a/datasette/app.py b/datasette/app.py index 186f192d..40416713 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -23,7 +23,13 @@ from pathlib import Path from markupsafe import Markup, escape from itsdangerous import URLSafeSerializer -from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader +from jinja2 import ( + ChoiceLoader, + Environment, + FileSystemLoader, + PrefixLoader, + StrictUndefined, +) from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound @@ -394,7 +400,10 @@ class Datasette: ] ) self.jinja_env = Environment( - loader=template_loader, autoescape=True, enable_async=True + loader=template_loader, + autoescape=True, + enable_async=True, + undefined=StrictUndefined, ) self.jinja_env.filters[""escape_css_string""] = escape_css_string self.jinja_env.filters[""quote_plus""] = urllib.parse.quote_plus ``` This causes Jinja to raise a hard error if there are any variables referenced in the template that are not available in the context. It's helping me spot things that are still missing, rather than just relying on failed unit tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/2031#issuecomment-1462921890,https://api.github.com/repos/simonw/datasette/issues/2031,1462921890,IC_kwDOBm6k_c5XMmqi,9599,simonw,2023-03-09T22:35:30Z,2023-03-09T22:35:30Z,OWNER,"> I've implemented the test (thanks for pointing me in the right direction!). > > At [tmcl-it/datasette:0.64.1+row-view-expand-labels](https://github.com/tmcl-it/datasette/tree/0.64.1%2Brow-view-expand-labels) I also have a variant of this patch that applies to the 0.64.x branch. Please let me know if you'd be interested in merging that as well and I'll open another PR. Sure, let's merge that one too - it can go out in the next `0.64.x` series release (maybe even a 0.65).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1605481359,Expand foreign key references in row view as well, https://github.com/simonw/datasette/pull/2034#issuecomment-1462921010,https://api.github.com/repos/simonw/datasette/issues/2034,1462921010,IC_kwDOBm6k_c5XMmcy,9599,simonw,2023-03-09T22:34:29Z,2023-03-09T22:34:29Z,OWNER,"Good catch, thanks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1613974869,remove an unused `app` var in cli.py, https://github.com/simonw/datasette/pull/1999#issuecomment-1461161256,https://api.github.com/repos/simonw/datasette/issues/1999,1461161256,IC_kwDOBm6k_c5XF40o,9599,simonw,2023-03-09T02:10:07Z,2023-03-09T02:10:07Z,OWNER,"Just ran into a `no such table: columns` error - which I think is because my new view code sometimes bypasses calling this method (currently done in `BaseView.dispatch_request()`): https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/base.py#L101-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461148579,https://api.github.com/repos/simonw/datasette/issues/1999,1461148579,IC_kwDOBm6k_c5XF1uj,9599,simonw,2023-03-09T01:54:10Z,2023-03-09T01:55:33Z,OWNER,Or... I could temporarily build a quick additional `CannedQueryView` subclass that just does the necessary bits to get the existing code to work. I'm going to try that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461148254,https://api.github.com/repos/simonw/datasette/issues/1999,1461148254,IC_kwDOBm6k_c5XF1pe,9599,simonw,2023-03-09T01:53:41Z,2023-03-09T01:53:41Z,OWNER,"Solving this is proving difficult: https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L1500-L1503 The problem is that calling `.data()` on `QueryView` only works here because we expect to ourselves be inside a `.data()` method, with all of the existing magic that knows how to render things that are returned by that. So I may need to substantially re-engineer how `QueryView` works in order to get this to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461075648,https://api.github.com/repos/simonw/datasette/issues/1999,1461075648,IC_kwDOBm6k_c5XFj7A,9599,simonw,2023-03-09T00:24:22Z,2023-03-09T00:24:22Z,OWNER,`127.0.0.1:8001/fixtures/neighborhood_search` fails because the forwarding to a canned query does not yet work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461074526,https://api.github.com/repos/simonw/datasette/issues/1999,1461074526,IC_kwDOBm6k_c5XFjpe,9599,simonw,2023-03-09T00:23:06Z,2023-03-09T00:23:06Z,OWNER," pytest tests/test_table_html.py Currently 44 failed, 24 passed in 7.53s Failures here: https://gist.github.com/simonw/df0a52cd7d820b776dc3dfc50e7cb778","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461070937,https://api.github.com/repos/simonw/datasette/issues/1999,1461070937,IC_kwDOBm6k_c5XFixZ,9599,simonw,2023-03-09T00:18:52Z,2023-03-09T00:19:36Z,OWNER,"I managed to get HTML view working! I did it by continuing to add more things to the extras and the `_html` bundle until the page loaded for me: ```diff async def extra_extras(): ""Available ?_extra= blocks"" return { @@ -1981,6 +2053,14 @@ async def extra_extras(): ""query"", ""display_columns"", ""display_rows"", + ""database"", + ""table"", + ""database_color"", + ""table_actions"", + ""filters"", + ""renderers"", + ""custom_table_templates"", + ""sorted_facet_results"", ] } @@ -2006,6 +2086,14 @@ async def extra_extras(): extra_query, extra_metadata, extra_extras, + extra_database, + extra_table, + extra_database_color, + extra_table_actions, + extra_filters, + extra_renderers, + extra_custom_table_templates, + extra_sorted_facet_results, ) ``` I'll probably refactor this into something cleaner, and maybe but a bunch of them in a `""html""` dictionary and update the templates to use `{{ html.filters }}` or similar. Will look at that once the tests are passing. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461047607,https://api.github.com/repos/simonw/datasette/issues/1999,1461047607,IC_kwDOBm6k_c5XFdE3,9599,simonw,2023-03-08T23:51:46Z,2023-03-08T23:51:46Z,OWNER,"This feels quite nice: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461044477,https://api.github.com/repos/simonw/datasette/issues/1999,1461044477,IC_kwDOBm6k_c5XFcT9,9599,simonw,2023-03-08T23:47:26Z,2023-03-08T23:47:26Z,OWNER,"I want to package together all of the extras that are needed for the HTML format. A few options for doing that: - Introduce `?_extra=_html` where the leading underscore indicates that this is a ""bundle"" of extras, then define a bundle that's everything needed for the HTML renderer - Have some other mechanism whereby different renderers can request a bundle of extras. I'm leaning towards the first option. I'll try that and see what it looks like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461023559,https://api.github.com/repos/simonw/datasette/issues/1999,1461023559,IC_kwDOBm6k_c5XFXNH,9599,simonw,2023-03-08T23:23:02Z,2023-03-08T23:23:02Z,OWNER,"To get this unblocked, I'm going to allow myself to pass non-JSON-serializable objects to the HTML template version of things. If I can get that working (and get the existing tests to pass) I can consider a later change that makes those JSON serializable - or admit that it's OK for the templates to have non-JSON data passed to them and figure out how best to document those variables independently from the JSON documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461002039,https://api.github.com/repos/simonw/datasette/issues/1999,1461002039,IC_kwDOBm6k_c5XFR83,9599,simonw,2023-03-08T22:58:16Z,2023-03-08T23:02:09Z,OWNER,"The reason for that `Row` thing is that it allows custom templates that do things like this: https://docs.datasette.io/en/stable/changelog.html#easier-custom-templates-for-table-rows ```html+jinja {% for row in display_rows %}

{{ row[""title""] }}

{{ row[""description""] }}

Category: {{ row.display(""category_id"") }}

{% endfor %} ``` Is that a good design? the `.display()` thing feels weird - I wonder if anyone has ever actually used that. It's documented here: https://docs.datasette.io/en/0.64.2/custom_templates.html#custom-templates > If you want to output the rendered HTML version of a column, including any links to foreign keys, you can use `{{ row.display(""column_name"") }}`. I can't see any examples of anyone using it in this code search: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+row.display It is however useful to have some kind of abstraction layer here that insulates the SQLite `Row` object, since having an extra layer will help if Datasette ever grows support for alternative database backends such as DuckDB or PostgreSQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460988975,https://api.github.com/repos/simonw/datasette/issues/1999,1460988975,IC_kwDOBm6k_c5XFOwv,9599,simonw,2023-03-08T22:42:57Z,2023-03-08T22:42:57Z,OWNER,"Aside idea: it might be interesting if there were ""lazy"" template variables available in the context: things that are not actually executed unless a template author requests them. Imagine if `metadata` was a lazy template reference, such that custom templates that don't display any metadata don't trigger it to be resolved (which might involve additional database queries some day).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460986533,https://api.github.com/repos/simonw/datasette/issues/1999,1460986533,IC_kwDOBm6k_c5XFOKl,9599,simonw,2023-03-08T22:40:28Z,2023-03-08T22:40:28Z,OWNER,"Figuring out what to do with `display_columns_and_rows()` is hard. That returns rows as this special kind of object, which is designed to be accessed from the HTML templates: https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L45-L71","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460970807,https://api.github.com/repos/simonw/datasette/issues/1999,1460970807,IC_kwDOBm6k_c5XFKU3,9599,simonw,2023-03-08T22:31:49Z,2023-03-08T22:33:03Z,OWNER,"For the HTML version, I need to decide where all of the stuff that happens in `async def extra_template()` is going to live. I think it's another one of those extra functions, triggered for `?_extra=context`. https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L813-L912","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460943097,https://api.github.com/repos/simonw/datasette/issues/1999,1460943097,IC_kwDOBm6k_c5XFDj5,9599,simonw,2023-03-08T22:09:24Z,2023-03-08T22:09:47Z,OWNER,"The ease with which I added that `?_extra=query` feature in https://github.com/simonw/datasette/pull/1999/commits/96e94f9b7b2db53865e61390bcce6761727f26d8 made me feel really confident that this architecture is going in the right direction. ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 8d3bb2c930..3e1db9c85f 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1913,6 +1913,13 @@ async def extra_request(): ""args"": request.args._data, } + async def extra_query(): + ""Details of the underlying SQL query"" + return { + ""sql"": sql, + ""params"": params, + } + async def extra_extras(): ""Available ?_extra= blocks"" return { @@ -1938,6 +1945,7 @@ async def extra_extras(): extra_primary_keys, extra_debug, extra_request, + extra_query, extra_extras, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460916405,https://api.github.com/repos/simonw/datasette/issues/1999,1460916405,IC_kwDOBm6k_c5XE9C1,9599,simonw,2023-03-08T21:43:27Z,2023-03-08T21:43:27Z,OWNER,"Just noticed that `_json=colname` is not working, and that's because it's handled by the renderer here: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/renderer.py#L29-L40 But that's not currently being called by my new code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460907148,https://api.github.com/repos/simonw/datasette/issues/1999,1460907148,IC_kwDOBm6k_c5XE6yM,9599,simonw,2023-03-08T21:34:30Z,2023-03-08T21:34:30Z,OWNER,"I'm going to hold off on that refactor until later, when I have tests to show me if the refactor works or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460906741,https://api.github.com/repos/simonw/datasette/issues/1999,1460906741,IC_kwDOBm6k_c5XE6r1,9599,simonw,2023-03-08T21:34:08Z,2023-03-08T21:34:08Z,OWNER,"So maybe I can refactor it to look a bit more like this: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1602-L1604 One thing that's useful here is that `is_view` is handled early, like this: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L466-L472 So if I omit the `is_view` bit from the extracted function I can simplify more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469,https://api.github.com/repos/simonw/datasette/issues/1999,1460905469,IC_kwDOBm6k_c5XE6X9,9599,simonw,2023-03-08T21:32:56Z,2023-03-08T21:32:56Z,OWNER,"I'd really like to extract this ugly logic out into a helper function: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1617-L1684 I copied it in and asked ChatGPT to ""List all of the variable that are either modified or assigned to by the above code"": > Modified variables: > > offset > where_clauses > params > order_by > > Assigned variables: > > sort_value > next_by_pk_clauses > order_by (when sort or sort_desc is not None) Then I asked which variables were used as inputs, and argued with it a bit about whether it should be counting functions. Eventually got to this: > My apologies for the oversight. Here are the variables needed as input by the above code, excluding any functions: > > _next > is_view > sort > sort_desc > use_rowid > pks > order_by Note that `use_rowid` is actually defined earlier in terms of two of those other variables: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1540","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/issues/2036#issuecomment-1460866243,https://api.github.com/repos/simonw/datasette/issues/2036,1460866243,IC_kwDOBm6k_c5XEwzD,9599,simonw,2023-03-08T20:57:34Z,2023-03-08T20:57:34Z,OWNER,This fix is released in 0.64.2 https://docs.datasette.io/en/stable/changelog.html#v0-64-2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,"`publish cloudrun` reuses image tags, which can lead to very surprising deploy problems", https://github.com/simonw/datasette/issues/2036#issuecomment-1460848869,https://api.github.com/repos/simonw/datasette/issues/2036,1460848869,IC_kwDOBm6k_c5XEsjl,9599,simonw,2023-03-08T20:40:55Z,2023-03-08T20:40:55Z,OWNER,"Here's the https://latest.datasette.io/ deployment that just went out, further demonstrating that this change is working correctly: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,"`publish cloudrun` reuses image tags, which can lead to very surprising deploy problems", https://github.com/simonw/datasette/issues/2037#issuecomment-1460840620,https://api.github.com/repos/simonw/datasette/issues/2037,1460840620,IC_kwDOBm6k_c5XEqis,9599,simonw,2023-03-08T20:33:00Z,2023-03-08T20:33:00Z,OWNER,Got the same failure again for a recent commit: https://github.com/simonw/datasette/actions/runs/4368239376/jobs/7640567282,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615891776,Test failure: FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError, https://github.com/simonw/datasette/issues/2037#issuecomment-1460838797,https://api.github.com/repos/simonw/datasette/issues/2037,1460838797,IC_kwDOBm6k_c5XEqGN,9599,simonw,2023-03-08T20:31:15Z,2023-03-08T20:31:15Z,OWNER,"It's this test here: https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/tests/test_cli.py#L181-L189 Added in: - #2033 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615891776,Test failure: FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError, https://github.com/simonw/datasette/issues/2037#issuecomment-1460838109,https://api.github.com/repos/simonw/datasette/issues/2037,1460838109,IC_kwDOBm6k_c5XEp7d,9599,simonw,2023-03-08T20:30:36Z,2023-03-08T20:30:36Z,OWNER,Instead of using `isolated_filesystem()` I could use a `tmpdir` fixture instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615891776,Test failure: FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError, https://github.com/simonw/datasette/issues/2036#issuecomment-1460827178,https://api.github.com/repos/simonw/datasette/issues/2036,1460827178,IC_kwDOBm6k_c5XEnQq,9599,simonw,2023-03-08T20:25:10Z,2023-03-08T20:25:10Z,OWNER,"https://console.cloud.google.com/run/detail/us-central1/new-service/revisions?project=datasette-222320 confirms that the image deployed is: Compared to https://console.cloud.google.com/run/detail/us-central1/datasette-io/revisions?project=datasette-222320 which shows that `datasette.io` is running: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,"`publish cloudrun` reuses image tags, which can lead to very surprising deploy problems", https://github.com/simonw/datasette/issues/2036#issuecomment-1460816528,https://api.github.com/repos/simonw/datasette/issues/2036,1460816528,IC_kwDOBm6k_c5XEkqQ,9599,simonw,2023-03-08T20:22:50Z,2023-03-08T20:23:20Z,OWNER,"Testing this manually: ``` % datasette publish cloudrun content.db --service new-service Creating temporary tarball archive of 2 file(s) totalling 13.8 MiB before compression. Uploading tarball of [.] to [gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz] Created [https://cloudbuild.googleapis.com/v1/projects/datasette-222320/locations/global/builds/290f41a4-e29a-443c-a1e5-c54513c6143d]. Logs are available at [ https://console.cloud.google.com/cloud-build/builds/290f41a4-e29a-443c-a1e5-c54513c6143d?project=99025868001 ]. ---- REMOTE BUILD OUTPUT ---- starting build ""290f41a4-e29a-443c-a1e5-c54513c6143d"" FETCHSOURCE Fetching storage object: gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483 Copying gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483... / [1 files][ 3.9 MiB/ 3.9 MiB] Operation completed over 1 objects/3.9 MiB. BUILD Already have image (with digest): gcr.io/cloud-builders/docker Sending build context to Docker daemon 14.52MB Step 1/9 : FROM python:3.11.0-slim-bullseye ... Installing collected packages: rfc3986, typing-extensions, sniffio, PyYAML, python-multipart, pluggy, pint, mergedeep, MarkupSafe, itsdangerous, idna, hupper, h11, click, certifi, asgiref, aiofiles, uvicorn, Jinja2, janus, click-default-group-wheel, asgi-csrf, anyio, httpcore, httpx, datasette Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.2 PyYAML-6.0 aiofiles-23.1.0 anyio-3.6.2 asgi-csrf-0.9 asgiref-3.6.0 certifi-2022.12.7 click-8.1.3 click-default-group-wheel-1.2.2 datasette-0.64.1 h11-0.14.0 httpcore-0.16.3 httpx-0.23.3 hupper-1.11 idna-3.4 itsdangerous-2.1.2 janus-1.0.0 mergedeep-1.3.4 pint-0.20.1 pluggy-1.0.0 python-multipart-0.0.6 rfc3986-1.5.0 sniffio-1.3.0 typing-extensions-4.5.0 uvicorn-0.20.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip available: 22.3 -> 23.0.1 [notice] To update, run: pip install --upgrade pip Removing intermediate container 8ccebfebebc9 ---> b972c85b38bb ... Successfully built 606b7c286d7f Successfully tagged gcr.io/datasette-222320/datasette-new-service:latest PUSH Pushing gcr.io/datasette-222320/datasette-new-service The push refers to repository [gcr.io/datasette-222320/datasette-new-service] 667b1dc69e5e: Preparing ... d8ddfcff216f: Pushed latest: digest: sha256:452daffb2d3d7a8579c2ab39854be285155252c9428b4c1c50caac6a3a269e3f size: 2004 DONE --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ID CREATE_TIME DURATION SOURCE IMAGES STATUS 290f41a4-e29a-443c-a1e5-c54513c6143d 2023-03-08T20:21:03+00:00 39S gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz gcr.io/datasette-222320/datasette-new-service (+1 more) SUCCESS Deploying container to Cloud Run service [new-service] in project [datasette-222320] region [us-central1] ✓ Deploying new service... Done. ✓ Creating Revision... ✓ Routing traffic... ✓ Setting IAM Policy... Done. Service [new-service] revision [new-service-00001-zon] has been deployed and is serving 100 percent of traffic. Service URL: https://new-service-j7hipcg4aq-uc.a.run.app ``` https://new-service-j7hipcg4aq-uc.a.run.app/ was deployed successfully.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,"`publish cloudrun` reuses image tags, which can lead to very surprising deploy problems", https://github.com/simonw/datasette/issues/2036#issuecomment-1460810523,https://api.github.com/repos/simonw/datasette/issues/2036,1460810523,IC_kwDOBm6k_c5XEjMb,9599,simonw,2023-03-08T20:17:01Z,2023-03-08T20:17:01Z,OWNER,"I'm going to solve this by using the service name in that `image_id` instead: ```python image_id = f""gcr.io/{project}/{service_name}"" ``` This is a nasty bug, so I'm going to backport it to a `0.64.2` release as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,"`publish cloudrun` reuses image tags, which can lead to very surprising deploy problems", https://github.com/simonw/datasette/issues/2036#issuecomment-1460809643,https://api.github.com/repos/simonw/datasette/issues/2036,1460809643,IC_kwDOBm6k_c5XEi-r,9599,simonw,2023-03-08T20:16:10Z,2023-03-08T20:16:10Z,OWNER,"I think the code at fault is here: https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/datasette/publish/cloudrun.py#L176-L182 That name ends up defaulting to `datasette` - so multiple different projects may end up deploying to the same `image_id`. What I think happened in the `datasette.io` bug is that this workflow: https://github.com/simonw/simonwillisonblog-backup/blob/bfb573e96d8622ab52b22fdcd54724fe6e59fd24/.github/workflows/backup.yml and this workflow: https://github.com/simonw/datasette.io/blob/4676db5bf4a3fc9f792ee270ec0c59eb902cd2c3/.github/workflows/deploy.yml both happened to run at the exact same time. And so the image that was pushed to `gcr.io/datasette-222320/datasette:latest` by the `simonw/simonwillisonblog-backup` action was then deployed by the `simonw/datasette.io/` action, which broke the site.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,"`publish cloudrun` reuses image tags, which can lead to very surprising deploy problems", https://github.com/simonw/datasette/pull/1999#issuecomment-1460760116,https://api.github.com/repos/simonw/datasette/issues/1999,1460760116,IC_kwDOBm6k_c5XEW40,9599,simonw,2023-03-08T19:48:52Z,2023-03-08T19:48:52Z,OWNER,"I'm trying to get `http://127.0.0.1:8001/fixtures/compound_three_primary_keys?_next=a,d,v` to return the correct results.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460759358,https://api.github.com/repos/simonw/datasette/issues/1999,1460759358,IC_kwDOBm6k_c5XEWs-,9599,simonw,2023-03-08T19:48:13Z,2023-03-20T18:47:12Z,OWNER,"Breaking this down into smaller steps: - [x] Get `?_next=` working - [x] Implement extensions - so `.json` is needed again for the JSON version, and anything without an extension is passed through a new code path for HTML - [ ] That HTML view should only access JSON data, which can be seen by using `.context` - this will require a lot of updates to templates (it may be necessary to still provide access to some helper functions though). This will form the basis of the ambition to fully document the template context. - [ ] Get a bunch of the existing table HTML and JSON tests to pass - [ ] Use those tests to refactor the nasty `_next` code, see https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469 - [ ] Figure out how the [register_output_renderer(datasette)](https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette) plugin hook should work","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/issues/2035#issuecomment-1460682625,https://api.github.com/repos/simonw/datasette/issues/2035,1460682625,IC_kwDOBm6k_c5XED-B,9599,simonw,2023-03-08T18:40:57Z,2023-03-08T18:40:57Z,OWNER,Pushed that prototype to a branch: https://github.com/simonw/datasette/commit/0fe844e9adb006a0138e83102ced1329d9155c59 / https://github.com/simonw/datasette/compare/sql-list-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2035#issuecomment-1460679434,https://api.github.com/repos/simonw/datasette/issues/2035,1460679434,IC_kwDOBm6k_c5XEDMK,9599,simonw,2023-03-08T18:39:35Z,2023-03-08T18:39:35Z,OWNER,"I should consider the existing design of magic parameters here: https://docs.datasette.io/en/stable/sql_queries.html#magic-parameters - `_actor_*` - `_header_*` - `_cookie_` - `_now_epoch` - `_now_date_utc` - `_now_datetime_utc` - `_random_chars_*` Should this new `id__list` syntax look more like those magic parameters, or is it OK to use `name__magic` syntax here instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2035#issuecomment-1460668431,https://api.github.com/repos/simonw/datasette/issues/2035,1460668431,IC_kwDOBm6k_c5XEAgP,9599,simonw,2023-03-08T18:35:34Z,2023-03-08T18:35:34Z,OWNER,"To implement this properly need to do the following: - Get the page to display multiple `id: [ text input here ]` fields such that re-submission works - Figure out how this should work for canned queries and for writable canned queries - Tests that cover queries, canned queries, writable canned queries And a bonus feature: what if the Datasette UI layer spotted `:id__list` parameters and used them to add a bit of JavaScript that allowed users to click a `+` button next to an `id` form field to add another one? Also, when a page is re-displayed for on of these queries it could potentially add an extra form field allowing people to add another value. Though this has an annoying problem: how to tell the difference between an additional `id` input field that the user chose not to populate, v.s. one that is supposed to represent an empty string? Maybe only support multiple `id` fields for users with JavaScript in order to avoid this problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2035#issuecomment-1460664619,https://api.github.com/repos/simonw/datasette/issues/2035,1460664619,IC_kwDOBm6k_c5XD_kr,9599,simonw,2023-03-08T18:32:29Z,2023-03-08T18:32:29Z,OWNER,"Got a prototype working: ```diff diff --git a/datasette/views/database.py b/datasette/views/database.py index 8d289105..6f9d8a44 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -226,6 +226,12 @@ class QueryView(DataView): ): db = await self.ds.resolve_database(request) database = db.name + # Disallow x__list query string parameters + invalid_params = [k for k in request.args if k.endswith(""__list"")] + if invalid_params: + raise DatasetteError( + ""Invalid query string parameters: {}"".format("", "".join(invalid_params)) + ) params = {key: request.args.get(key) for key in request.args} if ""sql"" in params: params.pop(""sql"") @@ -258,6 +264,11 @@ class QueryView(DataView): for named_parameter in named_parameters if not named_parameter.startswith(""_"") } + # Handle any __list parameters + for named_parameter in named_parameters: + if named_parameter.endswith(""__list""): + list_values = request.args.getlist(named_parameter[:-6]) + params[named_parameter] = json.dumps(list_values) # Set to blank string if missing from params for named_parameter in named_parameters: ``` This isn't yet doing the right thing on form re-submission: it breaks because it attempts to pass through the `?id__list=` invalid parameter. But I did manage to get it to do this through careful editing of the URL: That was this URL: `http://127.0.0.1:8034/content?sql=select+%3Aid__list%2C*+from+releases+where+id+in+(select+value+from+json_each(%3Aid__list))&id=62642726&id=18402901&id=38714866`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2035#issuecomment-1460659382,https://api.github.com/repos/simonw/datasette/issues/2035,1460659382,IC_kwDOBm6k_c5XD-S2,9599,simonw,2023-03-08T18:28:00Z,2023-03-08T18:28:00Z,OWNER,"Also: `datasette-explain` may need to be updated to understand how to handle this: `ERROR: conn=, sql = 'explain select * from releases where id in (select id from json_each(:id__list))', params = None: You did not supply a value for binding parameter :id__list.` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2035#issuecomment-1460654136,https://api.github.com/repos/simonw/datasette/issues/2035,1460654136,IC_kwDOBm6k_c5XD9A4,9599,simonw,2023-03-08T18:25:46Z,2023-03-08T18:25:46Z,OWNER,"Trickiest part of the implementation here is that it needs to know to output three `id` HTML form fields on the page, such that their values are persisted when the form is submitted a second time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2035#issuecomment-1460639749,https://api.github.com/repos/simonw/datasette/issues/2035,1460639749,IC_kwDOBm6k_c5XD5gF,9599,simonw,2023-03-08T18:17:31Z,2023-03-08T18:17:31Z,OWNER,"Since we are pre-1.0 it's still OK to implement a feature that disallows `?id__list=` in the URL, but allows `:id__list` in SQL queries to reference the JSON list of parameters. So I'm going to prototype this as the `:id__list` feature and see how it feels.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2035#issuecomment-1460637906,https://api.github.com/repos/simonw/datasette/issues/2035,1460637906,IC_kwDOBm6k_c5XD5DS,9599,simonw,2023-03-08T18:16:31Z,2023-03-08T18:16:31Z,OWNER,"I'm pretty sold on this as a feature now. The main question I have is which of these options to implement: 1. `?id=1&?id=2` results in `:id` in the query being `[""1"", ""2""]` - no additional syntax required 2. `:id` in the query continues to reference just the first of those parameters - but `:id__list` (or some other custom syntax) instead gets `[""1"", ""2""]` - or, if the URL is `?id=1` - gets `[""1""]` Actually on writing these out I realize that option 2 is the ONLY valid option. It's no good building a query that works against a JSON list if the user might pass just a single ID, `?id=1`, resulting in their query breaking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2035#issuecomment-1460632758,https://api.github.com/repos/simonw/datasette/issues/2035,1460632758,IC_kwDOBm6k_c5XD3y2,9599,simonw,2023-03-08T18:13:49Z,2023-03-08T18:13:49Z,OWNER,"https://github.com/rclement/datasette-dashboards/issues/54 makes the excellent point that the ` ``` Submitting that form with the middle two options selected navigates to: `https://www.example.com/?id=32&id=15`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2035#issuecomment-1460628199,https://api.github.com/repos/simonw/datasette/issues/2035,1460628199,IC_kwDOBm6k_c5XD2rn,9599,simonw,2023-03-08T18:11:31Z,2023-03-08T18:11:31Z,OWNER,"One variant on this idea: maybe you have to specify in your query that you want it to be the JSON list version, not the single item (first `?id=` parameter version)? Maybe with syntax like this: where id in (select value from json_each(:id__list)) Datasette would automatically pass `{""id"": ""11"", ""id__list"": '[""11"", ""32"", ""62""]'}` as arguments to the `db.execute()` method, if the page was called with `?id=11&id=32&id=62`. This is more explicit, though the syntax is a bit uglier (maybe there's a nicer design for this?). I also worry about `?id__list=` conflicting with this, but I think that's a risk I can take - tell people not to do that, or even block `?id__list=` style parameters entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2035#issuecomment-1460621871,https://api.github.com/repos/simonw/datasette/issues/2035,1460621871,IC_kwDOBm6k_c5XD1Iv,9599,simonw,2023-03-08T18:08:25Z,2023-03-08T18:09:04Z,OWNER,"My current preferred solution is to lean into SQLite's JSON support. What if the query page spotted `?id=11&id=32&id=62` and turned that into a JSON string called `:id:` with a value of `[""11"", ""32"", ""62""]`? Note that this is still a string, not a list. This avoids a nasty problem that occurred in PHP world, where `?id[]=1&id[]=2` would result in an actual PHP array object, which often broke underlying code that had expected `$_GET[""id""]` to be a string, not an array. So in a query you'd be able to do this: where id in (select value from json_each(:id)) And then call it with `?id=11&id=32&id=62`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2035#issuecomment-1460618433,https://api.github.com/repos/simonw/datasette/issues/2035,1460618433,IC_kwDOBm6k_c5XD0TB,9599,simonw,2023-03-08T18:06:34Z,2023-03-08T18:06:34Z,OWNER,"One way to do this would be to dynamically generate the `where id in (?, ?, ?)` with the correct number of question marks, then feed in a list from `request.args.getlist(""id"")` - but that would require rewriting the SQL query text to add those question marks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page, https://github.com/simonw/datasette/issues/2033#issuecomment-1457117383,https://api.github.com/repos/simonw/datasette/issues/2033,1457117383,IC_kwDOBm6k_c5W2djH,9599,simonw,2023-03-06T22:28:55Z,2023-03-06T22:28:55Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugins.html#installing-plugins,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1612296210,`datasette install -r requirements.txt`, https://github.com/simonw/datasette/pull/2031#issuecomment-1456997425,https://api.github.com/repos/simonw/datasette/issues/2031,1456997425,IC_kwDOBm6k_c5W2AQx,9599,simonw,2023-03-06T21:04:27Z,2023-03-06T21:06:34Z,OWNER,"This is a very neat fix, for something I've been wanting for a while. Add a unit test for the row HTML page - I suggest against this page: https://latest.datasette.io/fixtures/foreign_key_references/1 - and I'll land this PR. You can model it on this test here: https://github.com/simonw/datasette/blob/a53b893c46453f35decc8c145c138671cee6140c/tests/test_table_html.py#L609-L632 I think adding it to `test_table_html.py` is OK, even though it's technically for the row page and not the table page. Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1605481359,Expand foreign key references in row view as well, https://github.com/simonw/datasette/pull/2028#issuecomment-1456914694,https://api.github.com/repos/simonw/datasette/issues/2028,1456914694,IC_kwDOBm6k_c5W1sEG,9599,simonw,2023-03-06T20:19:37Z,2023-03-06T20:19:37Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1590839187,add Python 3.11 classifier, https://github.com/simonw/datasette/issues/2024#issuecomment-1426031395,https://api.github.com/repos/simonw/datasette/issues/2024,1426031395,IC_kwDOBm6k_c5U_4Mj,9599,simonw,2023-02-10T16:11:53Z,2023-02-10T16:11:53Z,OWNER,Relevant: https://til.simonwillison.net/sqlite/enabling-wal-mode,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1579973223,Mention WAL mode in documentation, https://github.com/simonw/datasette/issues/262#issuecomment-1423067724,https://api.github.com/repos/simonw/datasette/issues/262,1423067724,IC_kwDOBm6k_c5U0kpM,9599,simonw,2023-02-08T18:33:32Z,2023-02-08T18:36:48Z,OWNER,"Just realized that it's useful to be able to tell what parameters were used to generate a page... but reflecting things like `_next` back in the JSON is confusing in the presence of `next`. So I'm going to add an extra for that information too. Not sure what to call it though: - `params` - confusing because in the code that's usually used for params passed to SQL queries - `query_string` - wouldn't that be a string, not params as a dictionary? I'm going to experiment with a `request` extra that returns some bits of information about the request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/pull/1999#issuecomment-1421988953,https://api.github.com/repos/simonw/datasette/issues/1999,1421988953,IC_kwDOBm6k_c5UwdRZ,9599,simonw,2023-02-08T04:35:44Z,2023-02-08T05:27:48Z,OWNER,"Next step: get `?_next=...` working (it is ignored at the moment, even though the returned JSON includes the `""next""` key). Then... figure out how to render HTML and other requested formats. Then get the tests to pass!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/issues/2019#issuecomment-1421784930,https://api.github.com/repos/simonw/datasette/issues/2019,1421784930,IC_kwDOBm6k_c5Uvrdi,9599,simonw,2023-02-08T01:28:25Z,2023-02-08T01:40:46Z,OWNER,"Rather than duplicate this rather awful hack: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L694-L714 I'm tempted to say that the code that calls the new pagination helper needs to ensure that the `sort` or `sort_desc` columns are selected. If it wants to ditch them later (e.g. because they were not included in `?_col=`) it can do that later once the results have come back.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1421600789,https://api.github.com/repos/simonw/datasette/issues/2019,1421600789,IC_kwDOBm6k_c5Uu-gV,9599,simonw,2023-02-07T23:12:40Z,2023-02-07T23:16:20Z,OWNER,"Most complicated example of a paginated query: https://latest.datasette.io/fixtures?sql=select%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++content%2C%0D%0A++sortable%2C%0D%0A++sortable_with_nulls%2C%0D%0A++sortable_with_nulls_2%2C%0D%0A++text%0D%0Afrom%0D%0A++sortable%0D%0Awhere%0D%0A++(%0D%0A++++sortable_with_nulls+is+null%0D%0A++++and+(%0D%0A++++++(pk1+%3E+%3Ap0)%0D%0A++++++or+(%0D%0A++++++++pk1+%3D+%3Ap0%0D%0A++++++++and+pk2+%3E+%3Ap1%0D%0A++++++)%0D%0A++++)%0D%0A++)%0D%0Aorder+by%0D%0A++sortable_with_nulls+desc%2C%0D%0A++pk1%2C%0D%0A++pk2%0D%0Alimit%0D%0A++101&p0=h&p1=r ```sql select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where ( sortable_with_nulls is null and ( (pk1 > :p0) or ( pk1 = :p0 and pk2 > :p1 ) ) ) order by sortable_with_nulls desc, pk1, pk2 limit 101 ``` Generated by this page: https://latest.datasette.io/fixtures/sortable?_next=%24null%2Ch%2Cr&_sort_desc=sortable_with_nulls The `_next=` parameter there decodes as `$null,h,r` - and those components are tilde-encoded, so this can be distinguished from an actual `$null` value which would be represented as `~24null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1421274434,https://api.github.com/repos/simonw/datasette/issues/2019,1421274434,IC_kwDOBm6k_c5Utu1C,9599,simonw,2023-02-07T18:42:42Z,2023-02-07T18:42:42Z,OWNER,I'm going to build completely separate tests for this in `test_pagination.py`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1420109153,https://api.github.com/repos/simonw/datasette/issues/2019,1420109153,IC_kwDOBm6k_c5UpSVh,9599,simonw,2023-02-07T02:32:36Z,2023-02-07T02:32:36Z,OWNER,"Doing this as a class makes sense to me. There are a few steps: - Instantiate the class with the information it needs, which includes sort order, page size, tiebreaker columns and SQL query and parameters - Generate the new SQL query that will actually be executed - maybe this takes the optional `_next` parameter? This returns the SQL and params that should be executed, where the SQL now includes pagination logic plus order by and limit - The calling code then gets to execute the SQL query to fetch the rows - Last step: those rows are passed to a paginator method which returns `(rows, next)` - where `rows` is the rows truncated to the correct length (really just with the last one cut off if it's too long for the length) and `next` is either `None` or a token, depending on if there should be a next page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1420106315,https://api.github.com/repos/simonw/datasette/issues/2019,1420106315,IC_kwDOBm6k_c5UpRpL,9599,simonw,2023-02-07T02:28:03Z,2023-02-07T02:28:36Z,OWNER,"So I think I can write an abstraction that applies keyset pagination to ANY arbitrary SQL query provided it is given the query, the existing params (so it can pick names for the new params that won't overlap with them), the desired sort order, any existing `_next` token AND the columns that should be used to tie-break any duplicates. Those tie breakers will be either the primary key(s) or `rowid` if none are provided. What about the case of SQL views, where offset/limit should be used instead? I'm inclined to have that as a separate pagination abstraction entirely, with the calling code deciding which pagination helper to use based on if keyset pagination makes sense or not. Might be easier to design a class structure for this starting with `OffsetPaginator`, then using that to inform the design of `KeysetPaginator`. Might put these in `datasette.utils.pagination` to start off with, then maybe extract them out to `sqlite-utils` later once they've proven themselves.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1420104254,https://api.github.com/repos/simonw/datasette/issues/2019,1420104254,IC_kwDOBm6k_c5UpRI-,9599,simonw,2023-02-07T02:24:46Z,2023-02-07T02:24:46Z,OWNER,"Even more complicated: https://latest.datasette.io/fixtures/sortable?sortable_with_nulls__notnull=1&_next=0~2E692704598586882%2Ce%2Cr&_sort=sortable_with_nulls_2 The rewritten SQL for that is: ```sql select * from (select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where ""sortable_with_nulls"" is not null) where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0) or (pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101 ``` And it still has the same number of explain steps as the current SQL witohut the subselect.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1420101175,https://api.github.com/repos/simonw/datasette/issues/2019,1420101175,IC_kwDOBm6k_c5UpQY3,9599,simonw,2023-02-07T02:22:11Z,2023-02-07T02:22:11Z,OWNER,"A more complex example: https://latest.datasette.io/fixtures/sortable?_next=0~2E2650566289400591%2Ca%2Cu&_sort=sortable_with_nulls_2 SQL: ```sql select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0) or (pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101 ``` https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+where+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p0=a&p1=u&p2=0.2650566289400591 Here's the explain: 49 steps long https://latest.datasette.io/fixtures?sql=explain+select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+where+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u Rewritten with a subselect: ```sql select * from ( select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable ) where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0) or (pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101 ``` https://latest.datasette.io/fixtures?sql=select+*+from+(%0D%0A++select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable%0D%0A)%0D%0Awhere+(sortable_with_nulls_2+%3E+%3Ap2+or+(sortable_with_nulls_2+%3D+%3Ap2+and+((pk1+%3E+%3Ap0)%0D%0A++or%0D%0A(pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1))))+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u And here's the explain for that - also 49 steps: https://latest.datasette.io/fixtures?sql=explain+select+*+from+%28%0D%0A++select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable%0D%0A%29%0D%0Awhere+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1420094396,https://api.github.com/repos/simonw/datasette/issues/2019,1420094396,IC_kwDOBm6k_c5UpOu8,9599,simonw,2023-02-07T02:18:11Z,2023-02-07T02:19:16Z,OWNER,"For the SQL underlying this page (the second page in that compound primary key paginated sequence): https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv The explain for the default query: https://latest.datasette.io/fixtures?sql=explain+select%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%2C%0D%0A++content%0D%0Afrom%0D%0A++compound_three_primary_keys%0D%0Awhere%0D%0A++%28%0D%0A++++%28pk1+%3E+%3Ap0%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3E+%3Ap1%0D%0A++++%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3D+%3Ap1%0D%0A++++++and+pk3+%3E+%3Ap2%0D%0A++++%29%0D%0A++%29%0D%0Aorder+by%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%0D%0Alimit%0D%0A++101&p0=a&p1=d&p2=v The explain for that query rewritten as this: ```sql explain select * from ( select pk1, pk2, pk3, content from compound_three_primary_keys ) where ( (pk1 > :p0) or ( pk1 = :p0 and pk2 > :p1 ) or ( pk1 = :p0 and pk2 = :p1 and pk3 > :p2 ) ) order by pk1, pk2, pk3 limit 101 ``` https://latest.datasette.io/fixtures?sql=explain+select+*+from+%28select+%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%2C%0D%0A++content%0D%0Afrom%0D%0A++compound_three_primary_keys%0D%0A%29%0D%0A++where%0D%0A++%28%0D%0A++++%28pk1+%3E+%3Ap0%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3E+%3Ap1%0D%0A++++%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3D+%3Ap1%0D%0A++++++and+pk3+%3E+%3Ap2%0D%0A++++%29%0D%0A++%29%0D%0Aorder+by%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%0D%0Alimit%0D%0A++101&p0=a&p1=d&p2=v Both explains have 31 steps and look pretty much identical.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1420088670,https://api.github.com/repos/simonw/datasette/issues/2019,1420088670,IC_kwDOBm6k_c5UpNVe,9599,simonw,2023-02-07T02:14:35Z,2023-02-07T02:14:35Z,OWNER,"Maybe the correct level of abstraction here is that pagination is something that happens to a SQL query that is defined as SQL and params, without an order by or limit. That's then wrapped in a sub-select and those things are added to it, plus the necessary `where` clauses depending on the page. Need to check that the query plan for pagination of a subquery isn't slower than the plan for pagination as it works today.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1419953256,https://api.github.com/repos/simonw/datasette/issues/2019,1419953256,IC_kwDOBm6k_c5UosRo,9599,simonw,2023-02-06T23:42:56Z,2023-02-06T23:43:10Z,OWNER,"Relevant issue: - https://github.com/simonw/datasette/issues/1773 Explains this comment: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L697 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1419928455,https://api.github.com/repos/simonw/datasette/issues/2019,1419928455,IC_kwDOBm6k_c5UomOH,9599,simonw,2023-02-06T23:21:50Z,2023-02-06T23:21:50Z,OWNER,"Found more logic relating to this: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L684-L732","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1419921228,https://api.github.com/repos/simonw/datasette/issues/2019,1419921228,IC_kwDOBm6k_c5UokdM,9599,simonw,2023-02-06T23:14:15Z,2023-02-06T23:14:15Z,OWNER,Crucial utility function: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/utils/__init__.py#L137-L160,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1419917661,https://api.github.com/repos/simonw/datasette/issues/2019,1419917661,IC_kwDOBm6k_c5Uojld,9599,simonw,2023-02-06T23:10:51Z,2023-02-06T23:10:51Z,OWNER,"I should turn `sort` and `sort_desc` into an object representing the sort order earlier in the code. I should also create something that bundles together `pks` and `use_rowid` and maybe `is_view` as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/2019#issuecomment-1419916684,https://api.github.com/repos/simonw/datasette/issues/2019,1419916684,IC_kwDOBm6k_c5UojWM,9599,simonw,2023-02-06T23:09:51Z,2023-02-06T23:10:13Z,OWNER,"The inputs and outputs for this are pretty complex. Inputs: - `?_next=` from the query string - `is_view` - is this for a table or view? If it's a view it uses offset/limit pagination - which could actually work for arbitrary queries too. Also views could have keyset pagination if they are known to be sorted by a particular column. - `sort` and `sort_desc` reflecting the current sort order - `use_rowid` for if the table is a rowid table with no primary key of its own - `pks` - the primary keys for the table - `params` - the current set of parameters, I think used just to count their length so new params can be added as `p5` etc without collisions. This could be handled with a `s0`, `s1` etc naming convention instead. Outputs: - `where_clauses` - a list of where clauses to add to the query - `params` - additional parameters to use with the query due to the new where clauses - `order_by` - the order by clause","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code, https://github.com/simonw/datasette/issues/262#issuecomment-1418288327,https://api.github.com/repos/simonw/datasette/issues/262,1418288327,IC_kwDOBm6k_c5UiVzH,9599,simonw,2023-02-05T22:57:58Z,2023-02-06T23:01:15Z,OWNER,"I think that does make sense: `?_extra=table` perhaps, which would add `{""table"": ""...""}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/issues/2016#issuecomment-1418288077,https://api.github.com/repos/simonw/datasette/issues/2016,1418288077,IC_kwDOBm6k_c5UiVvN,9599,simonw,2023-02-05T22:56:43Z,2023-02-05T22:56:43Z,OWNER,"This absolutely makes sense. One of the biggest goals for Datasette 1.0 is ""documented template contexts"" - for any default template in Datasette that people might want to over-ride there should be documentation that describes the available context variables, plus tests that ensure they don't accidentally get broken by future changes. Ensuring description/title/etc are available on the index page feels like it fits well into that bucket.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1571207083,Database metadata fields like description are not available in the index page template's context, https://github.com/simonw/datasette/issues/2011#issuecomment-1410827249,https://api.github.com/repos/simonw/datasette/issues/2011,1410827249,IC_kwDOBm6k_c5UF4Px,9599,simonw,2023-01-31T17:58:54Z,2023-01-31T17:58:54Z,OWNER,"I think this is the relevant code: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/facets.py#L260-L268","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1564769997,"Applied facet did not result in an ""x"" icon to dismiss it", https://github.com/simonw/datasette/issues/2010#issuecomment-1409406327,https://api.github.com/repos/simonw/datasette/issues/2010,1409406327,IC_kwDOBm6k_c5UAdV3,9599,simonw,2023-01-30T21:51:58Z,2023-01-30T21:51:58Z,OWNER,"Here's a quick prototype I knocked up for this: ```diff diff --git a/datasette/static/app.css b/datasette/static/app.css index 71437bd4..d763bcff 100644 --- a/datasette/static/app.css +++ b/datasette/static/app.css @@ -695,7 +695,48 @@ p.zero-results { +/* Force table to not be like tables anymore */ +body.row table.rows-and-columns, +body.row .rows-and-columns thead, +body.row .rows-and-columns tbody, +body.row .rows-and-columns th, +body.row .rows-and-columns td, +body.row .rows-and-columns tr { + display: block; +} + +/* Hide table headers (but not display: none;, for accessibility) */ +body.row .rows-and-columns thead tr { + position: absolute; + top: -9999px; + left: -9999px; +} + +body.row .rows-and-columns tr { + border: 1px solid #ccc; + margin-bottom: 1em; + border-radius: 10px; + background-color: white; + padding: 0.2rem; +} +body.row .rows-and-columns td { + /* Behave like a ""row"" */ + border: none; + border-bottom: 1px solid #eee; + padding: 0; + padding-left: 10%; + padding-bottom: 0.3em; +} + +body.row .rows-and-columns td:before { + display: block; + color: black; + padding-bottom: 0.2em; + font-size: 0.8em; + font-weight: bold; + background-color: #f5f5f5; +} /* Overrides ===============================================================*/ diff --git a/datasette/templates/row.html b/datasette/templates/row.html index 1d1b0bfd..339eb643 100644 --- a/datasette/templates/row.html +++ b/datasette/templates/row.html @@ -5,6 +5,9 @@ {% block extra_head %} {{- super() -}} +{% endblock %} + {% block content %}

Create an API token

-

This token will allow API access with the same abilities as your current user.

+

This token will allow API access with the same abilities as your current user, {{ request.actor.id }}

{% if errors %} {% for error in errors %} @@ -27,8 +36,39 @@ - + +
+ Restrict actions that can be performed using this token +

All databases and tables

+
    + {% for permission in all_permissions %} +
    • + {% endfor %} +
+ + {% for database in databases %} +

All tables in database: {{ database }}

+
    + {% for permission in database_permissions %} +
    • + {% endfor %} +
+ {% endfor %} +

Specific tables

+ {% for dbt in database_with_tables %} + {% for table in dbt.tables %} +

{{ dbt.database }}: {{ table }}

+
    + {% for permission in table_permissions %} +
    • + {% endfor %} +
+ {% endfor %} + {% endfor %} +
+ + {% if token %}
diff --git a/datasette/views/special.py b/datasette/views/special.py index 30345d14..48357f87 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -231,12 +231,37 @@ class CreateTokenView(BaseView): return await self.render( [""create_token.html""], request, - {""actor"": request.actor}, + { + ""actor"": request.actor, + ""all_permissions"": self.ds.permissions.keys(), + ""database_permissions"": [ + key + for key, value in self.ds.permissions.items() + if value.takes_database + ], + ""table_permissions"": [ + key + for key, value in self.ds.permissions.items() + if value.takes_resource + ], + ""databases"": [k for k in self.ds.databases.keys() if k != ""_internal""], + ""database_with_tables"": [ + { + ""database"": db.name, + ""tables"": await db.table_names(), + } + for db in self.ds.databases.values() + if db.name != ""_internal"" + ], + }, ) async def post(self, request): self.check_permission(request) post = await request.post_vars() + from pprint import pprint + + pprint(post) errors = [] duration = None if post.get(""expire_type""): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1947#issuecomment-1350013016,https://api.github.com/repos/simonw/datasette/issues/1947,1350013016,IC_kwDOBm6k_c5Qd5BY,9599,simonw,2022-12-13T23:16:24Z,2022-12-13T23:17:17Z,OWNER,"Slightly tricky thing here is that it should only show permissions that the user themselves has - on databases and tables that they have permission to access. I have a nasty feeling this may require looping through _everything_ and running every permission check, which could get very expensive if there are plugins involved that do their own storage check to resolve a permission. It's that classic permission system problem: how to efficiently iterate through everything the user has permission to do in one go? Might be that I have to punt on that, and show the user a list of permissions to select that they might not actually have ability for.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1947#issuecomment-1350008636,https://api.github.com/repos/simonw/datasette/issues/1947,1350008636,IC_kwDOBm6k_c5Qd388,9599,simonw,2022-12-13T23:14:33Z,2022-12-13T23:14:33Z,OWNER,"Checkbox interface looks like this. It's not beautiful but it's good enough for the moment: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1947#issuecomment-1350002434,https://api.github.com/repos/simonw/datasette/issues/1947,1350002434,IC_kwDOBm6k_c5Qd2cC,9599,simonw,2022-12-13T23:11:50Z,2022-12-13T23:11:59Z,OWNER,"I think checkboxes will work well. Here's the data I get back from them (as `post_vars()`): ``` {'all:debug-menu': 'on', 'all:insert-row': 'on', 'expire_duration': '', 'expire_type': '', 'table:fixtures:delete-row': 'on', 'table:fixtures:drop-table': 'on', 'table:fixtures:view-query': 'on'} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1947#issuecomment-1349975255,https://api.github.com/repos/simonw/datasette/issues/1947,1349975255,IC_kwDOBm6k_c5QdvzX,9599,simonw,2022-12-13T23:00:11Z,2022-12-13T23:00:11Z,OWNER,"My `
+ +
+ Restrict actions that can be performed using this token +

Restrict actions that can be performed using this token:

+

All databases and tables:

+

+
+ {% if token %} diff --git a/datasette/views/special.py b/datasette/views/special.py index 30345d14..9d0fcd31 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -231,7 +231,17 @@ class CreateTokenView(BaseView): return await self.render( [""create_token.html""], request, - {""actor"": request.actor}, + { + ""actor"": request.actor, + ""all_permissions"": self.ds.permissions.keys(), + ""database_permissions"": [key for key, value in self.ds.permissions.items() if value.takes_database], + ""table_permissions"": [key for key, value in self.ds.permissions.items() if value.takes_resource], + ""databases"": self.ds.databases.keys(), + ""database_with_tables"": [{ + ""database"": db.name, + ""tables"": await db.table_names(), + } for db in self.ds.databases.values()], + }, ) async def post(self, request): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1947#issuecomment-1349974287,https://api.github.com/repos/simonw/datasette/issues/1947,1349974287,IC_kwDOBm6k_c5QdvkP,9599,simonw,2022-12-13T22:59:44Z,2022-12-13T22:59:44Z,OWNER,"Got an option group thing working: But... it strikes me that any time you're considering a `` for this. The usability for keyboards is still pretty awful, but it's a niche enough feature that maybe that's OK for the moment? ```javascript var select = document.querySelector('select'); var selected = Array.from(temp0.options).filter(o => o.selected).map(o => o.value) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1950#issuecomment-1349864950,https://api.github.com/repos/simonw/datasette/issues/1950,1349864950,IC_kwDOBm6k_c5QdU32,9599,simonw,2022-12-13T22:11:15Z,2022-12-13T22:11:15Z,OWNER,"Most places I use that exception at the moment set their own non-500 status error: ``` datasette % rg DatasetteError datasette/handle_exception.py 7:from .views.base import DatasetteError 33: elif isinstance(exception, DatasetteError): datasette/filters.py 2:from datasette.views.base import DatasetteError 22: raise DatasetteError(""_where= is not allowed"", status=403) 141: raise DatasetteError( datasette/views/table.py 34:from .base import BaseView, DataView, DatasetteError, ureg, _error 178: raise DatasetteError( 192: raise DatasetteError( 390: raise DatasetteError(""Cannot use _sort and _sort_desc at the same time"") 394: raise DatasetteError(f""Cannot sort table by {sort}"") 400: raise DatasetteError(f""Cannot sort table by {sort_desc}"") datasette/views/base.py 39:class DatasetteError(Exception): 219: raise DatasetteError(str(e), title=""Invalid SQL"", status=400) 222: raise DatasetteError(str(e)) 224: except DatasetteError: 382: raise DatasetteError( 402: raise DatasetteError(str(e), title=""Invalid SQL"", status=400) 405: raise DatasetteError(str(e)) 407: except DatasetteError: datasette/views/table2.py 28:from .base import DataView, DatasetteError, ureg 296: raise DatasetteError( 310: raise DatasetteError( 472: raise DatasetteError(""Cannot use _sort and _sort_desc at the same time"") 476: raise DatasetteError(f""Cannot sort table by {sort}"") 482: raise DatasetteError(f""Cannot sort table by {sort_desc}"") datasette/views/database.py 31:from .base import BaseView, DatasetteError, DataView, _error 188: raise DatasetteError(""Invalid database"", status=404) 190: raise DatasetteError(""Cannot download in-memory databases"", status=404) 194: raise DatasetteError(""Cannot download database"", status=404) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1495241162,"Bad ?_sort returns a 500 error, should be a 400", https://github.com/simonw/datasette/issues/1950#issuecomment-1349855620,https://api.github.com/repos/simonw/datasette/issues/1950,1349855620,IC_kwDOBm6k_c5QdSmE,9599,simonw,2022-12-13T22:08:50Z,2022-12-13T22:08:50Z,OWNER,https://github.com/simonw/datasette/blob/d4b98d3924dec625a99236e65b1b169ff957381f/datasette/views/table.py#L392-L400,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1495241162,"Bad ?_sort returns a 500 error, should be a 400", https://github.com/simonw/datasette/issues/1914#issuecomment-1347801679,https://api.github.com/repos/simonw/datasette/issues/1914,1347801679,IC_kwDOBm6k_c5QVdJP,9599,simonw,2022-12-13T06:15:54Z,2022-12-13T06:15:54Z,OWNER,"Should make sure that every API that returns an object as the top level (that's almost all of them) includes `""ok"": true` to indicate no errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468689139,Finalize design of JSON for Datasette 1.0, https://github.com/simonw/datasette/issues/1947#issuecomment-1347775760,https://api.github.com/repos/simonw/datasette/issues/1947,1347775760,IC_kwDOBm6k_c5QVW0Q,9599,simonw,2022-12-13T05:38:47Z,2022-12-13T05:38:47Z,OWNER,I'm going to hide the options for reducing the scope of the token inside a details/summary element.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1937#issuecomment-1347770871,https://api.github.com/repos/simonw/datasette/issues/1937,1347770871,IC_kwDOBm6k_c5QVVn3,9599,simonw,2022-12-13T05:30:43Z,2022-12-13T05:30:43Z,OWNER,"Also you should need `update-row` permission to use the `""replace"": true` option - I should add that rule to `/-/insert` add well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1483320357,/db/-/create API should require insert-rows permission to use row: or rows: option, https://github.com/simonw/datasette/issues/1947#issuecomment-1347768549,https://api.github.com/repos/simonw/datasette/issues/1947,1347768549,IC_kwDOBm6k_c5QVVDl,9599,simonw,2022-12-13T05:25:56Z,2022-12-13T22:29:12Z,OWNER,- [x] I should add a `--database` example to that help text.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1947#issuecomment-1347768328,https://api.github.com/repos/simonw/datasette/issues/1947,1347768328,IC_kwDOBm6k_c5QVVAI,9599,simonw,2022-12-13T05:25:31Z,2022-12-13T22:25:46Z,OWNER,"https://latest.datasette.io/-/create-token currently looks like this: ![Image](https://user-images.githubusercontent.com/9599/207458002-7c46940b-22c0-45d3-a668-ec7f1082588c.png) As a reminder, the CLI options that this needs to provide an alternative to are: https://github.com/simonw/datasette/blob/d4b98d3924dec625a99236e65b1b169ff957381f/docs/cli-reference.rst#L619-L638","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/pull/1938#issuecomment-1347767048,https://api.github.com/repos/simonw/datasette/issues/1938,1347767048,IC_kwDOBm6k_c5QVUsI,9599,simonw,2022-12-13T05:23:18Z,2022-12-13T05:23:18Z,OWNER,"I landed this already: - #1636 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485488236,"""permissions"" blocks in metadata.json/yaml", https://github.com/simonw/datasette/issues/1948#issuecomment-1347766530,https://api.github.com/repos/simonw/datasette/issues/1948,1347766530,IC_kwDOBm6k_c5QVUkC,9599,simonw,2022-12-13T05:22:19Z,2022-12-13T05:22:19Z,OWNER,"I tested: ``` {""id"": ""root"", ""_r"": {""a"": ""view-table""}} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493404423,500 error on permission debug page when testing actors with _r, https://github.com/simonw/datasette/issues/1855#issuecomment-1347761892,https://api.github.com/repos/simonw/datasette/issues/1855,1347761892,IC_kwDOBm6k_c5QVTbk,9599,simonw,2022-12-13T05:14:25Z,2022-12-13T05:14:25Z,OWNER,New documentation: https://docs.datasette.io/en/latest/authentication.html#restricting-the-actions-that-a-token-can-perform,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1947#issuecomment-1347760109,https://api.github.com/repos/simonw/datasette/issues/1947,1347760109,IC_kwDOBm6k_c5QVS_t,9599,simonw,2022-12-13T05:12:00Z,2022-12-13T05:12:00Z,OWNER,"For the UI: I think I'm going to dump a whole bunch of form elements on the page (so you can set up to 3 of each category of limit without any JavaScript), then add JavaScript that hides all but one of the options and gives you a ""add another"" widget that adds multiple more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1855#issuecomment-1347759522,https://api.github.com/repos/simonw/datasette/issues/1855,1347759522,IC_kwDOBm6k_c5QVS2i,9599,simonw,2022-12-13T05:11:43Z,2022-12-13T05:11:43Z,OWNER,"Decided to do the `/-/create-token` UI in a separate ticket: - #1947","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1946#issuecomment-1347733217,https://api.github.com/repos/simonw/datasette/issues/1946,1347733217,IC_kwDOBm6k_c5QVMbh,9599,simonw,2022-12-13T04:28:45Z,2022-12-13T04:28:45Z,OWNER,"Demo of the new feature: ``` % datasette create-token --secret s root dstok_eyJhIjoicm9vdCIsInRva2VuIjoiZHN0b2siLCJ0IjoxNjcwOTA1NjgwfQ.pqSWOwCSNp678hEWl9l5o7m1GaM % datasette --get /-/actor.json {""actor"": null} % DATASETTE_SECRET=s datasette --get /-/actor.json --token dstok_eyJhIjoicm9vdCIsInRva2VuIjoiZHN0b2siLCJ0IjoxNjcwOTA1NjgwfQ.pqSWOwCSNp678hEWl9l5o7m1GaM {""actor"": {""id"": ""root"", ""token"": ""dstok""}} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493339206,`datasette --get` mechanism for sending tokens, https://github.com/simonw/datasette/issues/1946#issuecomment-1347732039,https://api.github.com/repos/simonw/datasette/issues/1946,1347732039,IC_kwDOBm6k_c5QVMJH,9599,simonw,2022-12-13T04:26:20Z,2022-12-13T04:26:20Z,OWNER,"Two options: - `--header ""Authorization: Bearer XXX""` which can be used to send any headers - `--token XXX` to specify the token, which is then sent using that header I like the second option more, simply because there are currently no other headers that affect how Datasette works. `--token` feels obvious and easy to use.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493339206,`datasette --get` mechanism for sending tokens, https://github.com/simonw/datasette/issues/1855#issuecomment-1347731288,https://api.github.com/repos/simonw/datasette/issues/1855,1347731288,IC_kwDOBm6k_c5QVL9Y,9599,simonw,2022-12-13T04:24:50Z,2022-12-13T04:24:50Z,OWNER,For the tests for `datasette create-token` it would be useful if `datasette --get` had a mechanism for sending an `Authorization: Bearer X` header.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1855#issuecomment-1347726302,https://api.github.com/repos/simonw/datasette/issues/1855,1347726302,IC_kwDOBm6k_c5QVKve,9599,simonw,2022-12-13T04:16:26Z,2022-12-13T04:16:26Z,OWNER,I'm going to move this code into `datasette/cli.py` - it's a bit unexpected having it live in `default_permissions.py` like this (I couldn't find the code when I went looking for it earlier).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1855#issuecomment-1347707683,https://api.github.com/repos/simonw/datasette/issues/1855,1347707683,IC_kwDOBm6k_c5QVGMj,9599,simonw,2022-12-13T03:55:35Z,2022-12-13T04:15:27Z,OWNER,"Help looks like this: ``` Usage: datasette create-token [OPTIONS] ID Create a signed API token for the specified actor ID Example: datasette create-token root --secret mysecret To only allow create-table: datasette create-token root --secret mysecret \ --all create-table Or to only allow insert-row against a specific table: datasette create-token root --secret myscret \ --resource mydb mytable insert-row Restricted actions can be specified multiple times using multiple --all, --database, and --resource options. Add --debug to see a decoded version of the token. Options: --secret TEXT Secret used for signing the API tokens [required] -e, --expires-after INTEGER Token should expire after this many seconds -a, --all ACTION Restrict token to this action -d, --database DB ACTION Restrict token to this action on this database -r, --resource DB RESOURCE ACTION Restrict token to this action on this database resource (a table, SQL view or named query) --debug Show decoded token --plugins-dir DIRECTORY Path to directory containing custom plugins --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1855#issuecomment-1347695728,https://api.github.com/repos/simonw/datasette/issues/1855,1347695728,IC_kwDOBm6k_c5QVDRw,9599,simonw,2022-12-13T03:30:09Z,2022-12-13T03:30:09Z,OWNER,"I just noticed this in the existing code: https://github.com/simonw/datasette/blob/c5d30b58a1cd1c66bbddcf3561db005543ecaf25/datasette/default_permissions.py#L195-L203 Hard-coding those action names should not be necessary any more, especially now we have `datasette.permissions` for looking up metadata about the permissions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1855#issuecomment-1347694871,https://api.github.com/repos/simonw/datasette/issues/1855,1347694871,IC_kwDOBm6k_c5QVDEX,9599,simonw,2022-12-13T03:28:15Z,2022-12-13T03:28:15Z,OWNER,"Initial prototype of the `create-token` command changes: ```diff diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 406dae40..bbe1247e 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -278,17 +278,55 @@ def register_commands(cli): help=""Token should expire after this many seconds"", type=int, ) + @click.option( + ""alls"", + ""-a"", + ""--all"", + type=str, + multiple=True, + help=""Restrict token to this permission"", + ) + @click.option( + ""databases"", + ""-d"", + ""--database"", + type=(str, str), + multiple=True, + help=""Restrict token to this permission on this database"", + ) + @click.option( + ""resources"", + ""-r"", + ""--resource"", + type=(str, str, str), + multiple=True, + help=""Restrict token to this permission on this database resource (a table, SQL view or named query)"", + ) @click.option( ""--debug"", help=""Show decoded token"", is_flag=True, ) - def create_token(id, secret, expires_after, debug): + def create_token(id, secret, expires_after, alls, databases, resources, debug): ""Create a signed API token for the specified actor ID"" ds = Datasette(secret=secret) bits = {""a"": id, ""token"": ""dstok"", ""t"": int(time.time())} if expires_after: bits[""d""] = expires_after + if alls or databases or resources: + bits[""_r""] = {} + if alls: + bits[""_r""][""a""] = list(alls) + if databases: + bits[""_r""][""d""] = {} + for database, action in databases: + bits[""_r""][""d""].setdefault(database, []).append(action) + if resources: + bits[""_r""][""r""] = {} + for database, table, action in resources: + bits[""_r""][""r""].setdefault(database, {}).setdefault( + table, [] + ).append(action) token = ds.sign(bits, namespace=""token"") click.echo(""dstok_{}"".format(token)) if debug: ``` Still needs tests, plus I'd like it to use abbreviations if available to keep the token length shorter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1855#issuecomment-1347693620,https://api.github.com/repos/simonw/datasette/issues/1855,1347693620,IC_kwDOBm6k_c5QVCw0,9599,simonw,2022-12-13T03:25:41Z,2022-12-13T03:25:41Z,OWNER,"I'm going to rename ""t"" in the magic format to ""r"" for resource.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1855#issuecomment-1347675456,https://api.github.com/repos/simonw/datasette/issues/1855,1347675456,IC_kwDOBm6k_c5QU-VA,9599,simonw,2022-12-13T02:57:46Z,2022-12-13T02:57:46Z,OWNER,"I was going to have the CLI command throw an error if you attempt to use a permission that isn't registered with Datasette, but then I remembered that one of the uses for the CLI tool is to create signed tokens that will work against other Datasette instances (via the `--secret` option) that might have different plugins installed that register different permission names. So I might have it output warnings instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1855#issuecomment-1347669087,https://api.github.com/repos/simonw/datasette/issues/1855,1347669087,IC_kwDOBm6k_c5QU8xf,9599,simonw,2022-12-13T02:45:15Z,2022-12-13T02:45:15Z,OWNER,The hardest piece here is the UI. I'm going to implement the CLI command first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1636#issuecomment-1347655074,https://api.github.com/repos/simonw/datasette/issues/1636,1347655074,IC_kwDOBm6k_c5QU5Wi,9599,simonw,2022-12-13T02:21:04Z,2022-12-13T02:21:23Z,OWNER,"The thing I'm stuck on at the moment is how to implement it such that an `allow` block for `create-table` at the root of the metadata will be checked correctly. Maybe the algorithm when `_resolve_metadata_permissions_blocks(datasette, actor, action, resource)` is called should do this: 1. If a root permission block matching that action exists, test with that 2. Next, if resource has been passed, check at the database level 3. If the resource included a table/query, check at that level too So everything is keyed off the incoming `action` name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042,"""permissions"" propery in metadata for configuring arbitrary permissions", https://github.com/simonw/datasette/issues/1636#issuecomment-1347648326,https://api.github.com/repos/simonw/datasette/issues/1636,1347648326,IC_kwDOBm6k_c5QU3tG,9599,simonw,2022-12-13T02:10:02Z,2022-12-13T02:10:02Z,OWNER,"The implementation for this will go here: https://github.com/simonw/datasette/blob/8bf06a76b51bc9ace7cf72cf0cca8f1da7704ea7/datasette/default_permissions.py#L81-L83 Here's the start of the tests (currently marked as `xfail`): https://github.com/simonw/datasette/blob/8bf06a76b51bc9ace7cf72cf0cca8f1da7704ea7/tests/test_permissions.py#L652-L689","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042,"""permissions"" propery in metadata for configuring arbitrary permissions", https://github.com/simonw/datasette/issues/1636#issuecomment-1347647298,https://api.github.com/repos/simonw/datasette/issues/1636,1347647298,IC_kwDOBm6k_c5QU3dC,9599,simonw,2022-12-13T02:08:46Z,2022-12-13T02:08:46Z,OWNER,"A bunch of the work for this just landed - in particular the new scheme is now documented (even though it doesn't work yet): https://docs.datasette.io/en/latest/authentication.html#other-permissions-in-metadata","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042,"""permissions"" propery in metadata for configuring arbitrary permissions", https://github.com/simonw/datasette/issues/1939#issuecomment-1347646516,https://api.github.com/repos/simonw/datasette/issues/1939,1347646516,IC_kwDOBm6k_c5QU3Q0,9599,simonw,2022-12-13T02:07:50Z,2022-12-13T02:07:50Z,OWNER,Documentation for the new hook: https://docs.datasette.io/en/latest/plugin_hooks.html#register-permissions-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1943#issuecomment-1347645615,https://api.github.com/repos/simonw/datasette/issues/1943,1347645615,IC_kwDOBm6k_c5QU3Cv,9599,simonw,2022-12-13T02:06:47Z,2022-12-13T02:06:47Z,OWNER,This URL is already used for the https://latest.datasette.io/-/permissions tool - but it could include a block on that page that tells you what permissions are available.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1490576818,`/-/permissions` should list available permissions, https://github.com/simonw/datasette/pull/1940#issuecomment-1347640542,https://api.github.com/repos/simonw/datasette/issues/1940,1347640542,IC_kwDOBm6k_c5QU1ze,9599,simonw,2022-12-13T02:02:10Z,2022-12-13T02:02:10Z,OWNER,"This PR ended up bundling part of the implementation of: - #1636 I'm going to be bad an NOT untangle that from this before I merge it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1486011362,register_permissions() plugin hook, https://github.com/simonw/datasette/pull/1940#issuecomment-1347634128,https://api.github.com/repos/simonw/datasette/issues/1940,1347634128,IC_kwDOBm6k_c5QU0PQ,9599,simonw,2022-12-13T01:51:56Z,2022-12-13T01:51:56Z,OWNER,Actually one last thing: I said that the error would only occur if the permissions differed in some way.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1486011362,register_permissions() plugin hook, https://github.com/simonw/datasette/pull/1940#issuecomment-1347620733,https://api.github.com/repos/simonw/datasette/issues/1940,1347620733,IC_kwDOBm6k_c5QUw99,9599,simonw,2022-12-13T01:33:06Z,2022-12-13T01:33:06Z,OWNER,"It's this change which triggers the failures: ```diff diff --git a/datasette/app.py b/datasette/app.py index 760063d5..defa9688 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -707,9 +707,12 @@ class Datasette: ) return crumbs - async def permission_allowed(self, actor, action, resource=None, default=False): + async def permission_allowed(self, actor, action, resource=None, default=None): """"""Check permissions using the permissions_allowed plugin hook"""""" result = None + # Use default from registered permission, if available + if default is None and action in self.permissions: + default = self.permissions[action].default for check in pm.hook.permission_allowed( datasette=self, actor=actor, ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1486011362,register_permissions() plugin hook, https://github.com/simonw/datasette/pull/1940#issuecomment-1347616055,https://api.github.com/repos/simonw/datasette/issues/1940,1347616055,IC_kwDOBm6k_c5QUv03,9599,simonw,2022-12-13T01:27:03Z,2022-12-13T01:27:03Z,OWNER,"I'm going to revert that last commit, see if I can get the tests running again and then apply the changes a line at a time to figure out which ones broke things.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1486011362,register_permissions() plugin hook, https://github.com/simonw/datasette/pull/1940#issuecomment-1345701316,https://api.github.com/repos/simonw/datasette/issues/1940,1345701316,IC_kwDOBm6k_c5QNcXE,9599,simonw,2022-12-12T00:10:59Z,2022-12-12T00:10:59Z,OWNER,"Here's my first test failure: ``` tests/test_permissions.py .......F >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow = {}, expected_anon = 403, expected_auth = 403, path = '/fixtures/compound_three_primary_keys' padlock_client = @pytest.mark.parametrize( ""allow,expected_anon,expected_auth"", [ (None, 200, 200), ({}, 403, 403), ({""id"": ""root""}, 403, 200), ], ) @pytest.mark.parametrize( ""path"", ( ""/"", ""/fixtures"", ""/fixtures/compound_three_primary_keys"", ""/fixtures/compound_three_primary_keys/a,a,a"", ""/fixtures/two"", # Query ), ) def test_view_padlock(allow, expected_anon, expected_auth, path, padlock_client): padlock_client.ds._metadata_local[""allow""] = allow fragment = ""🔒"" anon_response = padlock_client.get(path) > assert expected_anon == anon_response.status E assert 403 == 200 E + where 200 = .status /Users/simon/Dropbox/Development/datasette/tests/test_permissions.py:61: AssertionError >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > /Users/simon/Dropbox/Development/datasette/tests/test_permissions.py(61)test_view_padlock() -> assert expected_anon == anon_response.status (Pdb) anon_response (Pdb) anon_response.status 200 (Pdb) path '/fixtures/compound_three_primary_keys' (Pdb) padlock_client.ds._metadata_ *** AttributeError: 'Datasette' object has no attribute '_metadata_' (Pdb) padlock_client.ds._metadata_local {'databases': {'fixtures': {'queries': {'two': {'sql': 'select 1 + 1', 'name': 'two'}, 'from_async_hook': {'sql': 'select 2', 'name': 'from_async_hook'}, 'from_hook': {'sql': ""select 1, 'null' as actor_id"", 'name': 'from_hook'}}, 'source': None, 'source_url': None, 'license': None, 'license_url': None, 'about': None, 'about_url': None}}, 'allow': {}} (Pdb) allow {} ``` It looks like I've broken the `allow` logic that notices that if there's an `""allow"": {}` on the root then anonymous users should not be allowed to view any pages.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1486011362,register_permissions() plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1345691103,https://api.github.com/repos/simonw/datasette/issues/1939,1345691103,IC_kwDOBm6k_c5QNZ3f,9599,simonw,2022-12-11T23:37:49Z,2022-12-11T23:37:49Z,OWNER,"Idea: a `/-/permissions` introspection endpoint for listing registered permissions ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/sqlite-utils/issues/517#issuecomment-1344965367,https://api.github.com/repos/simonw/sqlite-utils/issues/517,1344965367,IC_kwDOCGYnMM5QKor3,9599,simonw,2022-12-10T01:26:31Z,2022-12-10T01:26:31Z,OWNER,At some point I should drop it from all of these other projects too: https://cs.github.com/?scopeName=All+repos&scope=&q=user%3Asimonw+%223.6%22+path%3A.github%2Fworkflows%2F*,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1487757143,Drop support for Python 3.6, https://github.com/simonw/datasette/issues/1942#issuecomment-1344959032,https://api.github.com/repos/simonw/datasette/issues/1942,1344959032,IC_kwDOBm6k_c5QKnI4,9599,simonw,2022-12-10T01:11:26Z,2022-12-10T01:11:26Z,OWNER,"One way this could work: if plugins request it, a block like this is added to the page: ```html ``` Then a function could be provided which extracts and parses that data: ```javascript var data = await datasette.jsonData(); ``` Why an `await`? Because then I could have it work the exact same way if the data is NOT available on the page - it could trigger a `fetch()` call for the same stuff. So loading it on the page becomes an optional performance optimization.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1487738738,Option for plugins to request that JSON be served on the page, https://github.com/simonw/datasette/issues/1939#issuecomment-1343872168,https://api.github.com/repos/simonw/datasette/issues/1939,1343872168,IC_kwDOBm6k_c5QGdyo,9599,simonw,2022-12-09T05:29:53Z,2022-12-09T05:29:53Z,OWNER,"I'm going to address those ideas for changes to the `permission_allowed()` in a separate issue. What would it take for the `register_permissions()` hook to be something I'm comfortable landing? I think it's mainly that the list of permissions it provides should Do More Stuff: - Participate in unit tests, in particular this one: https://github.com/simonw/datasette/blob/e539c1c024bc62d88df91d9107cbe37e7f0fe55f/tests/conftest.py#L79-L102 - That new `default` option should be respected - maybe if you omit `default=` from a call to `permission_allowed()` it could fall back on the default from there? - Log a warning if you attempt to check a permission that wasn't registered Then I can use the permissions - in particular their metadata - to help implement his: - #1636","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1941#issuecomment-1343869900,https://api.github.com/repos/simonw/datasette/issues/1941,1343869900,IC_kwDOBm6k_c5QGdPM,9599,simonw,2022-12-09T05:25:19Z,2022-12-09T05:25:19Z,OWNER,"I don't plan to implement this for Datasette 1.0, but it should be something that can be added later on in a minor version bump.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1486036269,Mechanism for supporting key rotation for DATASETTE_SECRET, https://github.com/simonw/datasette/issues/1939#issuecomment-1343858998,https://api.github.com/repos/simonw/datasette/issues/1939,1343858998,IC_kwDOBm6k_c5QGak2,9599,simonw,2022-12-09T05:12:17Z,2022-12-09T05:12:17Z,OWNER,Draft docs for the new plugin hook: https://datasette--1940.org.readthedocs.build/en/1940/plugin_hooks.html#register-permissions-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343856781,https://api.github.com/repos/simonw/datasette/issues/1939,1343856781,IC_kwDOBm6k_c5QGaCN,9599,simonw,2022-12-09T05:10:00Z,2022-12-09T05:10:00Z,OWNER,Made a draft PR so ReadTheDocs would deploy my new documentation somewhere.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343844555,https://api.github.com/repos/simonw/datasette/issues/1939,1343844555,IC_kwDOBm6k_c5QGXDL,9599,simonw,2022-12-09T04:48:28Z,2022-12-09T04:48:28Z,OWNER,"I'm going to try a spike in a branch with `datasette.action_allowed(...)` and a `register_permissions()` plugin hook, to see what they look like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343844112,https://api.github.com/repos/simonw/datasette/issues/1939,1343844112,IC_kwDOBm6k_c5QGW8Q,9599,simonw,2022-12-09T04:47:28Z,2022-12-09T04:47:28Z,OWNER,"I think `action_allowed` is my favourite, even though there's a little bit of concept overlap with `table_actions` and `database_actions`. I never really liked those plugin hook names much to be honest, especially since they are inconsistent with `menu_links`: https://github.com/simonw/datasette/blob/d67f812b7327c7075732688f3df728807503dc58/datasette/hookspecs.py#L123-L135","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343843352,https://api.github.com/repos/simonw/datasette/issues/1939,1343843352,IC_kwDOBm6k_c5QGWwY,9599,simonw,2022-12-09T04:45:50Z,2022-12-09T04:45:50Z,OWNER,"Another option: ```python if await datasette.actor_can(actor, ""insert-data""...) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343842362,https://api.github.com/repos/simonw/datasette/issues/1939,1343842362,IC_kwDOBm6k_c5QGWg6,9599,simonw,2022-12-09T04:43:38Z,2022-12-09T04:43:38Z,OWNER,"Asked ChatGPT for some alternative names, I didn't like any of them: is_permission_granted has_permission check_permission is_action_allowed check_access_permission permission_check validate_permission check_actor_permission verify_permission check_authorization ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343753386,https://api.github.com/repos/simonw/datasette/issues/1939,1343753386,IC_kwDOBm6k_c5QGAyq,9599,simonw,2022-12-09T02:20:20Z,2022-12-09T02:21:01Z,OWNER,"It's also referenced in this plugin hook: ```python @hookspec def permission_allowed(datasette, actor, action, resource): """"""Check if actor is allowed to perform this action - return True, False or None"""""" ``` But more importantly, in these ones: ```python @hookspec def table_actions(datasette, actor, database, table, request): """"""Links for the table actions menu"""""" @hookspec def database_actions(datasette, actor, database, request): """"""Links for the database actions menu"""""" ``` So the word ""action"" is already used within Datasette to refer to those things - which are _almost_ but not quite the same as actions-as-permissions: many of the things that show up in those menus relate to permissions the user has, but not necessarily all of them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343751860,https://api.github.com/repos/simonw/datasette/issues/1939,1343751860,IC_kwDOBm6k_c5QGAa0,9599,simonw,2022-12-09T02:18:11Z,2022-12-09T02:19:23Z,OWNER,"Should I rename ""permission"" to ""action"" elsewhere too? Maybe have a `register_actions(...)` plugin hook instead of adding `register_permissions(...)`? What else could the word ""action"" mean? Currently it's used in the codebase to refer to GitHub Actions, and for code like this: ```python if await self.permission_allowed( actor=actor, action=""view-instance"", default=True ): ``` Which is already revealing the confusion between ""permission"" and ""action"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343751261,https://api.github.com/repos/simonw/datasette/issues/1939,1343751261,IC_kwDOBm6k_c5QGARd,9599,simonw,2022-12-09T02:17:14Z,2022-12-09T02:17:14Z,OWNER,"One option: ```python async def action_allowed(self, actor, action, database=None, resource=None): ``` `action_allowed` fixes the `permission` v.s. `action` thing a bit, and is a new name that doesn't clash with the existing method. I dropped `default` because that's now a property of the permission itself. `table` is now called `resource` and `database` is a separate parameter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343749617,https://api.github.com/repos/simonw/datasette/issues/1939,1343749617,IC_kwDOBm6k_c5QF_3x,9599,simonw,2022-12-09T02:15:54Z,2022-12-09T02:15:54Z,OWNER,"What if I came up with a new method name for this, which could co-exist with the old one while that old one was deprecated?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343744338,https://api.github.com/repos/simonw/datasette/issues/1939,1343744338,IC_kwDOBm6k_c5QF-lS,9599,simonw,2022-12-09T02:08:42Z,2022-12-09T02:08:42Z,OWNER,Extracted a TIL: https://til.simonwillison.net/github/github-code-search-api-uses,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343734812,https://api.github.com/repos/simonw/datasette/issues/1939,1343734812,IC_kwDOBm6k_c5QF8Qc,9599,simonw,2022-12-09T01:57:07Z,2022-12-09T01:57:07Z,OWNER,"This search is better: datasette permission_allowed -user:simonw -path:datasette/** -path:docs/** -path:tests/** language:python That returns 11 results: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+permission_allowed+-user%3Asimonw+-path%3Adatasette%2F**+-path%3Adocs%2F**+-path%3Atests%2F**+language%3Apython 3 are forks of my repos. The rest are all by four users: - [20after4/ddd](https://github.com/20after4/ddd) - [emg110/datasette-graphql](https://github.com/emg110/datasette-graphql) - [next-LI/datasette-csv-importer](https://github.com/next-LI/datasette-csv-importer) - [next-LI/datasette-demo](https://github.com/next-LI/datasette-demo) - [next-LI/datasette-live-config](https://github.com/next-LI/datasette-live-config) - [next-LI/datasette-live-permissions](https://github.com/next-LI/datasette-live-permissions) - [next-LI/datasette-search-all](https://github.com/next-LI/datasette-search-all) - [next-LI/datasette-surveys](https://github.com/next-LI/datasette-surveys) - [next-LI/datasette-write](https://github.com/next-LI/datasette-write) - [rclement/datasette-dashboards](https://github.com/rclement/datasette-dashboards) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343728929,https://api.github.com/repos/simonw/datasette/issues/1939,1343728929,IC_kwDOBm6k_c5QF60h,9599,simonw,2022-12-09T01:48:11Z,2022-12-09T01:52:33Z,OWNER,"This code search shows a bunch of repos I don't know about that would be affected by this change: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+permission_allowed+-user%3Asimonw# These (and likely more): Repositories - [20after4/ddd](https://github.com/20after4/ddd) - [next-LI/datasette-csv-importer](https://github.com/next-LI/datasette-csv-importer) - [digital-land/datasette](https://github.com/digital-land/datasette) - [mroswell/datasette](https://github.com/mroswell/datasette) - [next-LI/datasette-live-config](https://github.com/next-LI/datasette-live-config) - [keladhruv/datasette](https://github.com/keladhruv/datasette) - [RhetTbull/datasette](https://github.com/RhetTbull/datasette) - [chriswedgwood/datasette](https://github.com/chriswedgwood/datasette) - [boan-anbo/datasette](https://github.com/boan-anbo/datasette) - [MattTriano/datasette](https://github.com/MattTriano/datasette) - [incadenza/datasette](https://github.com/incadenza/datasette) - [robdyke/datasette](https://github.com/robdyke/datasette) - [ctb/datasette](https://github.com/ctb/datasette) - [eyeseast/datasette](https://github.com/eyeseast/datasette) - [symbol-management/api-match-audit](https://github.com/symbol-management/api-match-audit) Actually a lot of those are forks of Datasette itself - so maybe this is manageable? Would be nice if I could come up with a GitHub search that excluded any repos with ""datasette"" as their exact name. https://docs.github.com/en/search-github/github-code-search/understanding-github-code-search-syntax#using-qualifiers says: > **Note:** The new code search beta does not currently support regular expressions or partial matching for repository names, so you will have to type the entire repository name (including the user prefix) for the `repo:` qualifier to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343727184,https://api.github.com/repos/simonw/datasette/issues/1939,1343727184,IC_kwDOBm6k_c5QF6ZQ,9599,simonw,2022-12-09T01:45:15Z,2022-12-09T01:45:15Z,OWNER,"Moving the concept of the default for the permission into this registry warrants a redesign of this method anyway: https://github.com/simonw/datasette/blob/e539c1c024bc62d88df91d9107cbe37e7f0fe55f/datasette/app.py#L706","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343724732,https://api.github.com/repos/simonw/datasette/issues/1939,1343724732,IC_kwDOBm6k_c5QF5y8,9599,simonw,2022-12-09T01:40:44Z,2022-12-09T01:43:25Z,OWNER,"```python Permission = collections.namedtuple( ""Permission"", (""name"", ""abbr"", ""takes_database"", ""takes_table"", ""default"") ) ``` I don`t think that design is quite right. - Elsewhere in the code the concept is called an ""action"" rather than a ""permission"" - I think I can stick with the `Permission` name here though, it's pretty clear - `takes_database` - is `takes_` the right verb here? - `takes_table` can also refer to a SQL view or a canned named query A question that was raised by the work in #1938 is whether you should be able to grant a permission like `insert-row` at the instance or database level - and if so, what does that look like? I think you should be able to do that, it doesn't make sense to have to grant it explicitly for every single table. So maybe `takes_table` and `takes_database` are the right names here? But `table` is still bad because it doesn't reflect views and canned queries. One thought is to use `resource` - but that will require a bunch of breaking changes to the existing APIs which treat resource as a tuple. Now's the best time to do that though before Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343722020,https://api.github.com/repos/simonw/datasette/issues/1939,1343722020,IC_kwDOBm6k_c5QF5Ik,9599,simonw,2022-12-09T01:36:05Z,2022-12-09T01:36:16Z,OWNER,"I originally added `permissions.py` for the permission debug tool in https://github.com/simonw/datasette/commit/c51d9246b996a2831c9bd6a1e205f6cb48b9a5f3 - I don't think anything else uses it yet. - #1881","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343721522,https://api.github.com/repos/simonw/datasette/issues/1939,1343721522,IC_kwDOBm6k_c5QF5Ay,9599,simonw,2022-12-09T01:35:15Z,2022-12-09T01:35:15Z,OWNER,"One concern I have about this: there are a bunch of existing plugins that do stuff with permissions that won't currently be using this hook. Do I break those plugins, forcing new releases of them for compatibility with Datasette 1.0? Or maybe I keep them working, but until they've upgraded to register their permissions there are things about them that won't work - e.g. you won't be able to configure their permissions in `metadata.yml` until they release something that does this hook. Best thing is probably for me to get this working in core first and then evaluate the impact it would have on existing plugins once I have some running code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1636#issuecomment-1343715746,https://api.github.com/repos/simonw/datasette/issues/1636,1343715746,IC_kwDOBm6k_c5QF3mi,9599,simonw,2022-12-09T01:27:41Z,2022-12-09T01:27:58Z,OWNER,"I may need to consult this file to figure out if the permission that is being checked can act at the database/table/instance level: https://github.com/simonw/datasette/blob/e539c1c024bc62d88df91d9107cbe37e7f0fe55f/datasette/permissions.py#L1-L19","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042,"""permissions"" propery in metadata for configuring arbitrary permissions", https://github.com/simonw/datasette/issues/1636#issuecomment-1343446071,https://api.github.com/repos/simonw/datasette/issues/1636,1343446071,IC_kwDOBm6k_c5QE1w3,9599,simonw,2022-12-08T22:16:17Z,2022-12-08T22:16:17Z,OWNER,First draft of documentation: https://datasette--1938.org.readthedocs.build/en/1938/authentication.html#other-permissions-in-metadata,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042,"""permissions"" propery in metadata for configuring arbitrary permissions", https://github.com/simonw/datasette/pull/1938#issuecomment-1343445885,https://api.github.com/repos/simonw/datasette/issues/1938,1343445885,IC_kwDOBm6k_c5QE1t9,9599,simonw,2022-12-08T22:16:03Z,2022-12-08T22:16:03Z,OWNER,Docs: https://datasette--1938.org.readthedocs.build/en/1938/authentication.html#other-permissions-in-metadata,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485488236,"""permissions"" blocks in metadata.json/yaml", https://github.com/simonw/datasette/issues/1636#issuecomment-1343440504,https://api.github.com/repos/simonw/datasette/issues/1636,1343440504,IC_kwDOBm6k_c5QE0Z4,9599,simonw,2022-12-08T22:10:28Z,2022-12-08T22:10:48Z,OWNER,"What if you want to grant `insert-row` to a user for ALL tables in a database, or even for all tables in all databases? You should be able to do that by putting that in the root `permissions:` block. Need to figure out how the implementation will handle that. Also: there are some permissions like `view-instance` or `debug-menu` for which putting them at the `database` or `table` or `query` level doesn't actually make any sense. Ideally the implementation would spot those on startup and refuse to start the server, with a helpful error message.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042,"""permissions"" propery in metadata for configuring arbitrary permissions", https://github.com/simonw/datasette/pull/1930#issuecomment-1343360006,https://api.github.com/repos/simonw/datasette/issues/1930,1343360006,IC_kwDOBm6k_c5QEgwG,9599,simonw,2022-12-08T21:12:28Z,2022-12-08T21:12:28Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473664029,Typo in JSON API `Updating a row` documentation, https://github.com/simonw/datasette/issues/1867#issuecomment-1341874378,https://api.github.com/repos/simonw/datasette/issues/1867,1341874378,IC_kwDOBm6k_c5P-2DK,9599,simonw,2022-12-08T02:07:06Z,2022-12-08T02:28:02Z,OWNER,"Basic version of this allows you to rename a table: ``` POST /db/table/-/rename { ""name"": ""new_table_name"" } ``` If a table with that new name already exists you will get an error - unless you pass `""replace"": true` in which case that table will be dropped and replaced by the new one. ``` POST /db/table/-/rename { ""name"": ""new_table_name"", ""replace"": true } ``` This is useful because it allows for that atomic replacement operation: upload brand new data into a `_tmp_name` table, then atomic rename and replace after the upload has completed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426080014,/db/table/-/rename API (also allows atomic replace), https://github.com/simonw/datasette/issues/1636#issuecomment-1341854373,https://api.github.com/repos/simonw/datasette/issues/1636,1341854373,IC_kwDOBm6k_c5P-xKl,9599,simonw,2022-12-08T01:43:35Z,2022-12-08T01:43:35Z,OWNER,I'm going to write the documentation for this first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042,"""permissions"" propery in metadata for configuring arbitrary permissions", https://github.com/simonw/datasette/issues/1936#issuecomment-1341849735,https://api.github.com/repos/simonw/datasette/issues/1936,1341849735,IC_kwDOBm6k_c5P-wCH,9599,simonw,2022-12-08T01:35:54Z,2022-12-08T01:35:54Z,OWNER,"Running that twice produced this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1483250004,Fix /db/table/-/upsert in the API explorer, https://github.com/simonw/datasette/issues/1936#issuecomment-1341849496,https://api.github.com/repos/simonw/datasette/issues/1936,1341849496,IC_kwDOBm6k_c5P-v-Y,9599,simonw,2022-12-08T01:35:35Z,2022-12-08T01:35:35Z,OWNER,"Related bug: you can send `""id"": null` and it works (it should throw an error): ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1483250004,Fix /db/table/-/upsert in the API explorer, https://github.com/simonw/datasette/issues/1937#issuecomment-1341848525,https://api.github.com/repos/simonw/datasette/issues/1937,1341848525,IC_kwDOBm6k_c5P-vvN,9599,simonw,2022-12-08T01:34:03Z,2022-12-08T01:34:03Z,OWNER,Check should go somewhere around here: https://github.com/simonw/datasette/blob/dee18ed8ce7af2ab8699bcb5a51a99f48301bc42/datasette/views/database.py#L625-L631,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1483320357,/db/-/create API should require insert-rows permission to use row: or rows: option, https://github.com/simonw/datasette/pull/1931#issuecomment-1341825314,https://api.github.com/repos/simonw/datasette/issues/1931,1341825314,IC_kwDOBm6k_c5P-qEi,9599,simonw,2022-12-08T01:03:18Z,2022-12-08T01:03:18Z,OWNER,"I broke this test: ``` ds_write = @pytest.mark.asyncio async def test_insert_row(ds_write): token = write_token(ds_write) response = await ds_write.client.post( ""/data/docs/-/insert"", json={""row"": {""title"": ""Test"", ""score"": 1.2, ""age"": 5}}, headers={ ""Authorization"": ""***"".format(token), ""Content-Type"": ""application/json"", }, ) expected_row = {""id"": 1, ""title"": ""Test"", ""score"": 1.2, ""age"": 5} > assert response.status_code == 201 E assert 500 == 201 E + where 500 = .status_code /home/runner/work/datasette/datasette/tests/test_api_write.py:43: AssertionError ----------------------------- Captured stderr call ----------------------------- Traceback (most recent call last): File ""/home/runner/work/datasette/datasette/datasette/app.py"", line 1447, in route_path response = await view(request, send) File ""/home/runner/work/datasette/datasette/datasette/views/base.py"", line 151, in view return await self.dispatch_request(request) File ""/home/runner/work/datasette/datasette/datasette/views/base.py"", line 105, in dispatch_request response = await handler(request) File ""/home/runner/work/datasette/datasette/datasette/views/table.py"", line 1228, in post row_pk_values_for_later = [tuple(row[pk] for pk in pks) for row in rows] File ""/home/runner/work/datasette/datasette/datasette/views/table.py"", line 1228, in row_pk_values_for_later = [tuple(row[pk] for pk in pks) for row in rows] File ""/home/runner/work/datasette/datasette/datasette/views/table.py"", line 1228, in row_pk_values_for_later = [tuple(row[pk] for pk in pks) for row in rows] KeyError: 'id' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/pull/1931#issuecomment-1341821213,https://api.github.com/repos/simonw/datasette/issues/1931,1341821213,IC_kwDOBm6k_c5P-pEd,9599,simonw,2022-12-08T00:58:21Z,2022-12-08T00:58:21Z,OWNER,"In the interests of shipping, I'm going to punt the API explorer to a later issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/pull/1931#issuecomment-1339968514,https://api.github.com/repos/simonw/datasette/issues/1931,1339968514,IC_kwDOBm6k_c5P3kwC,9599,simonw,2022-12-06T20:28:47Z,2022-12-06T20:28:47Z,OWNER,"Should the `""return"": true` mode reflect the order in which the rows were provided when the API was called? I think it should. Since this is small enough to happily fit in Python memory (thanks to the `max_insert_rows` setting) I can load the fresh data from the database and then sort it in Python space before returning it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/issues/1855#issuecomment-1339952692,https://api.github.com/repos/simonw/datasette/issues/1855,1339952692,IC_kwDOBm6k_c5P3g40,9599,simonw,2022-12-06T20:15:50Z,2022-12-06T20:16:00Z,OWNER,"That commit there https://github.com/simonw/datasette/commit/6da17d5529773dfe41b53ed4ce5a6ecb46ed2457 (which will be squash-merged in a PR later on) made it so that `_r` was correctly copied across from the token to the created actor, and fixed a bug in the code that checks permissions against it for resources. I needed that mechanism to write a test that exercised different API permissions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/pull/1931#issuecomment-1339911152,https://api.github.com/repos/simonw/datasette/issues/1931,1339911152,IC_kwDOBm6k_c5P3Wvw,9599,simonw,2022-12-06T19:38:12Z,2022-12-06T19:38:12Z,OWNER,Documentation: https://datasette--1931.org.readthedocs.build/en/1931/json_api.html#upserting-rows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/issues/1927#issuecomment-1339910494,https://api.github.com/repos/simonw/datasette/issues/1927,1339910494,IC_kwDOBm6k_c5P3Wle,9599,simonw,2022-12-06T19:37:39Z,2022-12-06T19:37:39Z,OWNER,"I'll finish this after I land: - #1931 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197,ignore:true/replace:true options for /db/-/create API, https://github.com/simonw/datasette/issues/1929#issuecomment-1339909159,https://api.github.com/repos/simonw/datasette/issues/1929,1339909159,IC_kwDOBm6k_c5P3WQn,9599,simonw,2022-12-06T19:36:23Z,2022-12-06T19:36:23Z,OWNER,https://docs.datasette.io/en/1.0a1/json_api.html 👍 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473659191,Incorrect link from the API explorer to the JSON API documentation, https://github.com/simonw/datasette/issues/1929#issuecomment-1339871933,https://api.github.com/repos/simonw/datasette/issues/1929,1339871933,IC_kwDOBm6k_c5P3NK9,9599,simonw,2022-12-06T19:23:48Z,2022-12-06T19:24:17Z,OWNER,"I can do that on this page: https://readthedocs.org/projects/datasette/versions/?version_filter=1.0 I'm making them both active and hidden: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473659191,Incorrect link from the API explorer to the JSON API documentation, https://github.com/simonw/datasette/issues/1929#issuecomment-1339867570,https://api.github.com/repos/simonw/datasette/issues/1929,1339867570,IC_kwDOBm6k_c5P3MGy,9599,simonw,2022-12-06T19:22:47Z,2022-12-06T19:22:47Z,OWNER,"Yeah I should deploy the docs for the alpha versions (the intention is that the docs exactly match the release you are using), thanks for spotting that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473659191,Incorrect link from the API explorer to the JSON API documentation, https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339834918,https://api.github.com/repos/simonw/sqlite-utils/issues/516,1339834918,IC_kwDOCGYnMM5P3EIm,9599,simonw,2022-12-06T19:00:18Z,2022-12-06T19:00:35Z,OWNER,"Right now the command produces no output at all. Maybe a `--verbose` mode that writes these numbers to standard error (or even standard output since it's an option)? Is there a better name than `--verbose` for this? `--summary` perhaps?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1479914599,Feature request: output number of ignored/replaced rows for insert command, https://github.com/simonw/datasette/pull/1931#issuecomment-1339784569,https://api.github.com/repos/simonw/datasette/issues/1931,1339784569,IC_kwDOBm6k_c5P2315,9599,simonw,2022-12-06T18:16:15Z,2022-12-06T18:17:56Z,OWNER,"Just noticed the insert API returns `{}` when it should return `{""ok"": true}` - will fix that here too. UPDATE: no it did that already, it was just the documentation that was wrong.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/pull/1931#issuecomment-1339768422,https://api.github.com/repos/simonw/datasette/issues/1931,1339768422,IC_kwDOBm6k_c5P2z5m,9599,simonw,2022-12-06T18:04:59Z,2022-12-06T18:04:59Z,OWNER,"I realized this API should require both the `insert-row` AND the `update-row` permissions, since calls to it could do either one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/issues/1878#issuecomment-1336100218,https://api.github.com/repos/simonw/datasette/issues/1878,1336100218,IC_kwDOBm6k_c5Po0V6,9599,simonw,2022-12-03T07:02:15Z,2022-12-03T07:02:15Z,OWNER,"Moved this work to a PR: - #1931","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1927#issuecomment-1336099588,https://api.github.com/repos/simonw/datasette/issues/1927,1336099588,IC_kwDOBm6k_c5Po0ME,9599,simonw,2022-12-03T06:58:14Z,2022-12-03T06:58:14Z,OWNER,I have not yet documented the new `insert` and `replace` options.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197,ignore:true/replace:true options for /db/-/create API, https://github.com/simonw/datasette/issues/1927#issuecomment-1336099533,https://api.github.com/repos/simonw/datasette/issues/1927,1336099533,IC_kwDOBm6k_c5Po0LN,9599,simonw,2022-12-03T06:57:52Z,2022-12-03T06:57:52Z,OWNER,I'm going to push what I have anyway. I'll keep this issue open while I think through the above comment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197,ignore:true/replace:true options for /db/-/create API, https://github.com/simonw/datasette/issues/1927#issuecomment-1336099368,https://api.github.com/repos/simonw/datasette/issues/1927,1336099368,IC_kwDOBm6k_c5Po0Io,9599,simonw,2022-12-03T06:56:36Z,2022-12-03T06:56:36Z,OWNER,"Neither of these options make sense if you didn't pass a `""pk""`. My initial implementation spotted if the `pk` was missing and looked it up from the table, but actually I don't think that makes sense - if you know the table exists and hence don't pass the `pk` you should be using `/-/insert` or `/-/upsert` instead. So maybe this work should expanded to include validation that checks if the table exists already - and if it does, confirms that the primary key (and maybe even the columns) are the same as for that existing table. Of course if you only send `row` or `rows` then checking `columns` doesn't completely make sense - but we could check that the rows you have sent are equal to or a subset of the columns in the table. We could even check the column types as well, as seen in: - #1910 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197,ignore:true/replace:true options for /db/-/create API, https://github.com/simonw/datasette/issues/1878#issuecomment-1336094562,https://api.github.com/repos/simonw/datasette/issues/1878,1336094562,IC_kwDOBm6k_c5Poy9i,9599,simonw,2022-12-03T06:27:50Z,2022-12-03T06:29:06Z,OWNER,"This adds it to the API explorer: ```diff diff --git a/datasette/views/special.py b/datasette/views/special.py index 1f84b094..1b4a9d3c 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -316,21 +316,37 @@ class ApiExplorerView(BaseView): request.actor, ""insert-row"", (name, table) ): pks = await db.primary_keys(table) - table_links.append( - { - ""path"": self.ds.urls.table(name, table) + ""/-/insert"", - ""method"": ""POST"", - ""label"": ""Insert rows into {}"".format(table), - ""json"": { - ""rows"": [ - { - column: None - for column in await db.table_columns(table) - if column not in pks - } - ] + table_links.extend( + [ + { + ""path"": self.ds.urls.table(name, table) + ""/-/insert"", + ""method"": ""POST"", + ""label"": ""Insert rows into {}"".format(table), + ""json"": { + ""rows"": [ + { + column: None + for column in await db.table_columns(table) + if column not in pks + } + ] + }, }, - } + { + ""path"": self.ds.urls.table(name, table) + ""/-/upsert"", + ""method"": ""POST"", + ""label"": ""Upsert rows into {}"".format(table), + ""json"": { + ""rows"": [ + { + column: None + for column in await db.table_columns(table) + if column not in pks + } + ] + }, + }, + ] ) if await self.ds.permission_allowed( request.actor, ""drop-table"", (name, table) ``` Except it doesn't quite, because the example JSON this generates is invalid as it does not include the primary key column. (Made me notice that the way example columns are created for `/-/insert` will fail for tables that don't have an auto-incrementing primary key)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1878#issuecomment-1336094470,https://api.github.com/repos/simonw/datasette/issues/1878,1336094470,IC_kwDOBm6k_c5Poy8G,9599,simonw,2022-12-03T06:27:13Z,2022-12-03T06:27:13Z,OWNER,"Tests are going to need to cover both rowid-only and compound primary key tables, including all of the error states.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1878#issuecomment-1336094381,https://api.github.com/repos/simonw/datasette/issues/1878,1336094381,IC_kwDOBm6k_c5Poy6t,9599,simonw,2022-12-03T06:26:25Z,2022-12-03T06:26:25Z,OWNER,"Initial prototype: ```diff diff --git a/datasette/app.py b/datasette/app.py index 125b4969..282c0984 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -40,7 +40,7 @@ from .views.special import ( PermissionsDebugView, MessagesDebugView, ) -from .views.table import TableView, TableInsertView, TableDropView +from .views.table import TableView, TableInsertView, TableUpsertView, TableDropView from .views.row import RowView, RowDeleteView, RowUpdateView from .renderer import json_renderer from .url_builder import Urls @@ -1292,6 +1292,10 @@ class Datasette: TableInsertView.as_view(self), r""/(?P[^\/\.]+)/(?P[^\/\.]+)/-/insert$"", ) + add_route( + TableUpsertView.as_view(self), + r""/(?P[^\/\.]+)/(?P
[^\/\.]+)/-/upsert$"", + ) add_route( TableDropView.as_view(self), r""/(?P[^\/\.]+)/(?P
[^\/\.]+)/-/drop$"", diff --git a/datasette/views/table.py b/datasette/views/table.py index 7ba78c11..ae0d6366 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1074,9 +1074,15 @@ class TableInsertView(BaseView): def __init__(self, datasette): self.ds = datasette - async def _validate_data(self, request, db, table_name): + async def _validate_data(self, request, db, table_name, pks, upsert): errors = [] + pks_list = [] + if isinstance(pks, str): + pks_list = [pks] + else: + pks_list = list(pks) + def _errors(errors): return None, errors, {} @@ -1135,6 +1141,15 @@ class TableInsertView(BaseView): # Validate columns of each row columns = set(await db.table_columns(table_name)) for i, row in enumerate(rows): + if upsert: + # It MUST have the primary key + missing_pks = [pk for pk in pks_list if pk not in row] + if missing_pks: + errors.append( + 'Row {} is missing primary key column(s): ""{}""'.format( + i, '"", ""'.join(missing_pks) + ) + ) invalid_columns = set(row.keys()) - columns if invalid_columns: errors.append( @@ -1146,7 +1161,7 @@ class TableInsertView(BaseView): return _errors(errors) return rows, errors, extras - async def post(self, request): + async def post(self, request, upsert=False): try: resolved = await self.ds.resolve_table(request) except NotFound as e: @@ -1164,7 +1179,12 @@ class TableInsertView(BaseView): request.actor, ""insert-row"", resource=(database_name, table_name) ): return _error([""Permission denied""], 403) - rows, errors, extras = await self._validate_data(request, db, table_name) + + pks = await db.primary_keys(table_name) + + rows, errors, extras = await self._validate_data( + request, db, table_name, pks, upsert + ) if errors: return _error(errors, 400) @@ -1172,15 +1192,19 @@ class TableInsertView(BaseView): replace = extras.get(""replace"") should_return = bool(extras.get(""return"", False)) - # Insert rows - def insert_rows(conn): + + def insert_or_upsert_rows(conn): table = sqlite_utils.Database(conn)[table_name] + kwargs = {} + if upsert: + kwargs[""pk""] = pks[0] if len(pks) == 1 else pks + else: + kwargs = {""ignore"": ignore, ""replace"": replace} if should_return: rowids = [] + method = table.upsert if upsert else table.insert for row in rows: - rowids.append( - table.insert(row, ignore=ignore, replace=replace).last_rowid - ) + rowids.append(method(row, **kwargs).last_rowid) return list( table.rows_where( ""rowid in ({})"".format("","".join(""?"" for _ in rowids)), @@ -1188,10 +1212,11 @@ class TableInsertView(BaseView): ) ) else: - table.insert_all(rows, ignore=ignore, replace=replace) + method_all = table.upsert_all if upsert else table.insert_all + method_all(rows, **kwargs) try: - rows = await db.execute_write_fn(insert_rows) + rows = await db.execute_write_fn(insert_or_upsert_rows) except Exception as e: return _error([str(e)]) result = {""ok"": True} @@ -1200,6 +1225,13 @@ class TableInsertView(BaseView): return Response.json(result, status=201) +class TableUpsertView(TableInsertView): + name = ""table-upsert"" + + async def post(self, request): + return await super().post(request, upsert=True) + + class TableDropView(BaseView): name = ""table-drop"" ``` Manual testing reveals that this mostly works... but it's not doing the right thing for `""return"": true` - it always returns an empty list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1878#issuecomment-1336073212,https://api.github.com/repos/simonw/datasette/issues/1878,1336073212,IC_kwDOBm6k_c5Potv8,9599,simonw,2022-12-03T05:38:49Z,2022-12-03T05:38:49Z,OWNER,And on Discord today: https://discord.com/channels/823971286308356157/823971286941302908/1048426072066236536,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1878#issuecomment-1336070843,https://api.github.com/repos/simonw/datasette/issues/1878,1336070843,IC_kwDOBm6k_c5PotK7,9599,simonw,2022-12-03T05:37:53Z,2022-12-03T05:37:53Z,OWNER,Also requested here: https://news.ycombinator.com/item?id=33839894,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1927#issuecomment-1335984268,https://api.github.com/repos/simonw/datasette/issues/1927,1335984268,IC_kwDOBm6k_c5PoYCM,9599,simonw,2022-12-03T00:26:26Z,2022-12-03T00:26:26Z,OWNER,"Also: the documentation should clarify that you can call this API multiple times when using the `rows` option. (It will probably grow `""alter"": true` soon too).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197,ignore:true/replace:true options for /db/-/create API, https://github.com/simonw/datasette/issues/1928#issuecomment-1335966329,https://api.github.com/repos/simonw/datasette/issues/1928,1335966329,IC_kwDOBm6k_c5PoTp5,9599,simonw,2022-12-02T23:47:11Z,2022-12-02T23:47:11Z,OWNER,Wrote about this extensively here: https://simonwillison.net/2022/Dec/2/datasette-write-api/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870889,https://api.github.com/repos/simonw/datasette/issues/1928,1335870889,IC_kwDOBm6k_c5Pn8Wp,9599,simonw,2022-12-02T21:41:09Z,2022-12-02T21:41:09Z,OWNER,"Got it working! https://simon.datasette.cloud/data/hacker_news_posts https://github.com/simonw/scrape-hacker-news-by-domain/blob/main/submit-to-datasette-cloud.sh","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870887,https://api.github.com/repos/simonw/datasette/issues/1928,1335870887,IC_kwDOBm6k_c5Pn8Wn,9599,simonw,2022-12-02T21:25:16Z,2022-12-02T21:25:16Z,OWNER,"Here's the change that should submit data to Datasette Cloud: https://github.com/simonw/scrape-hacker-news-by-domain/commit/848bb7e835a9fb87cd656362591835179cd1dc1b I ran `delete from hacker_news_posts` against my instance so https://simon.datasette.cloud/data/hacker_news_posts is now empty.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870884,https://api.github.com/repos/simonw/datasette/issues/1928,1335870884,IC_kwDOBm6k_c5Pn8Wk,9599,simonw,2022-12-02T21:19:58Z,2022-12-02T21:19:58Z,OWNER,"But until I fix this issue: - https://github.com/simonw/datasette/issues/1927 I need to insert freshly scraped data like this: ```bash export ROWS=$( jq -n --argjson rows ""$(cat simonwillison-net.json)"" \ '{ ""rows"": $rows, ""replace"": true }' ) curl -X POST \ https://simon.datasette.cloud/data/hacker_news_posts/-/insert \ -H ""Content-Type: application/json"" \ -H ""Authorization: Bearer $DS_TOKEN"" \ -d $ROWS ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870883,https://api.github.com/repos/simonw/datasette/issues/1928,1335870883,IC_kwDOBm6k_c5Pn8Wj,9599,simonw,2022-12-02T21:19:10Z,2022-12-02T21:19:10Z,OWNER,"I created the `hacker_news_posts` table like this: ```bash export ROWS=$( jq -n --argjson rows ""$(cat simonwillison-net.json)"" \ '{ ""table"": ""hacker_news_posts"", ""rows"": $rows, ""pk"": ""id"", ""replace"": true }' ) # Use curl to POST some JSON to a URL curl -X POST \ https://simon.datasette.cloud/data/-/create \ -H ""Content-Type: application/json"" \ -H ""Authorization: Bearer $DS_TOKEN"" \ -d $ROWS ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870879,https://api.github.com/repos/simonw/datasette/issues/1928,1335870879,IC_kwDOBm6k_c5Pn8Wf,9599,simonw,2022-12-02T21:18:25Z,2022-12-02T21:18:25Z,OWNER,"This is the SQL view for the atom feed: ```sql CREATE VIEW hacker_news_posts_atom as select id as atom_id, title as atom_title, url, commentsUrl as atom_link, dt || 'Z' as atom_updated, 'Submitter: ' || submitter || ' - ' || points || ' points, ' || numComments || ' comments' as atom_content from hacker_news_posts order by dt desc limit 100; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870877,https://api.github.com/repos/simonw/datasette/issues/1928,1335870877,IC_kwDOBm6k_c5Pn8Wd,9599,simonw,2022-12-02T21:17:50Z,2022-12-02T21:17:50Z,OWNER,https://simon.datasette.cloud/data/hacker_news_posts_atom.atom is working now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1636#issuecomment-1334759315,https://api.github.com/repos/simonw/datasette/issues/1636,1334759315,IC_kwDOBm6k_c5Pjs-T,9599,simonw,2022-12-02T04:46:32Z,2022-12-02T04:46:32Z,OWNER,"Thankfully all of the logic for this already lives in just one place: https://github.com/simonw/datasette/blob/d7e5e3c9f98d194fdfb12f1ecc60ed5b3afbc464/datasette/default_permissions.py#L23-L59","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042,"""permissions"" propery in metadata for configuring arbitrary permissions", https://github.com/simonw/datasette/issues/1636#issuecomment-1334758766,https://api.github.com/repos/simonw/datasette/issues/1636,1334758766,IC_kwDOBm6k_c5Pjs1u,9599,simonw,2022-12-02T04:45:16Z,2022-12-02T04:45:16Z,OWNER,"Also, this is another thing which should live in `config.yml` rather than being crammed into `metadata.yml` - but I can fix that when I address: - #493","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042,"""permissions"" propery in metadata for configuring arbitrary permissions", https://github.com/simonw/datasette/issues/1636#issuecomment-1334757597,https://api.github.com/repos/simonw/datasette/issues/1636,1334757597,IC_kwDOBm6k_c5Pjsjd,9599,simonw,2022-12-02T04:42:35Z,2022-12-02T04:42:35Z,OWNER,"Should I call this key `permissions` or something else? Some options: - `permissions` - `perms` - shorter to type - `allow` - I like the word, but might be confusing to change its meaning since we use it already","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042,"""permissions"" propery in metadata for configuring arbitrary permissions", https://github.com/simonw/datasette/issues/1636#issuecomment-1334673179,https://api.github.com/repos/simonw/datasette/issues/1636,1334673179,IC_kwDOBm6k_c5PjX8b,9599,simonw,2022-12-02T02:07:20Z,2022-12-02T04:27:07Z,OWNER,"So the new mechanism needs to extend that to handle all of the other permissions as well. The simplest design I can think of is this (here illustrated using YAML): ```yaml # instance-level permissions - give every logged in user the debug menu: permissions: debug-menu: id: * databases: content: # Allow bob to create-table in the content database permissions: create-table: id: bob ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042,"""permissions"" propery in metadata for configuring arbitrary permissions", https://github.com/simonw/datasette/issues/1636#issuecomment-1334666806,https://api.github.com/repos/simonw/datasette/issues/1636,1334666806,IC_kwDOBm6k_c5PjWY2,9599,simonw,2022-12-02T01:58:40Z,2022-12-02T02:00:53Z,OWNER,"Current design: ```json { ""databases"": { ""private"": { ""allow"": { ""id"": ""*"" } } } } ``` This can be applied at the instance, database, table or query level within the nested JSON. https://docs.datasette.io/en/stable/authentication.html#controlling-access-to-specific-databases It's actually controlling the following permissions: - `view-instance` - `view-database` - `view-table` - `view-query` There's also a special case for allowing SQL queries,at the instance and database level: ```json { ""databases"": { ""mydatabase"": { ""allow_sql"": { ""id"": ""root"" } } } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042,"""permissions"" propery in metadata for configuring arbitrary permissions", https://github.com/simonw/datasette/issues/1926#issuecomment-1334508062,https://api.github.com/repos/simonw/datasette/issues/1926,1334508062,IC_kwDOBm6k_c5Pivoe,9599,simonw,2022-12-01T22:06:12Z,2022-12-01T22:06:12Z,OWNER,Released: https://pypi.org/project/datasette/1.0a1/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1471969984,Release notes for 1.0a1 (and release it), https://github.com/simonw/datasette/issues/1924#issuecomment-1334165225,https://api.github.com/repos/simonw/datasette/issues/1924,1334165225,IC_kwDOBm6k_c5Phb7p,9599,simonw,2022-12-01T18:15:15Z,2022-12-01T18:15:15Z,OWNER,Updated docs at the bottom of this section: https://docs.datasette.io/en/latest/json_api.html#inserting-rows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470509936,Docs for replace:true and ignore:true options for insert API, https://github.com/simonw/datasette/issues/1924#issuecomment-1333042785,https://api.github.com/repos/simonw/datasette/issues/1924,1333042785,IC_kwDOBm6k_c5PdJ5h,9599,simonw,2022-12-01T02:00:06Z,2022-12-01T02:00:06Z,OWNER,"Looks like I did implement these already: https://github.com/simonw/datasette/blob/9a1536b52a07e32da5900652da1bd7894c58fa9f/tests/test_api_write.py#L270-L273 But forgot to document them!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470509936,Docs for replace:true and ignore:true options for insert API, https://github.com/simonw/datasette/issues/1924#issuecomment-1333025076,https://api.github.com/repos/simonw/datasette/issues/1924,1333025076,IC_kwDOBm6k_c5PdFk0,9599,simonw,2022-12-01T01:37:09Z,2022-12-01T01:37:09Z,OWNER,"Related question: what happens if you attempt to insert rows without either of these settings and 1:10 of them is an integrity error due to an existing primary key? Does the entire batch fail to be inserted? Should it? This may be the point that I need to think hard about how to use transactions here. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470509936,Docs for replace:true and ignore:true options for insert API, https://github.com/simonw/datasette/issues/1924#issuecomment-1333023273,https://api.github.com/repos/simonw/datasette/issues/1924,1333023273,IC_kwDOBm6k_c5PdFIp,9599,simonw,2022-12-01T01:34:48Z,2022-12-01T01:34:48Z,OWNER,"This will enable some very cool Datasette write API demos - for example, Git scrapers that insert-replace the most recent copy of the scraped data to a table somewhere (which can then produce an atom feed with https://datasette.io/plugins/datasette-atom)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470509936,Docs for replace:true and ignore:true options for insert API, https://github.com/simonw/datasette/issues/1922#issuecomment-1332903011,https://api.github.com/repos/simonw/datasette/issues/1922,1332903011,IC_kwDOBm6k_c5Pcnxj,9599,simonw,2022-11-30T23:45:29Z,2022-11-30T23:45:29Z,OWNER,"That worked for the preflight request - got this now: So it looks like error responses (in this case for permission denied) are missing CORS headers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332855687,https://api.github.com/repos/simonw/datasette/issues/1922,1332855687,IC_kwDOBm6k_c5PccOH,9599,simonw,2022-11-30T23:09:31Z,2022-11-30T23:09:31Z,OWNER,"Still getting a CORS header. I tried it in Chrome, which outputs helpful messages to the console: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1923#issuecomment-1332851215,https://api.github.com/repos/simonw/datasette/issues/1923,1332851215,IC_kwDOBm6k_c5PcbIP,9599,simonw,2022-11-30T23:04:56Z,2022-11-30T23:04:56Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470320227,latest.datasette.io Cloud Run deploys failing, https://github.com/simonw/datasette/issues/1923#issuecomment-1332842435,https://api.github.com/repos/simonw/datasette/issues/1923,1332842435,IC_kwDOBm6k_c5PcY_D,9599,simonw,2022-11-30T22:58:33Z,2022-11-30T22:58:33Z,OWNER,https://stackoverflow.com/questions/74490465/github-actions-failing-for-setup-gcloud/74562740#74562740 suggests trying Python 3.9.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470320227,latest.datasette.io Cloud Run deploys failing, https://github.com/simonw/datasette/issues/1923#issuecomment-1332835664,https://api.github.com/repos/simonw/datasette/issues/1923,1332835664,IC_kwDOBm6k_c5PcXVQ,9599,simonw,2022-11-30T22:50:10Z,2022-11-30T22:51:25Z,OWNER,https://stackoverflow.com/questions/74490465/github-actions-failing-for-setup-gcloud/74562526#74562526 suggests setting `version: '318.0.0'` of `google-github-actions/setup-gcloud`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470320227,latest.datasette.io Cloud Run deploys failing, https://github.com/simonw/datasette/issues/1922#issuecomment-1332698636,https://api.github.com/repos/simonw/datasette/issues/1922,1332698636,IC_kwDOBm6k_c5Pb14M,9599,simonw,2022-11-30T20:25:50Z,2022-11-30T20:25:50Z,OWNER,"I just shipped this: Access-Control-Allow-Methods: GET, POST, HEAD, OPTIONS I'll try this out on `latest.datasette.io` - but I need to research more to check if this is a safe thing to do or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332689547,https://api.github.com/repos/simonw/datasette/issues/1922,1332689547,IC_kwDOBm6k_c5PbzqL,9599,simonw,2022-11-30T20:16:21Z,2022-11-30T20:16:46Z,OWNER,"That notebook: ```javascript viewof token = Inputs.text({ label: ""Your API token"" }) ``` ```javascript viewof createResponse = Inputs.button(""Create table"", { value: null, reduce: async () => { const response = await fetch( ""https://latest.datasette.io/ephemeral/-/create"", { method: ""POST"", mode: ""cors"", headers: { Authorization: `Bearer {$token}`, ""Content-Type"": ""application/json"" }, body: JSON.stringify({ table: ""my_new_table"", row: { task: ""Demonstrate a JSON creation from another domain"" } }) } ); return await response.json(); } }) ``` Based on this tip: https://talk.observablehq.com/t/best-pattern-for-click-here-to-submit-your-results-to-an-api-backend/7353/3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332688245,https://api.github.com/repos/simonw/datasette/issues/1922,1332688245,IC_kwDOBm6k_c5PbzV1,9599,simonw,2022-11-30T20:15:08Z,2022-11-30T20:15:08Z,OWNER,"Still getting a CORS error: My hunch is this is because I'm not sending `Access-Control-Allow-Methods: GET,HEAD,POST`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332585861,https://api.github.com/repos/simonw/datasette/issues/1922,1332585861,IC_kwDOBm6k_c5PbaWF,9599,simonw,2022-11-30T18:43:46Z,2022-11-30T18:43:46Z,OWNER,"Here's what Django Rest Framework does: https://github.com/encode/django-rest-framework/blob/1ae812ea209392ad76cc5d2f35f9f7fb337f63e4/rest_framework/views.py#L514-L521 ```python def options(self, request, *args, **kwargs): """""" Handler method for HTTP 'OPTIONS' request. """""" if self.metadata_class is None: return self.http_method_not_allowed(request, *args, **kwargs) data = self.metadata_class().determine_metadata(request, self) return Response(data, status=status.HTTP_200_OK) ``` That default `determine_metadata` method looks like this: https://github.com/encode/django-rest-framework/blob/1ae812ea209392ad76cc5d2f35f9f7fb337f63e4/rest_framework/metadata.py#L61-L71 ```python def determine_metadata(self, request, view): metadata = OrderedDict() metadata['name'] = view.get_view_name() metadata['description'] = view.get_view_description() metadata['renders'] = [renderer.media_type for renderer in view.renderer_classes] metadata['parses'] = [parser.media_type for parser in view.parser_classes] if hasattr(view, 'get_serializer'): actions = self.determine_actions(request, view) if actions: metadata['actions'] = actions return metadata ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332580395,https://api.github.com/repos/simonw/datasette/issues/1922,1332580395,IC_kwDOBm6k_c5PbZAr,9599,simonw,2022-11-30T18:38:22Z,2022-11-30T18:38:22Z,OWNER,"> [@simon](https://fedi.simonwillison.net/@simon) IMO, it should always be a 2XX series response, typically with no content & an extra `Allow` header with a list of HTTP verbs it responds to. https://mastodon.social/@daniellindsley/109434186252099323","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332572453,https://api.github.com/repos/simonw/datasette/issues/1922,1332572453,IC_kwDOBm6k_c5PbXEl,9599,simonw,2022-11-30T18:30:38Z,2022-11-30T18:30:54Z,OWNER,Started a conversation about how OPTIONS should work on Mastodon: https://fedi.simonwillison.net/@simon/109434148676475291,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332561813,https://api.github.com/repos/simonw/datasette/issues/1922,1332561813,IC_kwDOBm6k_c5PbUeV,9599,simonw,2022-11-30T18:20:05Z,2022-11-30T18:20:05Z,OWNER,"Weird, GitHub reply with a 404! ``` ~ % curl -X OPTIONS https://github.com/ -i HTTP/2 404 server: GitHub.com date: Wed, 30 Nov 2022 18:19:39 GMT content-type: text/html; charset=utf-8 content-length: 0 strict-transport-security: max-age=31536000; includeSubdomains; preload x-frame-options: deny x-content-type-options: nosniff x-xss-protection: 0 referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubusercontent.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events; font-src github.githubassets.com; form-action 'self' github.com gist.github.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com objects-origin.githubusercontent.com secured-user-images.githubusercontent.com/ opengraph.githubassets.com github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; worker-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/ vary: Accept-Encoding, Accept, X-Requested-With x-github-request-id: DD6B:5ACA:102E8A6:1164A99:63879EBB ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332561059,https://api.github.com/repos/simonw/datasette/issues/1922,1332561059,IC_kwDOBm6k_c5PbUSj,9599,simonw,2022-11-30T18:19:20Z,2022-11-30T18:19:20Z,OWNER,"Two test failures: ``` ____________________________ test_homepage_options _____________________________ [gw0] linux -- Python 3.11.0 /opt/hostedtoolcache/Python/3.11.0/x64/bin/python app_client = def test_homepage_options(app_client): response = app_client.get(""/"", method=""OPTIONS"") > assert response.status == 405 E assert 200 == 405 E + where 200 = .status /home/runner/work/datasette/datasette/tests/test_html.py:58: AssertionError ______________________ test_client_methods[options-/-405] ______________________ [gw1] linux -- Python 3.11.0 /opt/hostedtoolcache/Python/3.11.0/x64/bin/python datasette = method = 'options', path = '/', expected_status = 405 @pytest.mark.asyncio @pytest.mark.parametrize( ""method,path,expected_status"", [ (""get"", ""/"", 200), (""options"", ""/"", 405), (""head"", ""/"", 200), (""put"", ""/"", 405), (""patch"", ""/"", 405), (""delete"", ""/"", 405), ], ) async def test_client_methods(datasette, method, path, expected_status): client_method = getattr(datasette.client, method) response = await client_method(path) assert isinstance(response, httpx.Response) > assert response.status_code == expected_status E assert 200 == 405 E + where 200 = .status_code /home/runner/work/datasette/datasette/tests/test_internals_datasette_client.py:29: AssertionError =============================== warnings summary =============================== tests/test_cli.py::test_inspect_cli_writes_to_file tests/test_cli.py::test_inspect_cli /home/runner/work/datasette/datasette/datasette/cli.py:163: DeprecationWarning: There is no current event loop loop = asyncio.get_event_loop() tests/test_cli_serve_get.py: 2 warnings tests/test_cli.py: 12 warnings tests/test_crossdb.py: 1 warning /home/runner/work/datasette/datasette/datasette/cli.py:591: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(ds.invoke_startup()) tests/test_cli_serve_get.py: 2 warnings tests/test_cli.py: 12 warnings tests/test_crossdb.py: 1 warning /home/runner/work/datasette/datasette/datasette/cli.py:594: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(check_databases(ds)) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========================== short test summary info ============================ FAILED tests/test_html.py::test_homepage_options - assert 200 == 405 + where 200 = .status FAILED tests/test_internals_datasette_client.py::test_client_methods[options-/-405] - assert 200 == 405 + where 200 = .status_code ====== 2 failed, 1195 passed, 1 skipped, 32 warnings in 191.06s (0:03:11) ====== Error: Process completed with exit code 1. ``` On reading https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/OPTIONS I feel like I should be a bit more thoughtful about how I treat OPTIONS - maybe it should work for every URL on the site, but return a `204 No Content` header? Comparing a few different sites: ``` ~ % curl -X OPTIONS https://www.google.com/ -i HTTP/2 405 allow: GET, HEAD date: Wed, 30 Nov 2022 18:18:15 GMT content-type: text/html; charset=UTF-8 server: gws content-length: 1592 x-xss-protection: 0 x-frame-options: SAMEORIGIN alt-svc: h3="":443""; ma=2592000,h3-29="":443""; ma=2592000,h3-Q050="":443""; ma=2592000,h3-Q046="":443""; ma=2592000,h3-Q043="":443""; ma=2592000,quic="":443""; ma=2592000; v=""46,43"" Error 405 (Method Not Allowed)!!1

405. That’s an error.

The request method OPTIONS is inappropriate for the URL /. That’s all we know. ~ % curl -X OPTIONS https://www.mozilla.org/ -i HTTP/2 405 content-type: text/html; charset=utf-8 content-length: 0 server: meinheld/1.0.2 date: Wed, 30 Nov 2022 18:18:38 GMT allow: GET, HEAD x-frame-options: DENY content-security-policy: child-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com www.googletagmanager.com www.google-analytics.com www.youtube-nocookie.com trackertest.org www.surveygizmo.com accounts.firefox.com accounts.firefox.com.cn www.youtube.com; connect-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com www.googletagmanager.com www.google-analytics.com region1.google-analytics.com logs.convertexperiments.com 1003350.metrics.convertexperiments.com 1003343.metrics.convertexperiments.com sentry.prod.mozaws.net o1069899.sentry.io o1069899.ingest.sentry.io https://accounts.firefox.com/ stage.cjms.nonprod.cloudops.mozgcp.net cjms.services.mozilla.com; frame-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com www.googletagmanager.com www.google-analytics.com www.youtube-nocookie.com trackertest.org www.surveygizmo.com accounts.firefox.com accounts.firefox.com.cn www.youtube.com; script-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com 'unsafe-inline' 'unsafe-eval' www.googletagmanager.com www.google-analytics.com tagmanager.google.com www.youtube.com s.ytimg.com cdn-3.convertexperiments.com app.convert.com data.track.convertexperiments.com 1003350.track.convertexperiments.com 1003343.track.convertexperiments.com; img-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com data: mozilla.org www.googletagmanager.com www.google-analytics.com adservice.google.com adservice.google.de adservice.google.dk creativecommons.org cdn-3.convertexperiments.com logs.convertexperiments.com images.ctfassets.net ad.doubleclick.net; style-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com 'unsafe-inline' app.convert.com; default-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com; font-src 'self' cache-control: max-age=600 expires: Wed, 30 Nov 2022 18:28:38 GMT x-backend-server: bedrock-prod-web-b95bc569d-grd25.iowa-a strict-transport-security: max-age=31536000 x-content-type-options: nosniff x-xss-protection: 1; mode=block referrer-policy: strict-origin-when-cross-origin via: 1.1 google, 1.1 6c90b631453c435bd0022caa657b67e8.cloudfront.net (CloudFront) x-cache: Error from cloudfront x-amz-cf-pop: SFO5-P2 x-amz-cf-id: A6-9mLztaE2tz840CbV9bXYiBMZRKEamDj6jGGEl1U7sg8egWfsDqg== ~ % curl -X OPTIONS https://example.com -i HTTP/2 200 allow: OPTIONS, GET, HEAD, POST cache-control: max-age=604800 content-type: text/html; charset=UTF-8 date: Wed, 30 Nov 2022 18:18:59 GMT expires: Wed, 07 Dec 2022 18:18:59 GMT server: EOS (vny/0451) content-length: 0 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332504654,https://api.github.com/repos/simonw/datasette/issues/1922,1332504654,IC_kwDOBm6k_c5PbGhO,9599,simonw,2022-11-30T17:27:39Z,2022-11-30T17:27:39Z,OWNER,I'll test this once it's deployed to https://latest.datasette.io/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332493004,https://api.github.com/repos/simonw/datasette/issues/1922,1332493004,IC_kwDOBm6k_c5PbDrM,9599,simonw,2022-11-30T17:18:10Z,2022-11-30T17:18:10Z,OWNER,"Here's why: https://github.com/simonw/datasette/blob/4ddd77e51254bda3bac990ea662bac2e6b29c5e0/datasette/views/base.py#L71-L79 That's code in `BaseView` - but it turns out the code that adds CORS headers is in the `DataView` subclass of that (which the various write API endpoints do not use). https://github.com/simonw/datasette/blob/4ddd77e51254bda3bac990ea662bac2e6b29c5e0/datasette/views/base.py#L158-L162","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332492092,https://api.github.com/repos/simonw/datasette/issues/1922,1332492092,IC_kwDOBm6k_c5PbDc8,9599,simonw,2022-11-30T17:17:21Z,2022-11-30T17:17:21Z,OWNER,I tried running `fetch()` with a POST from a separate domain and got a browser error because it did a GET against the `/db/-/create` endpoint and the 405 method not supported response did not include the CORS headers.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1605#issuecomment-1331694246,https://api.github.com/repos/simonw/datasette/issues/1605,1331694246,IC_kwDOBm6k_c5PYAqm,9599,simonw,2022-11-30T06:18:41Z,2022-11-30T06:18:41Z,OWNER,"Those sounds to me like they should be promoted to documented, supported internals.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1918#issuecomment-1331658629,https://api.github.com/repos/simonw/datasette/issues/1918,1331658629,IC_kwDOBm6k_c5PX3-F,9599,simonw,2022-11-30T05:21:51Z,2022-11-30T05:21:51Z,OWNER,"Much better: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469044738,API explorer should list mutable databases first, https://github.com/simonw/datasette/issues/1919#issuecomment-1331657404,https://api.github.com/repos/simonw/datasette/issues/1919,1331657404,IC_kwDOBm6k_c5PX3q8,9599,simonw,2022-11-30T05:19:43Z,2022-11-30T05:19:43Z,OWNER,"This is the test: https://github.com/simonw/datasette/blob/8404b21556d133c89eda4bd1bf5335ed9a0785d6/tests/test_api_write.py#L342-L401 I'm suspicious that there's a timing error of some sort but I can't think what it might be.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469062686,Intermittent `test_delete_row` test failure , https://github.com/simonw/datasette/issues/1916#issuecomment-1331651721,https://api.github.com/repos/simonw/datasette/issues/1916,1331651721,IC_kwDOBm6k_c5PX2SJ,9599,simonw,2022-11-30T05:10:27Z,2022-11-30T05:10:27Z,OWNER,"They should return 405 method not allowed with an `{""ok"":false, ""error"": ""Method not allowed""}` body.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469015001,GET requests against POST endpoints should not 500 error, https://github.com/simonw/datasette/issues/1917#issuecomment-1331644751,https://api.github.com/repos/simonw/datasette/issues/1917,1331644751,IC_kwDOBm6k_c5PX0lP,9599,simonw,2022-11-30T04:59:22Z,2022-11-30T04:59:22Z,OWNER,"Yeah it looks like I introduced this bug here: https://github.com/simonw/datasette/commit/fb7e70d5e72a951efe4b29ad999d8915c032d021","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469043836,Don't allow writable API to edit the `_memory` database, https://github.com/simonw/datasette/issues/1917#issuecomment-1331644078,https://api.github.com/repos/simonw/datasette/issues/1917,1331644078,IC_kwDOBm6k_c5PX0au,9599,simonw,2022-11-30T04:58:06Z,2022-11-30T04:58:06Z,OWNER,"The problem might actually be here: https://github.com/simonw/datasette/blob/9f5321ff1eca58c469a45cc406d7eb5ad05accbd/datasette/app.py#L280-L281 `is_mutable` defaults to `True`, so this line should probably be: ```python self.add_database(Database(self, is_mutable=False, is_memory=True), name=""_memory"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469043836,Don't allow writable API to edit the `_memory` database, https://github.com/simonw/datasette/issues/1915#issuecomment-1331479606,https://api.github.com/repos/simonw/datasette/issues/1915,1331479606,IC_kwDOBm6k_c5PXMQ2,9599,simonw,2022-11-30T00:09:06Z,2022-11-30T00:09:06Z,OWNER,One last feature: I want to show an indication on the table page that the table has X seconds left to live.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531,Interactive demo of Datasette 1.0 write APIs, https://github.com/simonw/datasette/issues/1915#issuecomment-1331479328,https://api.github.com/repos/simonw/datasette/issues/1915,1331479328,IC_kwDOBm6k_c5PXMMg,9599,simonw,2022-11-30T00:08:41Z,2022-11-30T00:08:41Z,OWNER,Five minute has now passed and https://latest.datasette.io/ephemeral/new_table is gone.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531,Interactive demo of Datasette 1.0 write APIs, https://github.com/simonw/datasette/issues/1915#issuecomment-1331478611,https://api.github.com/repos/simonw/datasette/issues/1915,1331478611,IC_kwDOBm6k_c5PXMBT,9599,simonw,2022-11-30T00:07:37Z,2022-11-30T00:07:37Z,OWNER,"Then I created an API token at https://latest.datasette.io/-/create-token and ran this: ``` curl -XPOST 'https://latest.datasette.io/ephemeral/new_table/-/insert' \ -H 'Authorization: Bearer xxx' \ -H 'Content-Type: application/json' \ -d '{""row"": {""name"": ""NAME""}}' ``` And it inserted a row into https://latest.datasette.io/ephemeral/new_table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531,Interactive demo of Datasette 1.0 write APIs, https://github.com/simonw/datasette/issues/1915#issuecomment-1331476246,https://api.github.com/repos/simonw/datasette/issues/1915,1331476246,IC_kwDOBm6k_c5PXLcW,9599,simonw,2022-11-30T00:04:35Z,2022-11-30T00:08:24Z,OWNER,"The new https://github.com/simonw/datasette-ephemeral-tables plugin is live now: https://latest.datasette.io/ephemeral - you have to navigate through https://latest.datasette.io/login-as-root first It work! I created a table using https://latest.datasette.io/-/api#path=%2Fephemeral%2F-%2Fcreate&json=%7B%0A++%22table%22%3A+%22new_table%22%2C%0A++%22columns%22%3A+%5B%0A++++%7B%0A++++++%22name%22%3A+%22id%22%2C%0A++++++%22type%22%3A+%22integer%22%0A++++%7D%2C%0A++++%7B%0A++++++%22name%22%3A+%22name%22%2C%0A++++++%22type%22%3A+%22text%22%0A++++%7D%0A++%5D%2C%0A++%22pk%22%3A+%22id%22%0A%7D&method=POST The table should vanish in a few minutes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531,Interactive demo of Datasette 1.0 write APIs, https://github.com/simonw/datasette/issues/1915#issuecomment-1331432223,https://api.github.com/repos/simonw/datasette/issues/1915,1331432223,IC_kwDOBm6k_c5PXAsf,9599,simonw,2022-11-29T23:06:17Z,2022-11-29T23:06:17Z,OWNER,To (slightly) discourage abuse I'm going to make the demo database only visible to the root user - so people can't create tables with rude names and have them show to the public on https://latest.datasette.io/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531,Interactive demo of Datasette 1.0 write APIs, https://github.com/simonw/datasette/issues/1915#issuecomment-1331331082,https://api.github.com/repos/simonw/datasette/issues/1915,1331331082,IC_kwDOBm6k_c5PWoAK,9599,simonw,2022-11-29T21:24:59Z,2022-11-29T21:34:53Z,OWNER,Maybe a plugin called `datasette-temporary-tables` or `datasette-demo-tables` or `datasette-demo-database`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531,Interactive demo of Datasette 1.0 write APIs, https://github.com/simonw/datasette/issues/1850#issuecomment-1331238841,https://api.github.com/repos/simonw/datasette/issues/1850,1331238841,IC_kwDOBm6k_c5PWRe5,9599,simonw,2022-11-29T20:11:20Z,2022-11-29T20:11:20Z,OWNER,"Released this in Datasette 1.0a0: - #1913","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1913#issuecomment-1331238029,https://api.github.com/repos/simonw/datasette/issues/1913,1331238029,IC_kwDOBm6k_c5PWRSN,9599,simonw,2022-11-29T20:10:35Z,2022-11-29T20:10:35Z,OWNER,"Released: - https://pypi.org/project/datasette/1.0a0/ - https://docs.datasette.io/en/latest/changelog.html#a0-2022-11-29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331226346,https://api.github.com/repos/simonw/datasette/issues/1913,1331226346,IC_kwDOBm6k_c5PWObq,9599,simonw,2022-11-29T20:00:16Z,2022-11-29T20:00:36Z,OWNER,"Looks like a fix is coming: https://github.com/pypa/twine/issues/940#issuecomment-1331225509 > > Note that `must_decode` was defined in `pkg_info/_compat.py`, and was thus never an API: before 1.9.0, it was only imported and used in `pkginfo/distribution.py'. > > Nevertheless, I will push out a 1.9.1 release of `pkginfo` which restores a deprecated compatibility alias.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331225277,https://api.github.com/repos/simonw/datasette/issues/1913,1331225277,IC_kwDOBm6k_c5PWOK9,9599,simonw,2022-11-29T19:59:14Z,2022-11-29T19:59:34Z,OWNER,I deleted the tag and tried creating a new release. Now running here: https://github.com/simonw/datasette/actions/runs/3577554546,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331216652,https://api.github.com/repos/simonw/datasette/issues/1913,1331216652,IC_kwDOBm6k_c5PWMEM,9599,simonw,2022-11-29T19:54:22Z,2022-11-29T19:54:22Z,OWNER,Filed a bug report here: https://bugs.launchpad.net/pkginfo/+bug/1998249,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331208206,https://api.github.com/repos/simonw/datasette/issues/1913,1331208206,IC_kwDOBm6k_c5PWKAO,9599,simonw,2022-11-29T19:51:31Z,2022-11-29T19:51:31Z,OWNER,https://pypi.org/project/pkginfo/#history - 1.9.0 came out 39 minutes ago!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331207334,https://api.github.com/repos/simonw/datasette/issues/1913,1331207334,IC_kwDOBm6k_c5PWJym,9599,simonw,2022-11-29T19:50:37Z,2022-11-29T19:50:37Z,OWNER,"https://pypi.org/project/setuptools/65.6.3/ came out most recently - 23rd November (wheel and twine are older). No search results at all for that error message. This is very weird, I would have expected it to have been reported by now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331205613,https://api.github.com/repos/simonw/datasette/issues/1913,1331205613,IC_kwDOBm6k_c5PWJXt,9599,simonw,2022-11-29T19:48:52Z,2022-11-29T19:48:52Z,OWNER,https://github.com/simonw/datasette/blob/07aad511769da9242260c850e8d975cbd8c29552/.github/workflows/publish.yml#L52-L61,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331204360,https://api.github.com/repos/simonw/datasette/issues/1913,1331204360,IC_kwDOBm6k_c5PWJEI,9599,simonw,2022-11-29T19:47:40Z,2022-11-29T19:47:40Z,OWNER,"... but the last step of the deploy failed, when it was meant to push to PyPI! ``` Uploading distributions to https://upload.pypi.org/legacy/ Traceback (most recent call last): File ""/opt/hostedtoolcache/Python/3.11.0/x64/bin/twine"", line 8, in sys.exit(main()) ^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/__main__.py"", line 33, in main error = cli.dispatch(sys.argv[1:]) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/cli.py"", line 123, in dispatch return main(args.args) ^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py"", line 198, in main return upload(upload_settings, parsed_args.dists) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py"", line 123, in upload packages_to_upload = [ ^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py"", line 124, in _make_package(filename, signatures, upload_settings) for filename in uploads ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py"", line 77, in _make_package package = package_file.PackageFile.from_filename(filename, upload_settings.comment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/package.py"", line 96, in from_filename meta = DIST_TYPES[dtype](filename) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/wheel.py"", line 42, in __init__ self.extractMetadata() File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/pkginfo/distribution.py"", line 121, in extractMetadata self.parse(data) File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/wheel.py"", line 89, in parse fp = io.StringIO(distribution.must_decode(data)) ^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'pkginfo.distribution' has no attribute 'must_decode'. Did you mean: '_must_decode'? Error: Process completed with exit code 1. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331203997,https://api.github.com/repos/simonw/datasette/issues/1913,1331203997,IC_kwDOBm6k_c5PWI-d,9599,simonw,2022-11-29T19:47:13Z,2022-11-29T19:47:13Z,OWNER,"Weird, retrying the tests DID get them to pass. https://github.com/simonw/datasette/actions/runs/3577355358/jobs/6016518244","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1877#issuecomment-1331201207,https://api.github.com/repos/simonw/datasette/issues/1877,1331201207,IC_kwDOBm6k_c5PWIS3,9599,simonw,2022-11-29T19:44:07Z,2022-11-29T19:44:07Z,OWNER,"I fixed the duplicate logic issue here: https://github.com/simonw/datasette/commit/ee64130fa8a5ff4a24791916c696e10cf2375102 - #1896 Decided not to address `views/table.py`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432012302,Refactor and tidy up final write API code, https://github.com/simonw/datasette/pull/1912#issuecomment-1331196531,https://api.github.com/repos/simonw/datasette/issues/1912,1331196531,IC_kwDOBm6k_c5PWHJz,9599,simonw,2022-11-29T19:39:10Z,2022-11-29T19:39:10Z,OWNER,"Annoyingly it looks like I can't rebase this one, and I don't want to squash-merge and lose the commits, so I'm going to do a regular merge instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468592292,Merge 1.0-dev (with initial write API) back into main, https://github.com/simonw/datasette/issues/1891#issuecomment-1331181922,https://api.github.com/repos/simonw/datasette/issues/1891,1331181922,IC_kwDOBm6k_c5PWDli,9599,simonw,2022-11-29T19:23:41Z,2022-11-29T19:23:41Z,OWNER,https://github.com/simonw/datasette/blob/4d49a5a39739476e1ada43f70a0029abcef07977/docs/changelog.rst#10a0-2022-11-29,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450303205,1.0a0 release notes, https://github.com/simonw/datasette/issues/1891#issuecomment-1331143292,https://api.github.com/repos/simonw/datasette/issues/1891,1331143292,IC_kwDOBm6k_c5PV6J8,9599,simonw,2022-11-29T18:57:40Z,2022-11-29T18:57:40Z,OWNER,I'm going to keep these short - they'll mostly be links to the documentation for the new features.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450303205,1.0a0 release notes, https://github.com/simonw/datasette/issues/1891#issuecomment-1331140747,https://api.github.com/repos/simonw/datasette/issues/1891,1331140747,IC_kwDOBm6k_c5PV5iL,9599,simonw,2022-11-29T18:55:42Z,2022-11-29T18:55:42Z,OWNER,"All features for the alpha are complete now. Release notes should be based on these commits: https://github.com/simonw/datasette/compare/0.63.2...6bda2257868a2cbd70b84b7a86a5bcb47dcc4874","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450303205,1.0a0 release notes, https://github.com/simonw/datasette/issues/1911#issuecomment-1331135709,https://api.github.com/repos/simonw/datasette/issues/1911,1331135709,IC_kwDOBm6k_c5PV4Td,9599,simonw,2022-11-29T18:50:58Z,2022-11-29T18:50:58Z,OWNER,Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468519699,`/db/-/create` should support creating tables with compound primary keys, https://github.com/simonw/datasette/issues/1911#issuecomment-1331120055,https://api.github.com/repos/simonw/datasette/issues/1911,1331120055,IC_kwDOBm6k_c5PV0e3,9599,simonw,2022-11-29T18:36:01Z,2022-11-29T18:36:01Z,OWNER,"Current API design: ``` POST //-/create ``` ```json { ""table"": ""name_of_new_table"", ""columns"": [ { ""name"": ""id"", ""type"": ""integer"" }, { ""name"": ""title"", ""type"": ""text"" } ], ""pk"": ""id"" } ``` I'm going to add a new `""pks""` key which is a list, and can be used in place of `""pk""`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468519699,`/db/-/create` should support creating tables with compound primary keys, https://github.com/simonw/datasette/issues/1863#issuecomment-1331089156,https://api.github.com/repos/simonw/datasette/issues/1863,1331089156,IC_kwDOBm6k_c5PVs8E,9599,simonw,2022-11-29T18:08:53Z,2022-11-29T18:08:53Z,OWNER,"I do think this needs type checking - I just tried and you really can send a string to an integer column and have it work, which feels bad.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1330974099,https://api.github.com/repos/simonw/datasette/issues/1863,1330974099,IC_kwDOBm6k_c5PVQ2T,9599,simonw,2022-11-29T17:03:00Z,2022-11-29T17:11:05Z,OWNER,I've decided that I won't do that validation for the first version of this - I'm going to teach `dclient` to send the correct types instead: https://github.com/simonw/dclient/issues/6#issuecomment-1330963953,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1909#issuecomment-1329624931,https://api.github.com/repos/simonw/datasette/issues/1909,1329624931,IC_kwDOBm6k_c5PQHdj,9599,simonw,2022-11-28T19:19:26Z,2022-11-28T19:19:26Z,OWNER,The list of states here is a good example of somewhere this might be useful: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=state&_facet_size=max,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1466952626,Option to sort facets alphabetically, https://github.com/simonw/datasette/issues/1605#issuecomment-1328169472,https://api.github.com/repos/simonw/datasette/issues/1605,1328169472,IC_kwDOBm6k_c5PKkIA,9599,simonw,2022-11-27T04:32:14Z,2022-11-27T04:32:14Z,OWNER,@eyeseast I started work on that plugin: https://github.com/simonw/datasette-export,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952,Scripted exports, https://github.com/simonw/datasette/issues/1863#issuecomment-1324539030,https://api.github.com/repos/simonw/datasette/issues/1863,1324539030,IC_kwDOBm6k_c5O8tyW,9599,simonw,2022-11-23T04:35:14Z,2022-11-23T04:35:14Z,OWNER,If I do that I should probably update `insert` to do those validation checks as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1324531750,https://api.github.com/repos/simonw/datasette/issues/1863,1324531750,IC_kwDOBm6k_c5O8sAm,9599,simonw,2022-11-23T04:20:47Z,2022-11-23T04:20:47Z,OWNER,"... which does imply that I'm going to do an extra layer of validation over what SQLite provides. SQLite will happily allow a text string to be added to a supposedly integer column. I'm not going to allow that - I'll return a validation error instead, unless the string can be safely coerced to the correct type.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1324531085,https://api.github.com/repos/simonw/datasette/issues/1863,1324531085,IC_kwDOBm6k_c5O8r2N,9599,simonw,2022-11-23T04:19:28Z,2022-11-23T04:19:28Z,OWNER,Had a design conversation with myself in https://github.com/simonw/dclient/issues/6 where I decided that the API should allow string values to be sent to integer columns which would be automatically converted *if possible to do so* - as an API usability feature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1903#issuecomment-1321262142,https://api.github.com/repos/simonw/datasette/issues/1903,1321262142,IC_kwDOBm6k_c5OwNw-,9599,simonw,2022-11-20T22:35:01Z,2022-11-20T22:35:01Z,OWNER,A want to call this `datasette/exceptions.py` inspired by Takahē: https://github.com/andrewgodwin/takahe/blob/f491fdb56e2de9200e14b855b5576009ca99dfa5/core/exceptions.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1455928469,Refactor all error classes into a datasette.exceptions module, https://github.com/simonw/datasette/issues/1905#issuecomment-1320721241,https://api.github.com/repos/simonw/datasette/issues/1905,1320721241,IC_kwDOBm6k_c5OuJtZ,9599,simonw,2022-11-19T01:12:05Z,2022-11-19T01:12:05Z,OWNER,Used it to deploy this: https://fivethirtyeight.datasettes.com/-/versions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1456012874,`publish heroku` failing due to old Python version, https://github.com/simonw/datasette/issues/1905#issuecomment-1320706596,https://api.github.com/repos/simonw/datasette/issues/1905,1320706596,IC_kwDOBm6k_c5OuGIk,9599,simonw,2022-11-19T00:41:44Z,2022-11-19T00:41:44Z,OWNER,"Tested a deploy: ``` % datasette publish heroku fixtures.db -n datasette-issue-1905 › Warning: heroku update available from 7.63.0 to 7.66.4. › Warning: heroku update available from 7.63.0 to 7.66.4. › Warning: heroku update available from 7.63.0 to 7.66.4. Creating datasette-issue-1905... done › Warning: heroku update available from 7.63.0 to 7.66.4. ▸ Couldn't detect GNU tar. Builds could fail due to decompression errors ▸ See https://devcenter.heroku.com/articles/platform-api-deploying-slugs#create-slug-archive ▸ Please install it, or specify the '--tar' option ▸ Falling back to node's built-in compressor -----> Building on the Heroku-22 stack -----> Determining which buildpack to use for this app -----> Python app detected -----> Using Python version specified in runtime.txt -----> Installing python-3.11.0 -----> Installing pip 22.3.1, setuptools 63.4.3 and wheel 0.37.1 -----> Installing SQLite3 -----> Installing requirements with pip Collecting datasette Downloading datasette-0.63.1-py3-none-any.whl (231 kB) ... -----> Running post-compile hook -----> Discovering process types Procfile declares types -> web -----> Compressing... Done: 28M -----> Launching... Released v3 https://datasette-issue-1905.herokuapp.com/ deployed to Heroku Starting November 28th, 2022, free Heroku Dynos, free Heroku Postgres, and free Heroku Data for Redis® will no longer be available. If you have apps using any of these resources, you must upgrade to paid plans by this date to ensure your apps continue to run and to retain your data. For students, we will announce a new program by the end of September. Learn more at https://blog.heroku.com/next-chapter ``` I had to then pay for the dino because I'd run out of free hours. https://datasette-issue-1905.herokuapp.com/-/versions shows: ```json { ""python"": { ""version"": ""3.11.0"", ""full"": ""3.11.0 (main, Oct 24 2022, 21:34:02) [GCC 11.2.0]"" }, ""datasette"": { ""version"": ""0.63.1"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1456012874,`publish heroku` failing due to old Python version, https://github.com/simonw/datasette/issues/1905#issuecomment-1320689643,https://api.github.com/repos/simonw/datasette/issues/1905,1320689643,IC_kwDOBm6k_c5OuB_r,9599,simonw,2022-11-19T00:17:19Z,2022-11-19T00:41:54Z,OWNER,"The tests don't cover this bit at the moment. Would be easier to write tests if there was a `--generate-dir` option as seen in https://datasette.io/plugins/datasette-publish-vercel","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1456012874,`publish heroku` failing due to old Python version, https://github.com/simonw/datasette/issues/1905#issuecomment-1320678715,https://api.github.com/repos/simonw/datasette/issues/1905,1320678715,IC_kwDOBm6k_c5Ot_U7,9599,simonw,2022-11-19T00:02:28Z,2022-11-19T00:02:28Z,OWNER,This is a strong argument for extracting the Heroku support out to a plugin - it would allow this to be fixed with a plugin release without needing to push a full release of Datasette itself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1456012874,`publish heroku` failing due to old Python version, https://github.com/simonw/datasette/issues/1891#issuecomment-1320625260,https://api.github.com/repos/simonw/datasette/issues/1891,1320625260,IC_kwDOBm6k_c5OtyRs,9599,simonw,2022-11-18T23:01:03Z,2022-11-18T23:01:48Z,OWNER,I think this actually needs to include a whole section of the documentation about the road to 1.0 - what to expect (planned breaking changes) etc. I can add that to the https://docs.datasette.io/en/stable/contributing.html page perhaps - or even create a Roadmap page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450303205,1.0a0 release notes, https://github.com/simonw/datasette/issues/1896#issuecomment-1320616559,https://api.github.com/repos/simonw/datasette/issues/1896,1320616559,IC_kwDOBm6k_c5OtwJv,9599,simonw,2022-11-18T22:51:14Z,2022-11-18T22:51:14Z,OWNER,New methods are documented here: https://docs.datasette.io/en/1.0-dev/internals.html#resolve-database-request,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452364777,Extract logic for resolving a URL to a database / table / row, https://github.com/simonw/datasette/issues/1903#issuecomment-1320614541,https://api.github.com/repos/simonw/datasette/issues/1903,1320614541,IC_kwDOBm6k_c5OtvqN,9599,simonw,2022-11-18T22:47:41Z,2022-11-18T22:47:41Z,OWNER,"When I do this it's important to update the documentation for `resolve_database()` and the like: https://github.com/simonw/datasette/blob/ee64130fa8a5ff4a24791916c696e10cf2375102/docs/internals.rst#L594","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1455928469,Refactor all error classes into a datasette.exceptions module, https://github.com/simonw/datasette/issues/1896#issuecomment-1320588299,https://api.github.com/repos/simonw/datasette/issues/1896,1320588299,IC_kwDOBm6k_c5OtpQL,9599,simonw,2022-11-18T22:16:59Z,2022-11-18T22:17:06Z,OWNER,"Found myself needing an `await db.view_exists()` method for this, similar to the existing `await db.table_exists()` one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452364777,Extract logic for resolving a URL to a database / table / row, https://github.com/simonw/datasette/issues/1863#issuecomment-1320563197,https://api.github.com/repos/simonw/datasette/issues/1863,1320563197,IC_kwDOBm6k_c5OtjH9,9599,simonw,2022-11-18T21:47:35Z,2022-11-18T21:48:07Z,OWNER,"Incomplete implementation of this view: ```python class RowUpdateView(BaseView): name = ""row-update"" def __init__(self, datasette): self.ds = datasette async def post(self, request): database_route = tilde_decode(request.url_vars[""database""]) table = tilde_decode(request.url_vars[""table""]) try: db = self.ds.get_database(route=database_route) except KeyError: return _error([""Database not found: {}"".format(database_route)], 404) database_name = db.name if not await db.table_exists(table): return _error([""Table not found: {}"".format(table)], 404) pk_values = urlsafe_components(request.url_vars[""pks""]) sql, params, pks = await row_sql_params_pks(db, table, pk_values) results = await db.execute(sql, params, truncate=True) rows = list(results.rows) if not rows: return _error([f""Record not found: {pk_values}""], 404) # Ensure user has permission to update this row if not await self.ds.permission_allowed( request.actor, ""update-row"", resource=(database_name, table) ): return _error([""Permission denied""], 403) body = await request.post_body() try: data = json.loads(body) except json.JSONDecodeError as e: return _error([""Invalid JSON: {}"".format(e)]) if not isinstance(data, dict): return _error([""JSON must be a dictionary""]) def update_row(conn): sqlite_utils.Database(conn)[table].update(pk_values, updates) await db.execute_write_fn(update_row) result = {""ok"": True} if data.get(""return""): result[""row""] = {""row-here"": ""TODO""} return Response.json(result, status=200) ``` This is before the refactor in: - #1896","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/pull/1898#issuecomment-1319642535,https://api.github.com/repos/simonw/datasette/issues/1898,1319642535,IC_kwDOBm6k_c5OqCWn,9599,simonw,2022-11-18T07:28:45Z,2022-11-18T07:28:45Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452485922,Use DOMContentLoaded instead of load event for CodeMirror initialization, https://github.com/simonw/datasette/issues/1899#issuecomment-1319642338,https://api.github.com/repos/simonw/datasette/issues/1899,1319642338,IC_kwDOBm6k_c5OqCTi,9599,simonw,2022-11-18T07:28:28Z,2022-11-18T07:28:28Z,OWNER,Demo: https://latest.datasette.io/fixtures,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452495049,Clicking within the CodeMirror area below the SQL (i.e. when there's only a single line) doesn't cause the editor to get focused , https://github.com/simonw/datasette/issues/1900#issuecomment-1319631421,https://api.github.com/repos/simonw/datasette/issues/1900,1319631421,IC_kwDOBm6k_c5Op_o9,9599,simonw,2022-11-18T07:13:00Z,2022-11-18T07:13:00Z,OWNER,"You get: ``` => [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye 0.9s => [internal] load build context 2.3s => => transferring context: 72.38MB 2.3s => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c 0.0s ``` I get: ``` => [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye 1.0s => [internal] load build context 0.0s => => transferring context: 705B 0.0s => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c 0.0s ``` Both the image name and the hash are _exactly_ the same. So why are you getting an error while mine works OK? For my machine: ``` ~ % docker --version Docker version 20.10.12, build e91ed57 ~ % uname -a Darwin Simons-MacBook-Pro-2.local 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:54 PDT 2022; root:xnu-8792.41.9~2/RELEASE_X86_64 x86_64 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319629469,https://api.github.com/repos/simonw/datasette/issues/1900,1319629469,IC_kwDOBm6k_c5Op_Kd,9599,simonw,2022-11-18T07:10:17Z,2022-11-18T07:10:17Z,OWNER,This is so weird! What version of Datasette do you get from `datasette --version` there - and what's your Docker version / operating system version?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1901#issuecomment-1319627012,https://api.github.com/repos/simonw/datasette/issues/1901,1319627012,IC_kwDOBm6k_c5Op-kE,9599,simonw,2022-11-18T07:07:03Z,2022-11-18T07:07:03Z,OWNER,"Here's the full list of 10 plugin releases for this issue: * [datasette-search-all 1.1.1](https://github.com/simonw/datasette-search-all/releases/tag/1.1.1) * [datasette-ripgrep 0.7.1](https://github.com/simonw/datasette-ripgrep/releases/tag/0.7.1) * [datasette-socrata 0.3.1](https://github.com/simonw/datasette-socrata/releases/tag/0.3.1) * [datasette-configure-fts 1.1.1](https://github.com/simonw/datasette-configure-fts/releases/tag/1.1.1) * [datasette-edit-templates 0.2](https://github.com/simonw/datasette-edit-templates/releases/tag/0.2) * [datasette-copyable 0.3.2](https://github.com/simonw/datasette-copyable/releases/tag/0.3.2) * [datasette-public 0.2.1](https://github.com/simonw/datasette-public/releases/tag/0.2.1) * [datasette-import-table 0.3.1](https://github.com/simonw/datasette-import-table/releases/tag/0.3.1) * [datasette-indieauth 1.2.2](https://github.com/simonw/datasette-indieauth/releases/tag/1.2.2) * [datasette-edit-schema 0.5.2](https://github.com/simonw/datasette-edit-schema/releases/tag/0.5.2)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319623911,https://api.github.com/repos/simonw/datasette/issues/1901,1319623911,IC_kwDOBm6k_c5Op9zn,9599,simonw,2022-11-18T07:02:56Z,2022-11-18T07:02:56Z,OWNER,That's all of them!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319588163,https://api.github.com/repos/simonw/datasette/issues/1901,1319588163,IC_kwDOBm6k_c5Op1FD,9599,simonw,2022-11-18T06:05:11Z,2022-11-18T06:05:11Z,OWNER,"For `datasette-copyable` I want to show breadcrumbs that take database/instance permissions into account, so I'm removing `{% block nav %}` entirely and replacing it with this: ```html+jinja {% block crumbs %} {{ crumbs.nav(request=request, database=database, table=table) }} {% endblock %} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1899#issuecomment-1319584553,https://api.github.com/repos/simonw/datasette/issues/1899,1319584553,IC_kwDOBm6k_c5Op0Mp,9599,simonw,2022-11-18T06:00:10Z,2022-11-18T06:01:50Z,OWNER,"I can't actually remember where that `min-height: 70px` came from. I just tried without it and it seems fine - especially since any time you add a newline in the editor it increases its height to fit. I ran this in the DevTools console: ```javascript document.querySelector('.cm-editor').style.minHeight = 'none'; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452495049,Clicking within the CodeMirror area below the SQL (i.e. when there's only a single line) doesn't cause the editor to get focused , https://github.com/simonw/datasette/issues/1900#issuecomment-1319583703,https://api.github.com/repos/simonw/datasette/issues/1900,1319583703,IC_kwDOBm6k_c5Opz_X,9599,simonw,2022-11-18T05:58:31Z,2022-11-18T05:58:31Z,OWNER,Could you provide full steps to reproduce plus a SpatiaLite database file that triggered this for you? I'm not able to recreate the problem.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319583281,https://api.github.com/repos/simonw/datasette/issues/1900,1319583281,IC_kwDOBm6k_c5Opz4x,9599,simonw,2022-11-18T05:57:44Z,2022-11-18T05:57:44Z,OWNER,"Did you use the `--spatialite` option? I just tried this: datasette package nps-spatialite.db It built the image OK (I didn't see the error you reported), but running the container failed with an error: ``` /tmp % docker run -p 8001:8001 7298e8e6bbfb Usage: datasette serve [OPTIONS] [FILES]... Try 'datasette serve --help' for help. Error: It looks like you're trying to load a SpatiaLite database without first loading the SpatiaLite module. Read more: https://docs.datasette.io/en/stable/spatialite.html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319582239,https://api.github.com/repos/simonw/datasette/issues/1900,1319582239,IC_kwDOBm6k_c5Opzof,9599,simonw,2022-11-18T05:55:38Z,2022-11-18T05:55:38Z,OWNER,"Trying this out locally with this 69MB SpatiaLite file I happened to have lying around (from testing `shapefile-to-sqlite` a while ago): https://static.simonwillison.net/static/2022/nps-spatialite.db ``` % datasette package nps-spatialite.db --spatialite ... => [2/6] COPY . /app 0.4s => [3/6] WORKDIR /app 0.0s => [4/6] RUN apt-get update && apt-get install -y python3-dev gcc libsqlite3-mod-spatialite && rm -rf /var/lib/apt/lists/* 29.6s => [5/6] RUN pip install -U datasette 12.0s => [6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json 2.6s => exporting to image 3.0s => => exporting layers 3.0s => => writing image sha256:4dfef1c373c5c057ef7ac22344f834d522acef24313a1b25d2eba9e500066b8f 0.0s ``` And then: docker run -p 8001:8001 4dfef1c373c5 This worked fine for me. I ran `datasette package` using Datasette 0.63.1.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319574972,https://api.github.com/repos/simonw/datasette/issues/1900,1319574972,IC_kwDOBm6k_c5Opx28,9599,simonw,2022-11-18T05:41:28Z,2022-11-18T05:41:28Z,OWNER,Oh this is with `datasette package`? That should work. Will investigate.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319571220,https://api.github.com/repos/simonw/datasette/issues/1900,1319571220,IC_kwDOBm6k_c5Opw8U,9599,simonw,2022-11-18T05:34:35Z,2022-11-18T05:34:35Z,OWNER,Which Docker image are you using here? It looks like it's missing SpatiaLite from the image.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1897#issuecomment-1319570586,https://api.github.com/repos/simonw/datasette/issues/1897,1319570586,IC_kwDOBm6k_c5Opwya,9599,simonw,2022-11-18T05:33:20Z,2022-11-18T05:33:20Z,OWNER,"One of the big changes still left to do for Datasette 1.0 is to unify the JSON representation with the context psssed to the templates (via an `?_extra=` mechanism to add extra context needed by the HTML templates), because a goal for 1.0 is for the template context to be a documented API contract such that custom templates won't break with future releases. As such I expect to do quite a bit of refactoring and cleanup on how the template context works later on.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1901#issuecomment-1319528359,https://api.github.com/repos/simonw/datasette/issues/1901,1319528359,IC_kwDOBm6k_c5Opmen,9599,simonw,2022-11-18T04:27:00Z,2022-11-18T04:27:00Z,OWNER,Also `datasette-indieauth` https://github.com/simonw/datasette-indieauth/blob/a08ce67ddad6098b1240adbeff37d040e4df53b1/datasette_indieauth/templates/indieauth.html#L5-L10,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319525520,https://api.github.com/repos/simonw/datasette/issues/1901,1319525520,IC_kwDOBm6k_c5OplyQ,9599,simonw,2022-11-18T04:21:14Z,2022-11-18T07:22:37Z,OWNER,This search helps too: [https://ripgrep.datasette.io/-/ripgrep?pattern=%7B%25+block+nav&literal=on&ignore=on&glob=%21datasette%2F**](https://ripgrep.datasette.io/-/ripgrep?pattern=%7B%25+block+nav&literal=on&ignore=on&glob=%21datasette%2F**),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319493475,https://api.github.com/repos/simonw/datasette/issues/1901,1319493475,IC_kwDOBm6k_c5Opd9j,9599,simonw,2022-11-18T03:19:25Z,2022-11-18T07:03:03Z,OWNER,"Other plugins this looks like it will affect: - [x] `datasette-ripgrep` https://github.com/simonw/datasette-ripgrep/blob/03446464420130368582022eeb5944993f64ec8f/datasette_ripgrep/templates/ripgrep.html#L37-L42 - [x] `datasette-socrata` https://github.com/simonw/datasette-socrata/blob/32fb256a461bf0e790eca10bdc7dd9d96c20f7c4/datasette_socrata/templates/datasette_socrata_error.html#L5-L10 - [x] `datasette-configure-fts` https://github.com/simonw/datasette-configure-fts/blob/eca742e5d4b9190fc22d68bc0a406c575e6d09a0/datasette_configure_fts/templates/configure_fts_database.html#L9-L14 - [x] `datasette-edit-templates` https://github.com/simonw/datasette-edit-templates/blob/f772aff4a2a4080c949746668a8ec6302dbeb0d9/datasette_edit_templates/templates/edit_template.html#L17-L23 - [x] `datasette-copyable` https://github.com/simonw/datasette-copyable/blob/204d5c912a8d48c49155c67fba7339d4bb26ab9a/datasette_copyable/templates/copyable.html#L36-L43 - [x] `datasette-public` https://github.com/simonw/datasette-public/blob/32b6a0ba53bd5714b6b41eddd8705b213c105efc/datasette_public/templates/public_table_change_privacy.html#L5-L11 - [x] `datasette-import-table` https://github.com/simonw/datasettecloud-datasette/blob/37d0fe525c6649c1aec3d1ee8bc35a684570e87f/templates/import_data.html#L5-L10 - [x] `datasette-edit-schema` (three places) - [x] `datasette-indieauth` https://github.com/simonw/datasette-indieauth/blob/a08ce67ddad6098b1240adbeff37d040e4df53b1/datasette_indieauth/templates/indieauth.html#L5-L10","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319483555,https://api.github.com/repos/simonw/datasette/issues/1901,1319483555,IC_kwDOBm6k_c5Opbij,9599,simonw,2022-11-18T03:02:35Z,2022-11-18T03:02:35Z,OWNER,Looks like this issue could affect a bunch of other plugins too: https://cs.github.com/?scopeName=All+repos&scope=&q=%3Cp+class%3D%22crumbs%22%3E+user%3Asimonw,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319482791,https://api.github.com/repos/simonw/datasette/issues/1901,1319482791,IC_kwDOBm6k_c5OpbWn,9599,simonw,2022-11-18T03:01:36Z,2022-11-18T03:01:36Z,OWNER,"Good catch. Looks like that bug was introduced by this change: https://github.com/simonw/datasette/commit/1a5e5f2aa951e5bd731067a49819efba68fbe8ef From: - https://github.com/simonw/datasette/issues/1831 The search all plugin includes this code which interacts poorly with that refactor: https://github.com/simonw/datasette-search-all/blob/847b55c368a285e4567627029624d7872ee75cac/datasette_search_all/templates/search_all.html#L31-L36 ```html+jinja {% block nav %}

home

{{ super() }} {% endblock %} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1897#issuecomment-1319478811,https://api.github.com/repos/simonw/datasette/issues/1897,1319478811,IC_kwDOBm6k_c5OpaYb,9599,simonw,2022-11-18T02:53:57Z,2022-11-18T02:53:57Z,OWNER,"I decided to just go for the view names, not their columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1319477721,https://api.github.com/repos/simonw/datasette/issues/1897,1319477721,IC_kwDOBm6k_c5OpaHZ,9599,simonw,2022-11-18T02:51:40Z,2022-11-18T02:51:40Z,OWNER,Views aren't currently available in the `_internal` schema.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1319435374,https://api.github.com/repos/simonw/datasette/issues/1897,1319435374,IC_kwDOBm6k_c5OpPxu,9599,simonw,2022-11-18T01:33:30Z,2022-11-18T01:33:30Z,OWNER,"Just noticed that this isn't including views, which it should.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1319401843,https://api.github.com/repos/simonw/datasette/issues/1897,1319401843,IC_kwDOBm6k_c5OpHlz,9599,simonw,2022-11-18T00:42:03Z,2022-11-18T00:42:23Z,OWNER,"This function works even if the SQLite JSON functions are not available: ```python async def _table_columns(datasette, database_name): internal = datasette.get_database(""_internal"") result = await internal.execute( ""select table_name, name from columns where database_name = ?"", [database_name], ) table_columns = {} for row in result.rows: table_columns.setdefault(row[""table_name""], []).append(row[""name""]) return table_columns ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1317840727,https://api.github.com/repos/simonw/datasette/issues/1897,1317840727,IC_kwDOBm6k_c5OjKdX,9599,simonw,2022-11-16T23:57:52Z,2022-11-16T23:57:52Z,OWNER,In terms of permissions: if you have `execute-sql` permission for a database then it's OK for you to see the table columns for that database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1317839781,https://api.github.com/repos/simonw/datasette/issues/1897,1317839781,IC_kwDOBm6k_c5OjKOl,9599,simonw,2022-11-16T23:56:47Z,2022-11-16T23:56:47Z,OWNER,I'm going to call this `table_columns` in the template context (because `schema` might mean `CREATE TABLE ...`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1317838892,https://api.github.com/repos/simonw/datasette/issues/1897,1317838892,IC_kwDOBm6k_c5OjKAs,9599,simonw,2022-11-16T23:55:42Z,2022-11-16T23:55:42Z,OWNER,"Here's where the schema is hard-coded at the moment: https://github.com/simonw/datasette/blob/00e233d7a7f6443cb95fb5227c23580c48551cad/datasette/templates/_codemirror_foot.html#L2-L7 I figured out how to extract that data from the `_internal` table in this comment: https://github.com/simonw/datasette/pull/1893#issuecomment-1317475720 Although that used JSON functions which may (in a real edge-case) not be available in the version of SQLite that Datasette is running on, so probably going to use a regular SQL query and then assemble the JSON separately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/pull/1893#issuecomment-1317837416,https://api.github.com/repos/simonw/datasette/issues/1893,1317837416,IC_kwDOBm6k_c5OjJpo,9599,simonw,2022-11-16T23:54:02Z,2022-11-16T23:54:02Z,OWNER,"I'm going to tackle #1897 in the next few minutes. Tests failed due to Prettier check, just pushed a fix so it would ignore `.bundle.js` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317831555,https://api.github.com/repos/simonw/datasette/issues/1893,1317831555,IC_kwDOBm6k_c5OjIOD,9599,simonw,2022-11-16T23:47:13Z,2022-11-16T23:47:13Z,OWNER,I'll open a follow-up issue to fix the schema.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317831425,https://api.github.com/repos/simonw/datasette/issues/1893,1317831425,IC_kwDOBm6k_c5OjIMB,9599,simonw,2022-11-16T23:47:05Z,2022-11-16T23:47:05Z,OWNER,"OK, let's do it! Thanks so much for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317829214,https://api.github.com/repos/simonw/datasette/issues/1893,1317829214,IC_kwDOBm6k_c5OjHpe,9599,simonw,2022-11-16T23:44:36Z,2022-11-16T23:44:36Z,OWNER,Deployed that to https://datasette-pr-1893.vercel.app/fixtures - looks good to me!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317797044,https://api.github.com/repos/simonw/datasette/issues/1893,1317797044,IC_kwDOBm6k_c5Oi_y0,9599,simonw,2022-11-16T23:08:34Z,2022-11-16T23:08:34Z,OWNER,"> I can push up a commit that uses the static fixtures schema for testing, but given that the query used to generate it is authed we would still need some work to make that work on live data, right? Yeah, push that up. I'm happy to wire in the query right after we land this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/issues/1896#issuecomment-1317757949,https://api.github.com/repos/simonw/datasette/issues/1896,1317757949,IC_kwDOBm6k_c5Oi2P9,9599,simonw,2022-11-16T22:27:47Z,2022-11-18T21:48:29Z,OWNER,"Open question: should `resolve_table()` know how to identify named canned queries too? I think not, at least for the moment. Feels a bit too specialist to expose in a documented API.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452364777,Extract logic for resolving a URL to a database / table / row, https://github.com/simonw/datasette/issues/1896#issuecomment-1317757112,https://api.github.com/repos/simonw/datasette/issues/1896,1317757112,IC_kwDOBm6k_c5Oi2C4,9599,simonw,2022-11-16T22:26:52Z,2022-11-16T22:33:13Z,OWNER,"Some ideas from walking the dog: Challenge: standard URL routing of request to database/table/row Standardize on the named components of the URL patterns - `database`, `table`, `pks` Async function that takes the request and the Datasette instance and returns a Resolved instance with: ``` .level - database or table or row (better name?) .database - the name of the database .db - the database object .table - the name of the table (or view) .is_view perhaps? .pk_values if it's a row ``` Should this attempt to resolve names queries too? ``` .where_sql - the where fragment you use .where_params - accompanying dictionary await datasette.resolve_request(request) ``` Or even better three methods: ```python datasette.resolve_database(request) datasette.resolve_table(request) datasette.resolve_row(request) ``` These can be typed correctly Methods raise `NotFound` if not found","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452364777,Extract logic for resolving a URL to a database / table / row, https://github.com/simonw/datasette/issues/1863#issuecomment-1317755263,https://api.github.com/repos/simonw/datasette/issues/1863,1317755263,IC_kwDOBm6k_c5Oi1l_,9599,simonw,2022-11-16T22:24:59Z,2022-11-16T22:24:59Z,OWNER,"In trying to write this I realize that there's a lot of duplicated code with delete row, specifically around resolving the incoming URL into a row (or a database or a table). Since this is so common, I think it's worth extracting the logic out first.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/pull/1893#issuecomment-1317746206,https://api.github.com/repos/simonw/datasette/issues/1893,1317746206,IC_kwDOBm6k_c5OizYe,9599,simonw,2022-11-16T22:17:24Z,2022-11-16T22:17:24Z,OWNER,Deployed 0a649e8f78c23e8db6869442eeb0dfe36a5443da: https://datasette-pr-1893.vercel.app/fixtures,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317744563,https://api.github.com/repos/simonw/datasette/issues/1893,1317744563,IC_kwDOBm6k_c5Oiy-z,9599,simonw,2022-11-16T22:16:03Z,2022-11-16T22:16:03Z,OWNER,"Honestly I'm not too bothered if table names with weird characters don't work correctly here - I care about those in the Datasette `fixtures.db` database because Datasette aims to support ANY valid SQLite database, so I need stuff in the test suite that includes weird edge cases like this. But I would hope very few people actually create tables with spaces in their names, so it's not a huge concern to me if autocompletion doesn't work properly for those.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317475720,https://api.github.com/repos/simonw/datasette/issues/1893,1317475720,IC_kwDOBm6k_c5OhxWI,9599,simonw,2022-11-16T18:25:16Z,2022-11-16T18:25:16Z,OWNER,"Here's a query that returns the exact JSON we need to pass to the schema: https://latest.datasette.io/_internal?sql=with+inner+as+%28%0D%0A++select%0D%0A++++table_name%2C%0D%0A++++json_group_array%28name%29+as+table_columns%0D%0A++from%0D%0A++++columns%0D%0A++where%0D%0A++++database_name+%3D+%3Adatabase%0D%0A++group+by%0D%0A++++table_name%0D%0A%29%0D%0Aselect%0D%0A++json_group_object%28table_name%2C+table_columns%29%0D%0Afrom%0D%0A++inner&database=fixtures ```sql with inner as ( select table_name, json_group_array(name) as table_columns from columns where database_name = :database group by table_name ) select json_group_object(table_name, table_columns) from inner ``` Returns (after pretty-printing): ```json { ""123_starts_with_digits"": [ ""content"" ], ""Table With Space In Name"": [ ""content"", ""pk"" ], ""attraction_characteristic"": [ ""name"", ""pk"" ], ""binary_data"": [ ""data"" ], ""complex_foreign_keys"": [ ""f1"", ""f2"", ""f3"", ""pk"" ], ""compound_primary_key"": [ ""content"", ""pk1"", ""pk2"" ], ""compound_three_primary_keys"": [ ""content"", ""pk1"", ""pk2"", ""pk3"" ], ""custom_foreign_key_label"": [ ""foreign_key_with_custom_label"", ""pk"" ], ""facet_cities"": [ ""id"", ""name"" ], ""facetable"": [ ""_city_id"", ""_neighborhood"", ""complex_array"", ""created"", ""distinct_some_null"", ""n"", ""on_earth"", ""pk"", ""planet_int"", ""state"", ""tags"" ], ""foreign_key_references"": [ ""foreign_key_compound_pk1"", ""foreign_key_compound_pk2"", ""foreign_key_with_blank_label"", ""foreign_key_with_label"", ""foreign_key_with_no_label"", ""pk"" ], ""infinity"": [ ""value"" ], ""no_primary_key"": [ ""a"", ""b"", ""c"", ""content"" ], ""primary_key_multiple_columns"": [ ""content"", ""content2"", ""id"" ], ""primary_key_multiple_columns_explicit_label"": [ ""content"", ""content2"", ""id"" ], ""roadside_attraction_characteristics"": [ ""attraction_id"", ""characteristic_id"" ], ""roadside_attractions"": [ ""address"", ""latitude"", ""longitude"", ""name"", ""pk"", ""url"" ], ""searchable"": [ ""name with . and spaces"", ""pk"", ""text1"", ""text2"" ], ""searchable_fts"": [ ""__langid"", ""docid"", ""name with . and spaces"", ""searchable_fts"", ""text1"", ""text2"" ], ""searchable_fts_docsize"": [ ""docid"", ""size"" ], ""searchable_fts_segdir"": [ ""end_block"", ""idx"", ""leaves_end_block"", ""level"", ""root"", ""start_block"" ], ""searchable_fts_segments"": [ ""block"", ""blockid"" ], ""searchable_fts_stat"": [ ""id"", ""value"" ], ""searchable_tags"": [ ""searchable_id"", ""tag"" ], ""select"": [ ""and"", ""group"", ""having"", ""json"" ], ""simple_primary_key"": [ ""content"", ""id"" ], ""sortable"": [ ""content"", ""pk1"", ""pk2"", ""sortable"", ""sortable_with_nulls"", ""sortable_with_nulls_2"", ""text"" ], ""table/with/slashes.csv"": [ ""content"", ""pk"" ], ""tags"": [ ""tag"" ], ""units"": [ ""distance"", ""frequency"", ""pk"" ] } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317465874,https://api.github.com/repos/simonw/datasette/issues/1893,1317465874,IC_kwDOBm6k_c5Ohu8S,9599,simonw,2022-11-16T18:21:17Z,2022-11-16T18:21:54Z,OWNER,"I was worrying about the server-side overhead of collecting together all of the tables and column names for databases that might have hundreds of tables... but then I remember that I built the `_internal` table precisely for this kind of thing - so gathering all of that data should still only be a single SQL query against an in-memory database. https://latest.datasette.io/login-as-root and then visit this page for an example query: https://latest.datasette.io/_internal?sql=select%0D%0A++database_name%2C%0D%0A++table_name%2C%0D%0A++json_group_array%28name%29%0D%0Afrom%0D%0A++columns%0D%0Awhere%0D%0A++database_name+%21%3D+%27_internal%27%0D%0Agroup+by%0D%0A++database_name%2C%0D%0A++table_name ```sql select database_name, table_name, json_group_array(name) from columns where database_name != '_internal' group by database_name, table_name ``` database_name | table_name | json_group_array(name) -- | -- | -- extra_database | searchable | [""pk"",""text1"",""text2""] extra_database | searchable_fts | [""__langid"",""content"",""docid"",""searchable_fts"",""text1"",""text2""] extra_database | searchable_fts_content | [""c0text1"",""c1text2"",""c2content"",""docid""] extra_database | searchable_fts_segdir | [""end_block"",""idx"",""leaves_end_block"",""level"",""root"",""start_block""] extra_database | searchable_fts_segments | [""block"",""blockid""] fixtures | 123_starts_with_digits | [""content""] fixtures | Table With Space In Name | [""content"",""pk""] fixtures | attraction_characteristic | [""name"",""pk""] fixtures | binary_data | [""data""] fixtures | complex_foreign_keys | [""f1"",""f2"",""f3"",""pk""] fixtures | compound_primary_key | [""content"",""pk1"",""pk2""] ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317456909,https://api.github.com/repos/simonw/datasette/issues/1893,1317456909,IC_kwDOBm6k_c5OhswN,9599,simonw,2022-11-16T18:17:39Z,2022-11-16T18:17:39Z,OWNER,"Tiny feature request (since you're in this code already) - I keep hitting Command+Enter on my macOS keyboard to submit the query, but the correct shortcut is Shift+Enter. Would be great if both worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317452541,https://api.github.com/repos/simonw/datasette/issues/1893,1317452541,IC_kwDOBm6k_c5Ohrr9,9599,simonw,2022-11-16T18:15:52Z,2022-11-16T18:15:52Z,OWNER,"Deployed latest copy with: ``` datasette publish vercel fixtures.db \ --project datasette-pr-1893 \ --about 'PR 1893' \ --about_url https://github.com/simonw/datasette/pull/1893 \ --scope datasette \ --branch eccb1c6c781d69d8ec3c542ef65c78a4a0927a7c ``` https://datasette-pr-1893.vercel.app/fixtures ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317449610,https://api.github.com/repos/simonw/datasette/issues/1893,1317449610,IC_kwDOBm6k_c5Ohq-K,9599,simonw,2022-11-16T18:14:28Z,2022-11-16T18:14:28Z,OWNER,"> I'm thinking of also adding `count` to the list since that's a common thing people would want to autocomplete. I notice BQ console highlights `count` in the same manner as other keywords like `select` as well. Huh, yeah we should definitely have `count` - surprised it's not on the list on https://www.sqlite.org/lang_keywords.html which is why we didn't get it from the GPT-3 generated schema.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316530539,https://api.github.com/repos/simonw/sqlite-utils/issues/512,1316530539,IC_kwDOCGYnMM5OeKlr,9599,simonw,2022-11-16T07:49:50Z,2022-11-16T07:49:50Z,OWNER,Tests passed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450952393,mypy failures in CI, https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316447182,https://api.github.com/repos/simonw/sqlite-utils/issues/512,1316447182,IC_kwDOCGYnMM5Od2PO,9599,simonw,2022-11-16T06:32:31Z,2022-11-16T06:32:31Z,OWNER,"Test failed again: https://github.com/simonw/sqlite-utils/actions/runs/3476950474/jobs/5812663096 `E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/s/spatialite/libsqlite3-mod-spatialite_4.3.0a-6build1_amd64.deb Unable to connect to azure.archive.ubuntu.com:http:` That looks like an intermittent error. I'll try running it again in the morning.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450952393,mypy failures in CI, https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316437748,https://api.github.com/repos/simonw/sqlite-utils/issues/512,1316437748,IC_kwDOCGYnMM5Odz70,9599,simonw,2022-11-16T06:24:31Z,2022-11-16T06:24:31Z,OWNER,"``` sqlite-utils % pipx run no_implicit_optional . Calculating full-repo metadata... Executing codemod... 11.43s 98% complete, 0.24s estimated for 5 files to go... ``` Then: ``` Finished codemodding 239 files! - Transformed 239 files successfully. - Skipped 0 files. - Failed to codemod 0 files. - 0 warnings were generated. ``` Here's the diff: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a06f4b7..e819d17 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -297,12 +297,12 @@ class Database: def __init__( self, - filename_or_conn: Union[str, pathlib.Path, sqlite3.Connection] = None, + filename_or_conn: Optional[Union[str, pathlib.Path, sqlite3.Connection]] = None, memory: bool = False, - memory_name: str = None, + memory_name: Optional[str] = None, recreate: bool = False, recursive_triggers: bool = True, - tracer: Callable = None, + tracer: Optional[Callable] = None, use_counts_table: bool = False, ): assert (filename_or_conn is not None and (not memory and not memory_name)) or ( @@ -341,7 +341,7 @@ class Database: self.conn.close() @contextlib.contextmanager - def tracer(self, tracer: Callable = None): + def tracer(self, tracer: Optional[Callable] = None): """""" Context manager to temporarily set a tracer function - all executed SQL queries will be passed to this. @@ -378,7 +378,7 @@ class Database: def register_function( self, - fn: Callable = None, + fn: Optional[Callable] = None, deterministic: bool = False, replace: bool = False, name: Optional[str] = None, @@ -879,7 +879,7 @@ class Database: pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, hash_id: Optional[str] = None, hash_id_columns: Optional[Iterable[str]] = None, @@ -1129,7 +1129,7 @@ class Database: sql += "" [{}]"".format(name) self.execute(sql) - def init_spatialite(self, path: str = None) -> bool: + def init_spatialite(self, path: Optional[str] = None) -> bool: """""" The ``init_spatialite`` method will load and initialize the SpatiaLite extension. The ``path`` argument should be an absolute path to the compiled extension, which @@ -1182,7 +1182,7 @@ class Queryable: def count_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, ) -> int: """""" @@ -1213,12 +1213,12 @@ class Queryable: def rows_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, - order_by: str = None, + order_by: Optional[str] = None, select: str = ""*"", - limit: int = None, - offset: int = None, + limit: Optional[int] = None, + offset: Optional[int] = None, ) -> Generator[dict, None, None]: """""" Iterate over every row in this table or view that matches the specified where clause. @@ -1251,11 +1251,11 @@ class Queryable: def pks_and_rows_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, - order_by: str = None, - limit: int = None, - offset: int = None, + order_by: Optional[str] = None, + limit: Optional[int] = None, + offset: Optional[int] = None, ) -> Generator[Tuple[Any, Dict], None, None]: """""" Like ``.rows_where()`` but returns ``(pk, row)`` pairs - ``pk`` can be a single value or tuple. @@ -1345,7 +1345,7 @@ class Table(Queryable): pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, batch_size: int = 100, hash_id: Optional[str] = None, @@ -1545,7 +1545,7 @@ class Table(Queryable): pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, hash_id: Optional[str] = None, hash_id_columns: Optional[Iterable[str]] = None, @@ -2464,7 +2464,7 @@ class Table(Queryable): columns: Optional[Iterable[str]] = None, limit: Optional[int] = None, offset: Optional[int] = None, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, quote: bool = False, ) -> Generator[dict, None, None]: @@ -2527,7 +2527,7 @@ class Table(Queryable): def delete_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, analyze: bool = False, ) -> ""Table"": ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450952393,mypy failures in CI, https://github.com/simonw/datasette/pull/1893#issuecomment-1316412234,https://api.github.com/repos/simonw/datasette/issues/1893,1316412234,IC_kwDOBm6k_c5OdttK,9599,simonw,2022-11-16T06:00:39Z,2022-11-16T06:01:36Z,OWNER,"Should note though that this is a classic example of GPT-3 making stuff up in places. > current: Returns the current date, time, or timestamp `select current` throws an error for me: https://latest.datasette.io/_memory?sql=select+current `select current_date, current_time, current_timestamp` works though: https://latest.datasette.io/_memory?sql=select+current_date%2C+current_time%2C+current_timestamp So let's drop `current` from the list. I'm OK with it though, I think it's likely good enough for the first attempt at this. We should drop `temp` and `temporary` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316401895,https://api.github.com/repos/simonw/datasette/issues/1893,1316401895,IC_kwDOBm6k_c5OdrLn,9599,simonw,2022-11-16T05:50:40Z,2022-11-16T05:50:40Z,OWNER,"So I think our dialect (at least to start with) should be: ``` keywords: ""and as asc between by case cast current current_date current_time current_timestamp desc distinct each else escape except exists explain filter first for from full generated group having if in index inner intersect into isnull join last left like limit not null or order outer over pragma primary query raise range regexp right rollback row select set table temp temporary then to union unique using values view virtual when where"", // https://www.sqlite.org/datatype3.html types: ""null integer real text blob"", builtin: """" ``` I left `builtin` blank here because I don't think we need any of those things at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316400688,https://api.github.com/repos/simonw/datasette/issues/1893,1316400688,IC_kwDOBm6k_c5Odq4w,9599,simonw,2022-11-16T05:49:31Z,2022-11-16T05:49:31Z,OWNER,"OK, I decided to play around with GPT-3 for this. https://beta.openai.com/playground I prompted: ``` For each keyword say it it can be used in a SQLite select statement: and as asc between by case cast cross current current_date current_time current_timestamp desc detach distinct each else end escape except exclude exclusive exists explain fail filter first following for foreign from full generated glob group groups having if ignore immediate in index indexed initially inner insert instead intersect into is isnull join key last left like limit match materialized natural no not nothing notnull null nulls of offset on or order others outer over partition plan pragma preceding primary query raise range recursive references regexp reindex release rename replace restrict returning right rollback row rows savepoint select set table temp temporary then ties to transaction trigger unbounded union unique update using vacuum values view virtual when where window with without - and: yes - as: yes - asc: yes - between: yes - by: yes - case: yes - cast: yes - current: yes - current_date: yes - current_time: yes - current_timestamp: yes - desc: yes - detach: no - distinct: yes - each: yes - else: yes - end: yes - escape: yes - except: no - exclude: yes - exclusive: no ``` Having given it these examples I ran it a few times: ``` - exists: yes - explain: no - fail: no - filter: yes - first: yes - following: no - for: yes - foreign: no - from: yes - full: no - generated: no - glob: yes - group: yes - groups: no - having: yes - if: yes - ignore: no - immediate: no - in: yes - index: yes - indexed: no - initially: no - inner: no - insert: no - instead: no - intersect: no - into: yes - is: yes - isnull: yes - join: yes - key: no - last: yes - left: yes - like: yes - limit: yes - match: no - materialized: no -natural: yes - no: no - not: no - nothing: no - notnull: yes - null: yes - nulls: yes - of: no - offset: yes - on: yes - or: yes - order: yes - others: yes - outer: no - over: no - partition: no - pragma: no - preceding: no - primary: no - query: no - raise: no - range: no - recursive: no - references: no - regexp: yes - reindex: no - release: no - rename: no - replace: no - restrict: no - returning: yes - right: yes - rollback: no - row: yes - rows: no - savepoint: no - select: yes - set: no - table: yes - temp: yes - temporary: yes - then: yes - ties: no - to: yes - transaction: no - trigger: no - unbounded: no - union: yes - unique: yes - update: no - using: yes - vacuum: no - values: yes - view: yes - virtual: no - when: yes - where: yes - window: no - with: yes - without: no ``` I added this: ``` A space separated list of all of the keywords for which the answer was yes: ``` It replied: ``` and as asc between by case cast current current_date current_time current_timestamp desc distinct each else escape except exists explain filter first for from full generated group having if in index inner intersect into isnull join last left like limit not null or order outer over pragma primary query raise range regexp right rollback row select set table temp temporary then to union unique using values view virtual when where ``` Then I tried doing this: ``` Produce a short summary of how each of those are used: - and: ``` This kind of worked too! It ended up defining every keyword, not just the ones for which the ""is it used in select"" was ""yes"", but it gave me this list: > - and: Used to combine multiple conditions in a WHERE clause > - as: Rename a column or table in the output of a query > - asc: Sort results in ascending order > - between: Used in a WHERE clause to specify a range > - by: Specifies how to sort the results of a query > - case: Used to create different outputs based on conditions > - cast: Converts a value from one data type to another > - current: Returns the current date, time, or timestamp > - current_date: Returns the current date > - current_time: Returns the current time > - current_timestamp: Returns the current timestamp > - desc: Sort results in descending order > - distinct: Selects only unique values > - each: Used with aggregate functions > - else: Used with the CASE statement > - escape: Used to escape special characters in a string > - except: Returns all rows from the first query that are not in the second query > - exists: Used with subqueries > - explain: Analyzes a query and returns information about it > - fail: Used with the ON CONFLICT clause > - filter: Used to specify conditions for aggregate functions > - first: Used to specify conditions for aggregate functions > - following: Used with window functions > - for: Used to specify conditions for aggregate functions > - foreign: Creates a foreign key > - from: Specifies the table(s) to select data from > - full: Used with outer joins > - generated: Used with generated columns > - glob: Used to match a string with a pattern > - group: Groups together rows that have the same values > - groups: Used with aggregate functions > - having: Specifies conditions for groups > - if: Used with the ON CONFLICT clause > - ignore: Used with the ON CONFLICT clause > - immediate: Used with the ON CONFLICT clause > - in: Used to check if a value is in a list > - index: Creates an index on a column or columns > - indexed: Used with the ON CONFLICT clause > - initially: Used with the ON CONFLICT clause > - inner: Creates an inner join > - insert: Inserts new data into a table > - instead: Used with triggers > - intersect: Returns all rows that are in both queries > - into: Specifies the table to insert data into > - is: Used to compare values > - isnull: Checks if a value is null > - join: Joins two or more tables together > - key: Creates a primary key or unique key > - last: Used to specify conditions for aggregate functions > - left: Creates a left join > - like: Checks if a value matches a pattern > - limit: Specifies the maximum number of rows to return > - not: Negates a condition > - null: Specifies a null value > - nulls: Used with the ORDER BY clause > - of: Used with window functions > - offset: Specifies the number of rows to skip > - on: Specifies conditions for a join > - or: Checks if either condition is true > - order: Specifies how to sort the results of a query > - others: Used with the CASE statement > - outer: Creates an outer join > - over: Used with window functions > - partition: Used with window functions > - pragma: Used to query or change various internal settings > - preceding: Used with window functions > - primary: Creates a primary key > - query: Used with the ON CONFLICT clause > - raise: Used with the ON CONFLICT clause > - range: Used with window functions > - recursive: Used with common table expressions > - references: Creates a foreign key > - regexp: Checks if a value matches a regular expression > - reindex: Recreates an index > - release: Used with savepoints > - rename: Changes the name of a table or column > - replace: Inserts new data into a table or updates existing data > - restrict: Used with savepoints > - returning: Returns data from inserted or updated rows > - right: Creates a right join > - rollback: Rolls back a transaction > - row: Used with aggregate functions > - select: Selects data from a table > - set: Changes the value of a column > - table: Specifies the table(s) to select data from > - temp: Creates a temporary table > - temporary: Creates a temporary table > - then: Used with the CASE statement > - ties: Used with the ORDER BY clause > - to: Used with common table expressions > - transaction: Starts a transaction > - trigger: Creates a trigger > - unbounded: Used with window functions > - union: Returns all rows from both queries > - unique: Creates a unique constraint > - update: Updates existing data in a table > - using: Specifies the table(s) to join > - vacuum: Optimizes a database > - values: Specifies the values to insert into a table > - view: Creates a view > - virtual: Creates a virtual table > - when: Used with the CASE statement > - where: Specifies conditions for a query > - window: Used with window functions > - with: Specifies common table expressions used in a query > - without: Used with indexes","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316340865,https://api.github.com/repos/simonw/datasette/issues/1893,1316340865,IC_kwDOBm6k_c5OdcSB,9599,simonw,2022-11-16T04:49:30Z,2022-11-16T04:49:43Z,OWNER,"> The main issue is that we don't pass the relevant table data down to QueryView. If you can come up with a static example JSON data structure example that does the right thing, I'm happy to refactor QueryView to make that available to the template - or even have a separate `fetch()` that grabs just the data needed for the autocomplete as a separate hit when the page loads (whichever has better performance implications). I'm working a fair amount in the view classes at the moment so adding this to that work would make sense. ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316294156,https://api.github.com/repos/simonw/datasette/issues/1893,1316294156,IC_kwDOBm6k_c5OdQ4M,9599,simonw,2022-11-16T04:00:12Z,2022-11-16T04:00:12Z,OWNER,Have you ever seen CodeMirror correctly auto-completing columns? I'm not entirely sure I believe that the feature works anywhere else.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316293353,https://api.github.com/repos/simonw/datasette/issues/1893,1316293353,IC_kwDOBm6k_c5OdQrp,9599,simonw,2022-11-16T03:59:03Z,2022-11-16T03:59:03Z,OWNER,"Deployed a fresh copy: ``` datasette publish vercel fixtures.db \ --branch b7b2942b13f9ea09cfa9f8c73e2869b9bd2349ae \ --project datasette-pr-1893 \ --about 'PR 1893' \ --about_url https://github.com/simonw/datasette/pull/1893 \ --scope datasette ``` https://datasette-pr-1893.vercel.app/fixtures","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/issues/1890#issuecomment-1316262169,https://api.github.com/repos/simonw/datasette/issues/1890,1316262169,IC_kwDOBm6k_c5OdJEZ,9599,simonw,2022-11-16T03:22:40Z,2022-11-16T03:22:40Z,OWNER,"Actually this works as it should in desktop Safari: ![autocomplete-safari](https://user-images.githubusercontent.com/9599/202075764-fbc4b4c8-c92f-4f69-81fd-84002de5aea7.gif) I'm going to just put up with the weird behaviour in Mobile Safari.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/pull/1893#issuecomment-1316253186,https://api.github.com/repos/simonw/datasette/issues/1893,1316253186,IC_kwDOBm6k_c5OdG4C,9599,simonw,2022-11-16T03:16:36Z,2022-11-16T03:16:36Z,OWNER,Yeah I haven't written this down anywhere but Datasette definitely has an undocumented preference for lower-case SQL.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/issues/1890#issuecomment-1316242752,https://api.github.com/repos/simonw/datasette/issues/1890,1316242752,IC_kwDOBm6k_c5OdEVA,9599,simonw,2022-11-16T03:10:52Z,2022-11-16T03:12:47Z,OWNER,"https://bugs.webkit.org/show_bug.cgi?id=201768 - "" Datalist option's label not used"" - marked as RESOLVED FIXED on March 31st 2020. The commit: https://trac.webkit.org/changeset/259330/webkit And here's the test mirrored on GitHub: https://cs.github.com/qtwebkit/webkit-mirror/blob/cc3fcd0b4bad1f7cf77c26e34aa01d16618d6d5e/LayoutTests/fast/forms/datalist/datalist-option-labels.html?q=datalist-option-labels.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1316240839,https://api.github.com/repos/simonw/datasette/issues/1890,1316240839,IC_kwDOBm6k_c5OdD3H,9599,simonw,2022-11-16T03:09:11Z,2022-11-16T03:09:11Z,OWNER,"Here's a polyfill for ``: https://github.com/mfranzke/datalist-polyfill It shouldn't be necessary now that Safari has shipped support (apparently added in https://developer.apple.com/documentation/safari-release-notes/safari-12_1-release-notes#3130314 Safari 12.1 in March 2019). But it does look like Safari doesn't support differing `label` and `value` attributes, though documentation about this is hard to come by.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/pull/1893#issuecomment-1316236448,https://api.github.com/repos/simonw/datasette/issues/1893,1316236448,IC_kwDOBm6k_c5OdCyg,9599,simonw,2022-11-16T03:04:57Z,2022-11-16T03:04:57Z,OWNER,If you rebase from `main` you should get the fix for that test failure.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/issues/1890#issuecomment-1316233532,https://api.github.com/repos/simonw/datasette/issues/1890,1316233532,IC_kwDOBm6k_c5OdCE8,9599,simonw,2022-11-16T03:00:58Z,2022-11-16T03:00:58Z,OWNER,"Oops, introduced a test failure: ``` def test_table_html_foreign_key_facets(app_client): response = app_client.get( ""/fixtures/foreign_key_references?_facet=foreign_key_with_blank_label"" ) assert response.status == 200 > assert ( '
  • ' ""- 1
    • "" ) in response.text E assert '
  • - 1
    • ' in '\n\n\n fixtures: foreign_key_references: 2 rows\n \n\n\n\n\n' E + where '\n\n\n fixtures: foreign_key_references: 2 rows\n \n\n\n\n\n' = .text ``` Need to fix this test: https://github.com/simonw/datasette/blob/eac028d3f77aa5473a5fcf59240635a1bca80f7d/tests/test_table_html.py#L616-L624","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/pull/1893#issuecomment-1316232588,https://api.github.com/repos/simonw/datasette/issues/1893,1316232588,IC_kwDOBm6k_c5OdB2M,9599,simonw,2022-11-16T03:00:04Z,2022-11-16T03:00:04Z,OWNER,"Oops, the tests are failing because of a test failure I introduced here: - #1890","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316231560,https://api.github.com/repos/simonw/datasette/issues/1893,1316231560,IC_kwDOBm6k_c5OdBmI,9599,simonw,2022-11-16T02:59:00Z,2022-11-16T02:59:00Z,OWNER,"The resize handle doesn't appear on Mobile Safari on iPhone - I don't think that particularly matters though. The textarea does get a weird border around it when focused on iPhone though. Focused: ![BF34E8FB-E35C-4CAB-9BFB-8EEF7E29B16C_1_201_a](https://user-images.githubusercontent.com/9599/202072748-c85bab94-a039-4ed6-8185-3cac25c78ed3.jpeg) Not focused: ![31A5CF38-D540-4A1A-8A7D-E29453D150F4_1_201_a](https://user-images.githubusercontent.com/9599/202072744-d9f0ea62-13b7-46ff-afe1-6d88d7fb8b53.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316227073,https://api.github.com/repos/simonw/datasette/issues/1893,1316227073,IC_kwDOBm6k_c5OdAgB,9599,simonw,2022-11-16T02:54:22Z,2022-11-16T02:54:32Z,OWNER,"If you can get a version of this working with table and column autocompletion just using a static JavaScript object in the source code with the right tables and columns, I'm happy to take on the work of turning that static object into something that Datasette includes in the page itself with all of the correct values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316141764,https://api.github.com/repos/simonw/datasette/issues/1893,1316141764,IC_kwDOBm6k_c5OcrrE,9599,simonw,2022-11-16T01:26:59Z,2022-11-16T01:26:59Z,OWNER,"Resizing works great for me - and the page automatically sizes the editor to fit an existing query, e.g. on https://datasette-pr-1893.vercel.app/fixtures?sql=select+id%2C+content%2C+content2%0D%0A++from+primary_key_multiple_columns_explicit_label%0D%0A++order+by+id%0D%0A++limit+101","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316137982,https://api.github.com/repos/simonw/datasette/issues/1893,1316137982,IC_kwDOBm6k_c5Ocqv-,9599,simonw,2022-11-16T01:23:47Z,2022-11-16T01:23:47Z,OWNER,"Autocomplete here looks promising (I've wanted that to work for years!), but it does currently show a whole bunch of suggestions which aren't part of the SQLite SQL dialect: ![autocomplete](https://user-images.githubusercontent.com/9599/202060211-51ec9f45-bc52-459a-a729-27fc2faadff9.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316135244,https://api.github.com/repos/simonw/datasette/issues/1893,1316135244,IC_kwDOBm6k_c5OcqFM,9599,simonw,2022-11-16T01:21:41Z,2022-11-16T01:21:41Z,OWNER,"I just deployed a demo instance like this (using the commit hash from this PR): ```bash datasette publish vercel fixtures.db \ --branch 544f7025900b78f63c34b9985522271ba5fd9c0f \ --project datasette-pr-1893 \ --scope datasette \ --about 'PR 1893' \ --about_url https://github.com/simonw/datasette/pull/1893 ``` Here's the result: https://datasette-pr-1893.vercel.app/fixtures","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/issues/1863#issuecomment-1315812212,https://api.github.com/repos/simonw/datasette/issues/1863,1315812212,IC_kwDOBm6k_c5ObbN0,9599,simonw,2022-11-15T20:12:02Z,2022-11-15T20:12:02Z,OWNER,"If the update succeeds it will return `{""ok"": true}`. For consistency with `/db/table/-/insert` you can pass `""return"": true` and it will return a `""row""` key with the now-updated full row.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1315809867,https://api.github.com/repos/simonw/datasette/issues/1863,1315809867,IC_kwDOBm6k_c5ObapL,9599,simonw,2022-11-15T20:09:44Z,2022-11-15T20:09:44Z,OWNER,"I'm also not going to implement `""alter"": true` yet (which would add any missing columns based on the update) - I'll hold that off for a later feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1315809260,https://api.github.com/repos/simonw/datasette/issues/1863,1315809260,IC_kwDOBm6k_c5Obafs,9599,simonw,2022-11-15T20:09:11Z,2022-11-15T20:09:11Z,OWNER,"I'm going to use the error format I've been experimenting with here: - #1875 ```json { ""type"": ""https://example.net/validation-error"", ""title"": ""Your request is not valid."", ""errors"": [ { ""detail"": ""must be a positive integer"", ""pointer"": ""#/age"" }, { ""detail"": ""must be 'green', 'red' or 'blue'"", ""pointer"": ""#/profile/color"" } ] } ``` I'm not quite ready to commit to a `type` URL though, so I'll leave that to be solved later should I fully embrace that RFC.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1315808062,https://api.github.com/repos/simonw/datasette/issues/1863,1315808062,IC_kwDOBm6k_c5ObaM-,9599,simonw,2022-11-15T20:08:04Z,2022-11-15T20:08:04Z,OWNER,"The initial design I'm going to implement will look like this: ``` POST /db/table/1/-/update Authorization: Bearer xxx Content-Type: application/json ``` ```json { ""update"": { ""name"": ""New name"" } } ``` Any fields that are not yet columns will return an error. Should it enforce types, in as much as an integer column should have a JSON integer passed to it, or should it allow strings containing valid integers? I'm going to allow strings, mainly as a workaround for the fact that JavaScript integers have a maximum size.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1892#issuecomment-1315805498,https://api.github.com/repos/simonw/datasette/issues/1892,1315805498,IC_kwDOBm6k_c5ObZk6,9599,simonw,2022-11-15T20:05:30Z,2022-11-15T20:05:30Z,OWNER,"One slight concern: https://latest.datasette.io/ will increasingly reflect a version that isn't the most recent production release. I might setup https://stable.datasette.io/ as a demo instance of the most recent non-alpha release to compensate for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450312343,Merge 1.0-dev branch back to main, https://github.com/simonw/datasette/issues/1892#issuecomment-1315804535,https://api.github.com/repos/simonw/datasette/issues/1892,1315804535,IC_kwDOBm6k_c5ObZV3,9599,simonw,2022-11-15T20:04:38Z,2022-11-15T20:04:38Z,OWNER,"I'll do this after the 1.0a0 release: - #1708","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450312343,Merge 1.0-dev branch back to main, https://github.com/simonw/datasette/issues/1890#issuecomment-1314891228,https://api.github.com/repos/simonw/datasette/issues/1890,1314891228,IC_kwDOBm6k_c5OX6Xc,9599,simonw,2022-11-15T07:23:01Z,2022-11-15T07:23:01Z,OWNER,"Annoying: Mobile Safari doesn't seem to support separate labels and values. I should probably disable this feature on that browser, at least for foreign key facets (for the moment).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314856513,https://api.github.com/repos/simonw/datasette/issues/1890,1314856513,IC_kwDOBm6k_c5OXx5B,9599,simonw,2022-11-15T06:56:29Z,2022-11-15T06:56:29Z,OWNER,"Looks like I can fix that like so: ```html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314850524,https://api.github.com/repos/simonw/datasette/issues/1890,1314850524,IC_kwDOBm6k_c5OXwbc,9599,simonw,2022-11-15T06:48:37Z,2022-11-15T06:48:37Z,OWNER,"Spotted a bug with this on https://latest.datasette.io/fixtures/facetable?_facet=_city_id - the `_city_id` column is a foreign key, so you need to type `1` or `2` - but the autocomplete list shows the full text names for the cities.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314849867,https://api.github.com/repos/simonw/datasette/issues/1890,1314849867,IC_kwDOBm6k_c5OXwRL,9599,simonw,2022-11-15T06:47:51Z,2022-11-15T06:47:51Z,OWNER,Demo now live here: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=party - select `party` and start typing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314848432,https://api.github.com/repos/simonw/datasette/issues/1890,1314848432,IC_kwDOBm6k_c5OXv6w,9599,simonw,2022-11-15T06:46:08Z,2022-11-15T06:46:08Z,OWNER,Wrote a TIL about ``: https://til.simonwillison.net/html/datalist,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1862#issuecomment-1314845667,https://api.github.com/repos/simonw/datasette/issues/1862,1314845667,IC_kwDOBm6k_c5OXvPj,9599,simonw,2022-11-15T06:42:09Z,2022-11-15T06:42:32Z,OWNER,"I implemented this as part of `/db/-/create`. https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table-from-example-data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425011030,"Create a new table from one or more records, `sqlite-utils` style", https://github.com/simonw/datasette/issues/1890#issuecomment-1314835740,https://api.github.com/repos/simonw/datasette/issues/1890,1314835740,IC_kwDOBm6k_c5OXs0c,9599,simonw,2022-11-15T06:30:26Z,2022-11-15T06:30:26Z,OWNER,That prototype actually works really well! I'm going to add that to `table.js`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314833881,https://api.github.com/repos/simonw/datasette/issues/1890,1314833881,IC_kwDOBm6k_c5OXsXZ,9599,simonw,2022-11-15T06:27:21Z,2022-11-15T06:27:21Z,OWNER,"Here's a prototype: ```javascript function createDataLists() { var facetResults = document.querySelectorAll("".facet-results [data-column]""); Array.from(facetResults).forEach(function (facetResult) { // Use link text from all links in the facet result var linkTexts = Array.from( facetResult.querySelectorAll(""li:not(.facet-truncated) a"") ).map(function (link) { return link.textContent; }); // Create a datalist element var datalist = document.createElement(""datalist""); datalist.id = ""datalist-"" + facetResult.dataset.column; // Create an option element for each link text linkTexts.forEach(function (linkText) { var option = document.createElement(""option""); option.value = linkText; datalist.appendChild(option); }); // Add the datalist to the facet result facetResult.appendChild(datalist); }); } createDataLists(); // When any select with name=_filter_column changes, update the datalist document.body.addEventListener(""change"", function (event) { if (event.target.name === ""_filter_column"") { event.target .closest("".filter-row"") .querySelector("".filter-value"") .setAttribute(""list"", ""datalist-"" + event.target.value); } }); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314829751,https://api.github.com/repos/simonw/datasette/issues/1890,1314829751,IC_kwDOBm6k_c5OXrW3,9599,simonw,2022-11-15T06:20:50Z,2022-11-15T06:20:50Z,OWNER,"This finds the right links on the page: document.querySelectorAll('.facet-results [data-column] li:not(.facet-truncated) a')","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314825019,https://api.github.com/repos/simonw/datasette/issues/1890,1314825019,IC_kwDOBm6k_c5OXqM7,9599,simonw,2022-11-15T06:13:36Z,2022-11-15T06:13:36Z,OWNER,"This could start out as a purely JavaScript enhancement for pages that already figured out the available values through faceting, like you suggested.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314823752,https://api.github.com/repos/simonw/datasette/issues/1890,1314823752,IC_kwDOBm6k_c5OXp5I,9599,simonw,2022-11-15T06:11:49Z,2022-11-15T06:11:49Z,OWNER,"I tried this out on https://congress-legislators.datasettes.com/legislators/legislator_terms for the `party` column - here's the demo: ![datalist](https://user-images.githubusercontent.com/9599/201839812-db887ce0-c4b9-432c-8620-5ac73f222a63.gif) I made this work by dropping the following HTML into the page in the browser DevTools: ```html ``` And then adding `list=""party""` to the input element in the filter form.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314821337,https://api.github.com/repos/simonw/datasette/issues/1890,1314821337,IC_kwDOBm6k_c5OXpTZ,9599,simonw,2022-11-15T06:08:19Z,2022-11-15T06:08:19Z,OWNER,"Oh interesting... this doesn't even need to be attached to the visible faceting feature, necessarily: Datasette could try to detect when a column has a limited number of options (which the faceting code handles already) and could turn those into an auto-complete interface. There's actually a native HTML element for this these days: the `` https://developer.mozilla.org/en-US/docs/Web/HTML/Element/datalist","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1882#issuecomment-1314813205,https://api.github.com/repos/simonw/datasette/issues/1882,1314813205,IC_kwDOBm6k_c5OXnUV,9599,simonw,2022-11-15T06:00:41Z,2022-11-15T06:00:41Z,OWNER,"Documentation: - https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table - https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table-from-example-data Wrote a TIL about how I wrote some of those tests with Copilot: https://til.simonwillison.net/gpt3/writing-test-with-copilot","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1875#issuecomment-1314620086,https://api.github.com/repos/simonw/datasette/issues/1875,1314620086,IC_kwDOBm6k_c5OW4K2,9599,simonw,2022-11-15T01:09:56Z,2022-11-15T01:09:56Z,OWNER,"Rough initial prototype: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 8b987221..518ac578 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1103,19 +1103,30 @@ class TableInsertView(BaseView): except json.JSONDecodeError as e: return _errors([""Invalid JSON: {}"".format(e)]) if not isinstance(data, dict): - return _errors([""JSON must be a dictionary""]) + return _errors([{""detail"": ""JSON must be a dictionary"", ""pointer"": ""#/""}]) keys = data.keys() # keys must contain ""row"" or ""rows"" if ""row"" not in keys and ""rows"" not in keys: return _errors(['JSON must have one or other of ""row"" or ""rows""']) rows = [] + was_single_row = False if ""row"" in keys: if ""rows"" in keys: - return _errors(['Cannot use ""row"" and ""rows"" at the same time']) + return _errors( + [ + { + ""detail"": 'Cannot use ""row"" and ""rows"" at the same time', + ""pointer"": ""#/row"", + } + ] + ) + was_single_row = True row = data[""row""] if not isinstance(row, dict): - return _errors(['""row"" must be a dictionary']) + return _errors( + [{""detail"": '""row"" must be a dictionary', ""pointer"": ""#/row""}] + ) rows = [row] data[""return""] = True else: @@ -1152,9 +1163,12 @@ class TableInsertView(BaseView): invalid_columns = set(row.keys()) - columns if invalid_columns: errors.append( - ""Row {} has invalid columns: {}"".format( - i, "", "".join(sorted(invalid_columns)) - ) + { + ""detail"": ""Invalid columns: {}"".format( + "", "".join(sorted(invalid_columns)) + ), + ""pointer"": ""#/blah/"", + } ) if errors: return _errors(errors) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314615592,https://api.github.com/repos/simonw/datasette/issues/1875,1314615592,IC_kwDOBm6k_c5OW3Eo,9599,simonw,2022-11-15T01:04:28Z,2022-11-15T01:04:28Z,OWNER,"Worth noting this bit in RFC 7807: > The fictional problem type here defines the ""errors"" extension, an > array that describes the details of each validation error. Each > member is an object containing ""detail"" to describe the issue, and > ""pointer"" to locate the problem within the request's content using a > JSON Pointer [JSON-POINTER]. So the list of `""errors""` with JSON Pointer isn't technically part of the spec, it's an imaginary extension. It fits what I need to do though, so I'm inclined to stick with it anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314545407,https://api.github.com/repos/simonw/datasette/issues/1875,1314545407,IC_kwDOBm6k_c5OWl7_,9599,simonw,2022-11-14T23:30:34Z,2022-11-14T23:30:34Z,OWNER,TIL: https://til.simonwillison.net/json/json-pointer,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314491884,https://api.github.com/repos/simonw/datasette/issues/1875,1314491884,IC_kwDOBm6k_c5OWY3s,9599,simonw,2022-11-14T22:26:11Z,2022-11-14T22:26:54Z,OWNER,"Spec looks pretty simple: > A JSON Pointer is a Unicode string (see [RFC4627], Section 3) > containing a sequence of zero or more reference tokens, each prefixed > by a `/` (%x2F) character. > > Because the characters `~` (%x7E) and `/` (%x2F) have special > meanings in JSON Pointer, `~` needs to be encoded as `~0` and `/` > needs to be encoded as `~1` when these characters appear in a > reference token.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314491150,https://api.github.com/repos/simonw/datasette/issues/1875,1314491150,IC_kwDOBm6k_c5OWYsO,9599,simonw,2022-11-14T22:25:20Z,2022-11-14T22:25:20Z,OWNER,"That's using JSON Pointer: https://www.rfc-editor.org/rfc/rfc6901 There's a Python library for that here https://github.com/stefankoegl/python-json-pointer/blob/master/jsonpointer.py - which looks simple and clean and well maintained and documented, but it only handles the ""what is at this pointer within this JSON object"" case - I need to generate the correct JSON pointer to explain where my error is. So I think I'll end up hand-rolling this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314488010,https://api.github.com/repos/simonw/datasette/issues/1875,1314488010,IC_kwDOBm6k_c5OWX7K,9599,simonw,2022-11-14T22:21:43Z,2022-11-14T22:21:43Z,OWNER,"Here's the most relevant example from the RFC spec: ``` POST /details HTTP/1.1 Host: account.example.com Accept: application/json ``` ```json { ""age"": 42.3, ""profile"": { ""color"": ""yellow"" } } ``` ``` HTTP/1.1 400 Bad Request Content-Type: application/problem+json Content-Language: en ``` ```json { ""type"": ""https://example.net/validation-error"", ""title"": ""Your request is not valid."", ""errors"": [ { ""detail"": ""must be a positive integer"", ""pointer"": ""#/age"" }, { ""detail"": ""must be 'green', 'red' or 'blue'"", ""pointer"": ""#/profile/color"" } ] } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1884#issuecomment-1314054300,https://api.github.com/repos/simonw/datasette/issues/1884,1314054300,IC_kwDOBm6k_c5OUuCc,9599,simonw,2022-11-14T16:40:06Z,2022-11-14T16:40:06Z,OWNER,"I wonder if there are any reasons that inspect SHOULD try to count virtual tables? Like are there any likely uses for a cirial table where the count is both interesting and likely to be accessed often enough that it's worth caching? I have an issue open to add a setting to disable table counts entirely: - #1818 Maybe that should be expanded to automatically disable row counts for virtual tables entirely? Which would mean no count would be shown for them in the UI. If you desperately wanted a count you would then have to run a count(*) query against them explicitly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/datasette/issues/1850#issuecomment-1313156167,https://api.github.com/repos/simonw/datasette/issues/1850,1313156167,IC_kwDOBm6k_c5ORSxH,9599,simonw,2022-11-14T06:23:39Z,2022-11-14T06:23:39Z,OWNER,The API explorer is now live here: https://latest-1-0-dev.datasette.io/-/api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1313155712,https://api.github.com/repos/simonw/datasette/issues/1850,1313155712,IC_kwDOBm6k_c5ORSqA,9599,simonw,2022-11-14T06:22:57Z,2022-11-14T06:22:57Z,OWNER,"I think the ability to create tokens should be protected by a `create-tokens` permission, not just a global setting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1855#issuecomment-1313148519,https://api.github.com/repos/simonw/datasette/issues/1855,1313148519,IC_kwDOBm6k_c5ORQ5n,9599,simonw,2022-11-14T06:13:43Z,2022-12-13T02:46:51Z,OWNER,"The `datasette create-token` command will need to be able to do this too. Right now that command looks like this: ``` % datasette create-token --help Usage: datasette create-token [OPTIONS] ID Create a signed API token for the specified actor ID Options: --secret TEXT Secret used for signing the API tokens [required] -e, --expires-after INTEGER Token should expire after this many seconds --debug Show decoded token --help Show this message and exit. ``` ``` % datasette create-token root --secret sec --debug -e 445 dstok_eyJhIjoicm9vdCIsInRva2VuIjoiZHN0b2siLCJ0IjoxNjY4NDA2MjEzLCJkIjo0NDV9.Hd6qRli6xRKkOIRQgZkPO5iN1wM Decoded: { ""a"": ""root"", ""token"": ""dstok"", ""t"": 1668406213, ""d"": 445 } ``` (The `--debug` bit adds the decoded token.) Syntax for adding ""insert row"" for everything, ""update row"" for all in the ""data"" database and ""delete row"" just for the docs / titles table: ``` datasette create-token root --secret sec \ --all insert-row \ --database data update-row \ --table docs titles delete-row ``` The `ir` / `ur` / `dr` options would work too. To add multiple permissions use these options multiple times: ``` datasette create-token root --secret sec \ --all insert-row \ --all delete-row ``` Short versions: `-a` and `-d` and `-t`. UPDATE: I have decided to use the term `resource` in the user-facing elements of this feature instead of `table`, since that can refer to a SQL view and a canned query as well. So `--resource` and `-r`, not `-t`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1888#issuecomment-1313139657,https://api.github.com/repos/simonw/datasette/issues/1888,1313139657,IC_kwDOBm6k_c5OROvJ,9599,simonw,2022-11-14T06:04:48Z,2022-11-14T06:04:48Z,OWNER,Demo: https://latest-1-0-dev.datasette.io/-/api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447439985,API explorer should take immutability into account, https://github.com/simonw/datasette/issues/1866#issuecomment-1313128913,https://api.github.com/repos/simonw/datasette/issues/1866,1313128913,IC_kwDOBm6k_c5ORMHR,9599,simonw,2022-11-14T05:48:22Z,2022-11-14T05:48:22Z,OWNER,"I changed my mind about the `""return_rows"": true` option - I'm going to rename it to `""return"": true`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1874#issuecomment-1313127054,https://api.github.com/repos/simonw/datasette/issues/1874,1313127054,IC_kwDOBm6k_c5ORLqO,9599,simonw,2022-11-14T05:45:00Z,2022-11-14T05:45:00Z,OWNER,"Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=&method=POST ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,API to drop a table, https://github.com/simonw/datasette/issues/1871#issuecomment-1313125870,https://api.github.com/repos/simonw/datasette/issues/1871,1313125870,IC_kwDOBm6k_c5ORLXu,9599,simonw,2022-11-14T05:42:50Z,2022-11-14T05:42:50Z,OWNER,Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=%7B%22confirm%22%3A+true%7D&method=POST,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1874#issuecomment-1313125123,https://api.github.com/repos/simonw/datasette/issues/1874,1313125123,IC_kwDOBm6k_c5ORLMD,9599,simonw,2022-11-14T05:41:20Z,2022-11-14T05:42:23Z,OWNER,"I also changed the confirmation JSON returned by this endpoint to add the `database` and `table` like so: ```json { ""ok"": true, ""database"": ""data"", ""table"": ""docs"", ""row_count"": 1, ""message"": ""Pass \""confirm\"": true to confirm"" } ``` Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,API to drop a table, https://github.com/simonw/datasette/issues/1874#issuecomment-1313119558,https://api.github.com/repos/simonw/datasette/issues/1874,1313119558,IC_kwDOBm6k_c5ORJ1G,9599,simonw,2022-11-14T05:30:27Z,2022-11-14T05:30:27Z,OWNER,Found a bug: you get a 500 error if you try this against an immutable database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,API to drop a table, https://github.com/simonw/datasette/issues/1850#issuecomment-1313115059,https://api.github.com/repos/simonw/datasette/issues/1850,1313115059,IC_kwDOBm6k_c5ORIuz,9599,simonw,2022-11-14T05:21:30Z,2022-11-14T05:21:30Z,OWNER,New documentation for these features currently lives here: https://docs.datasette.io/en/1.0-dev/json_api.html#the-json-write-api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1875#issuecomment-1313114283,https://api.github.com/repos/simonw/datasette/issues/1875,1313114283,IC_kwDOBm6k_c5ORIir,9599,simonw,2022-11-14T05:20:00Z,2022-11-14T05:20:00Z,OWNER,"I started a conversation about JSON error standards on Mastodon here: https://fedi.simonwillison.net/web/@simon/109338725610487457 Quite a few people pointed to this RFC independently.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1887#issuecomment-1313113642,https://api.github.com/repos/simonw/datasette/issues/1887,1313113642,IC_kwDOBm6k_c5ORIYq,9599,simonw,2022-11-14T05:18:51Z,2022-11-14T05:18:51Z,OWNER,Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447388809,Add a confirm step to the drop table API, https://github.com/simonw/datasette/issues/1887#issuecomment-1313097713,https://api.github.com/repos/simonw/datasette/issues/1887,1313097713,IC_kwDOBm6k_c5OREfx,9599,simonw,2022-11-14T05:00:54Z,2022-11-14T05:00:54Z,OWNER,"I'm going to add a `""confirm"": true` option to the API. Without that, it returns a note about how many rows will be deleted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447388809,Add a confirm step to the drop table API, https://github.com/simonw/datasette/issues/1871#issuecomment-1313097057,https://api.github.com/repos/simonw/datasette/issues/1871,1313097057,IC_kwDOBm6k_c5OREVh,9599,simonw,2022-11-14T04:59:28Z,2022-11-14T04:59:28Z,OWNER,In playing with the API explorer just now I realized it's way too easy to accidentally drop a table using it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1313072900,https://api.github.com/repos/simonw/datasette/issues/1871,1313072900,IC_kwDOBm6k_c5OQ-cE,9599,simonw,2022-11-14T04:15:50Z,2022-11-14T04:15:50Z,OWNER,"For the example links - I'm going to have these at the bottom of the page so you don't have to scroll past them. Ideally these would take the user's permissions into account. This could make the page expensive to load, but I'm going to risk it for the moment. Something like this then: > - data > - /data/-/create - create table > - /data/table1/-/insert - insert into table1 > - /data/table1/-/drop - drop table1 I won't bother with per-row demo links (for update and delete) because there could be thousands of them for each table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1313062699,https://api.github.com/repos/simonw/datasette/issues/1871,1313062699,IC_kwDOBm6k_c5OQ78r,9599,simonw,2022-11-14T04:03:29Z,2022-11-14T04:12:41Z,OWNER,"Two things left before I close this issue: - [x] I want to preserve the state of the forms in the URL - probably after a `#` - [ ] Instead of hard-coding the current examples, I want to provide a list of links which populate the forms","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1886#issuecomment-1313052863,https://api.github.com/repos/simonw/datasette/issues/1886,1313052863,IC_kwDOBm6k_c5OQ5i_,9599,simonw,2022-11-14T03:40:50Z,2022-11-14T03:40:50Z,OWNER,"Tim Sherratt on Twitter: https://twitter.com/wragge/status/1591930345469153282 > Where do I start? The [#GLAMWorkbench](https://twitter.com/hashtag/GLAMWorkbench?src=hashtag_click) now includes a number of examples where GLAM data is harvested, processed, and then made available for exploration via Datasette. > > https://glam-workbench.net/ > > For example the GLAM Name Index Search brings together 10+ million entries from 240 indexes and provides an aggregated search using the Datasette search-all plugin: > > https://glam-workbench.net/name-search/ > > Most recently I converted PDFs of the Tasmanian Postal Directories to a big Datasette instance: https://updates.timsherratt.org/2022/09/15/from-pdfs-to.html the process is documented and reusable.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1871#issuecomment-1312822353,https://api.github.com/repos/simonw/datasette/issues/1871,1312822353,IC_kwDOBm6k_c5OQBRR,9599,simonw,2022-11-13T21:07:40Z,2022-11-13T21:07:40Z,OWNER,I'm going to need extra code to toggle POST closed when GET opens and vice-versa.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1312821031,https://api.github.com/repos/simonw/datasette/issues/1871,1312821031,IC_kwDOBm6k_c5OQA8n,9599,simonw,2022-11-13T21:02:06Z,2022-11-13T21:03:11Z,OWNER,"Actually no, I'm going to add a class of `details-menu` to the other details elements that SHOULD be closed. That way custom templates using `
    ` won't close in a surprising way.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1312816451,https://api.github.com/repos/simonw/datasette/issues/1871,1312816451,IC_kwDOBm6k_c5OP_1D,9599,simonw,2022-11-13T20:39:26Z,2022-11-13T20:39:34Z,OWNER,I'm going to add a special `no-auto-close` class to these and teach that code not to close them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1312816292,https://api.github.com/repos/simonw/datasette/issues/1871,1312816292,IC_kwDOBm6k_c5OP_yk,9599,simonw,2022-11-13T20:38:42Z,2022-11-13T20:38:42Z,OWNER,"The current API explorer uses details/summary elements for the GET and POST dialogs. I only want one of these to be open at a time, to reflect that you can make either a GET or a POST. I just noticed that clicking anywhere else on the page closes both elements, which isn't what I want to happen. Turns out that's because of this code I added as part of Datasette's menu implementation! https://github.com/simonw/datasette/blob/9f54f00a50a4d950cfd69a0ff3526ae82c858826/datasette/templates/_close_open_menus.html#L2-L15","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1882#issuecomment-1312582512,https://api.github.com/repos/simonw/datasette/issues/1882,1312582512,IC_kwDOBm6k_c5OPGtw,9599,simonw,2022-11-12T22:11:18Z,2022-11-12T22:11:18Z,OWNER,"I like this: ```json { ""ok"": true, ""database"": ""data"", ""table"": ""agai2n"", ""table_url"": ""http://127.0.0.1:8001/data/agai2n"", ""schema"": ""CREATE TABLE [agai2n] (\n [hello] INTEGER\n)"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312581121,https://api.github.com/repos/simonw/datasette/issues/1882,1312581121,IC_kwDOBm6k_c5OPGYB,9599,simonw,2022-11-12T22:01:32Z,2022-11-12T22:01:32Z,OWNER,I'm going to change it to `table` in the output AND the input.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312581008,https://api.github.com/repos/simonw/datasette/issues/1882,1312581008,IC_kwDOBm6k_c5OPGWQ,9599,simonw,2022-11-12T22:00:52Z,2022-11-12T22:00:52Z,OWNER,"Tried out my prototype in the API explorer: The `""name""` on the output is bothering me a bit - should it be `table_name` or `table` instead? Problem is I really like `name` for the input, so should it be consistent to have the same name on the output here, or should I aim for consistency with other endpoints that might return `table` rather than the ambiguous `name` elsewhere?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312580348,https://api.github.com/repos/simonw/datasette/issues/1882,1312580348,IC_kwDOBm6k_c5OPGL8,9599,simonw,2022-11-12T21:55:54Z,2022-11-12T21:56:45Z,OWNER,"What should this API return? I think the name of the table (`name`), the URL to that table (`table_url` - for consistency with how faceting API works already) and the schema of the table (`schema`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312575048,https://api.github.com/repos/simonw/datasette/issues/1882,1312575048,IC_kwDOBm6k_c5OPE5I,9599,simonw,2022-11-12T21:22:58Z,2022-11-12T21:22:58Z,OWNER,Need to validate the table name. SQLite supports almost any table name - but they can't contain a newline character and cannot start with `sqlite_` - according to https://stackoverflow.com/questions/3694276/what-are-valid-table-names-in-sqlite/43049720#43049720,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312556044,https://api.github.com/repos/simonw/datasette/issues/1882,1312556044,IC_kwDOBm6k_c5OPAQM,9599,simonw,2022-11-12T19:29:11Z,2022-11-12T19:29:11Z,OWNER,"Thought of an edge-case: with `sqlite-utils` one feature I really like is that I can pipe data into it without caring if the table already exists or not: cat data.json | sqlite-utils insert my.db mytable - How could this new API support that? I thought about adding a `""create"": true` option to `/db/table/-/insert` which creates the table if it doesn't already exist, but if I do that I'll need to start adding other options to that endpoint - to set the primary key, add foreign keys and suchlike - which would be ignored except for the cases where the table was being created from scratch. This doesn't feel right to me - I want to keep those options here, on `/db/-/create`. One idea I had was to implement it such that you can call `/db/-/create` multiple times for the same table, but only if you are using the `""rows""` option. If so, and if the rows can be safely inserted, it would let you do that. But instead, I'm going to outsource this to the CLI tool I plan to write that feeds data into this API. I'm already planning to use that tool for CSV inserts (so the API doesn't need to accept CSV directly). I think it's a good place for other usability enhancements like ""insert this, creating the table if it does not exist"" as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1883#issuecomment-1311314981,https://api.github.com/repos/simonw/datasette/issues/1883,1311314981,IC_kwDOBm6k_c5OKRQl,9599,simonw,2022-11-11T07:15:48Z,2022-11-11T07:15:48Z,OWNER,I released that fix in Datasette 0.63.1: https://docs.datasette.io/en/stable/changelog.html#v0-63-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311299535,https://api.github.com/repos/simonw/datasette/issues/1883,1311299535,IC_kwDOBm6k_c5OKNfP,9599,simonw,2022-11-11T06:54:58Z,2022-11-11T06:54:58Z,OWNER,"This time deployed with: ``` cd demos/apache-proxy fly deploy --build-arg DATASETTE_REF=8d9a957c6329d26cc1e417b5d6911640d74765eb ``` To ensure the exact commit with the fix. And that fixed it! ``` % curl -i 'https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid' HTTP/2 302 date: Fri, 11 Nov 2022 06:54:45 GMT server: Fly/b1863e2e7 (2022-11-09) location: /prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1 link: ; rel=preload content-type: text/plain x-proxied-by: Apache2 Debian via: 2 fly.io fly-request-id: 01GHJQGBSXBR7E53TY0EKMQ9PA-sjc ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311292463,https://api.github.com/repos/simonw/datasette/issues/1883,1311292463,IC_kwDOBm6k_c5OKLwv,9599,simonw,2022-11-11T06:44:24Z,2022-11-11T06:44:24Z,OWNER,"Modifying that test to the following does indeed cause a failure: ```python def test_base_url_affects_filter_redirects(app_client_base_url_prefix): response = app_client_base_url_prefix.get( ""/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid"" ) assert response.status == 302 assert ( response.headers[""location""] == ""/prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1"" ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311291632,https://api.github.com/repos/simonw/datasette/issues/1883,1311291632,IC_kwDOBm6k_c5OKLjw,9599,simonw,2022-11-11T06:43:00Z,2022-11-11T06:43:00Z,OWNER,"https://datasette-apache-proxy-demo.datasette.io/prefix/-/asgi-scope is useful: It confirms that `/prefix/` is nowhere to be seen in the incoming request data: ``` 'path': '/-/asgi-scope', 'query_string': b'', 'raw_path': b'/-/asgi-scope', ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311290115,https://api.github.com/repos/simonw/datasette/issues/1883,1311290115,IC_kwDOBm6k_c5OKLMD,9599,simonw,2022-11-11T06:40:14Z,2022-11-11T06:41:56Z,OWNER,"I modified that config file to have this line instead: ``` ProxyPass /prefix/ http://127.0.0.1:8001/ nocanon ``` And then deployed it by running: flyctl deploy --build-arg DATASETTE_REF=main This does NOT seem to have fixed the bug: ``` ~ % curl -i 'https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid' HTTP/2 302 date: Fri, 11 Nov 2022 06:40:01 GMT server: Fly/b1863e2e7 (2022-11-09) location: /fixtures/binary_data?_sort=rowid&rowid__exact=1 link: ; rel=preload content-type: text/plain x-proxied-by: Apache2 Debian via: 2 fly.io fly-request-id: 01GHJPNCF51CJ626EWZEHK2CH9-sjc ``` https://datasette-apache-proxy-demo.datasette.io/prefix/-/versions seems to confirm that this is the latest deployed version (0.63), so it looks like the deploy worked.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311286593,https://api.github.com/repos/simonw/datasette/issues/1883,1311286593,IC_kwDOBm6k_c5OKKVB,9599,simonw,2022-11-11T06:34:09Z,2022-11-11T06:34:09Z,OWNER,"https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass includes this note: > Normally, mod_proxy will canonicalise ProxyPassed URLs. But this may be incompatible with some backends, particularly those that make use of *PATH_INFO*. The optional *nocanon* keyword suppresses this and passes the URL path ""raw"" to the backend.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311284537,https://api.github.com/repos/simonw/datasette/issues/1883,1311284537,IC_kwDOBm6k_c5OKJ05,9599,simonw,2022-11-11T06:30:38Z,2022-11-11T06:30:38Z,OWNER,"Is there a chance that it's Apache that's messing with that `location:` header here, not Datasette? https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/demos/apache-proxy/000-default.conf#L7-L13","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311283301,https://api.github.com/repos/simonw/datasette/issues/1883,1311283301,IC_kwDOBm6k_c5OKJhl,9599,simonw,2022-11-11T06:28:38Z,2022-11-11T06:29:33Z,OWNER,"`path_with_added_args(request, redirect_params)` should be preserving the current path from the request. https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/datasette/utils/__init__.py#L273-L286","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311282970,https://api.github.com/repos/simonw/datasette/issues/1883,1311282970,IC_kwDOBm6k_c5OKJca,9599,simonw,2022-11-11T06:28:05Z,2022-11-11T06:28:05Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/datasette/views/table.py#L227-L249,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311280709,https://api.github.com/repos/simonw/datasette/issues/1883,1311280709,IC_kwDOBm6k_c5OKI5F,9599,simonw,2022-11-11T06:25:27Z,2022-11-11T06:25:27Z,OWNER,"I tried adding this test but it passed! I expected it to fail: ```python def test_base_url_affects_filter_redirects(app_client_base_url_prefix): response = app_client_base_url_prefix.get( ""/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid"" ) assert response.status == 302 assert ( response.headers[""location""] == ""/prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1"" ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311278678,https://api.github.com/repos/simonw/datasette/issues/1883,1311278678,IC_kwDOBm6k_c5OKIZW,9599,simonw,2022-11-11T06:22:37Z,2022-11-11T06:22:37Z,OWNER,"If you view source on that page the HTML looks correct: ```html
    ``` (I just added a test that confirms this too.) But... it looks like the bug is in the redirection code. https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid returns the following: location: /fixtures/binary_data?_sort=rowid&rowid__exact=1 Which is incorrect.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311273461,https://api.github.com/repos/simonw/datasette/issues/1883,1311273461,IC_kwDOBm6k_c5OKHH1,9599,simonw,2022-11-11T06:16:08Z,2022-11-11T06:16:08Z,OWNER,"Great catch, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1880#issuecomment-1311273063,https://api.github.com/repos/simonw/datasette/issues/1880,1311273063,IC_kwDOBm6k_c5OKHBn,9599,simonw,2022-11-11T06:15:28Z,2022-11-11T06:15:28Z,OWNER,"The `_internal` database is intended to help Datasette handle much larger attached databases. Right now Datasette attempts to show every database on the https://latest.datasette.io/ index page and every table on the https://latest.datasette.io/fixtures database index page - but these are not paginated. If you had a database containing 1,000 tables the database index page would get pretty slow. So I want to be able to paginate (and search) those. But to paginate them it's useful to have them in a database table itself, since then I can paginate using SQL. My plan for `_internal` is to use it to implement those advanced browsing features. I've not completed this work yet though. See this issue for more details on that: - #417","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1433576351,Datasette with many and large databases > Memory use, https://github.com/simonw/datasette/issues/1880#issuecomment-1311271298,https://api.github.com/repos/simonw/datasette/issues/1880,1311271298,IC_kwDOBm6k_c5OKGmC,9599,simonw,2022-11-11T06:12:29Z,2022-11-11T06:12:29Z,OWNER,"I think you may have misunderstood this feature. This is talking about the `_internal` in-memory database, which maintains a set of tables that list the databases and tables that are attached to Datasette. They're not a copy of the data itself - just a list of table names, column names and database names. You can see what that database looks like by signing in as root - running `datasette --root` and clicking the link. Or you can see an example here: - Click the button on https://latest.datasette.io/login-as-root - Now visit https://latest.datasette.io/_internal For the example instance that looks like this: The two most interesting tables in there are these ones: As you can see, it's just the table schema itself and the columns that make up the tables. Even if you have hundreds of databases connected each with hundreds of tables this should still only add up to a few MB of RAM.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1433576351,Datasette with many and large databases > Memory use, https://github.com/simonw/datasette/issues/1884#issuecomment-1311269045,https://api.github.com/repos/simonw/datasette/issues/1884,1311269045,IC_kwDOBm6k_c5OKGC1,9599,simonw,2022-11-11T06:08:28Z,2022-11-11T06:08:28Z,OWNER,Does that work if you add `--load-extension spatialite`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/datasette/issues/1882#issuecomment-1302818784,https://api.github.com/repos/simonw/datasette/issues/1882,1302818784,IC_kwDOBm6k_c5Np2_g,9599,simonw,2022-11-04T00:25:18Z,2022-11-04T16:12:39Z,OWNER,"On that basis I think the core API design should change to this: ``` POST /db/-/create Authorization: Bearer xxx Content-Type: application/json { ""name"": ""my new table"", ""columns"": [ { ""name"": ""id"", ""type"": ""integer"" }, { ""name"": ""title"", ""type"": ""text"" } ] ""pk"": ""id"" } ``` This leaves room for a `""rows"": []` key at the root too. Having that as a child of `""table""` felt unintuitive to me, and I didn't like the way this looked either: ```json { ""table"": { ""name"": ""my_new_table"" }, ""rows"": [ {""id"": 1, ""title"": ""Title""} ] } ``` Weird to have the table `name` nested inside `table` when `rows` wasn't.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1302818153,https://api.github.com/repos/simonw/datasette/issues/1882,1302818153,IC_kwDOBm6k_c5Np21p,9599,simonw,2022-11-04T00:23:58Z,2022-11-04T00:23:58Z,OWNER,"I made a decision here that this endpoint should also accept an optional `""rows"": [...]` list which is used to automatically create the table using a schema derived from those example rows (which then get inserted): - https://github.com/simonw/datasette/issues/1862#issuecomment-1302817807","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1862#issuecomment-1302817807,https://api.github.com/repos/simonw/datasette/issues/1862,1302817807,IC_kwDOBm6k_c5Np2wP,9599,simonw,2022-11-04T00:23:13Z,2022-11-04T00:23:13Z,OWNER,"I don't like this on `/db/table/-/insert` - I think it makes more sense to optionally pass a `""rows""` key to the `/db/-/create` endpoint instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425011030,"Create a new table from one or more records, `sqlite-utils` style", https://github.com/simonw/datasette/issues/1862#issuecomment-1302817500,https://api.github.com/repos/simonw/datasette/issues/1862,1302817500,IC_kwDOBm6k_c5Np2rc,9599,simonw,2022-11-04T00:22:31Z,2022-11-04T00:22:31Z,OWNER,"Maybe this is a feature added to the existing `/db/table/-/insert` endpoint? Bit weird that you can call that endpoint for a table that doesn't exist yet, but it fits the `sqlite-utils` way of creating tables which I've found very pleasant over the past few years. So perhaps the API looks like this: ``` POST //
    /-/insert Content-Type: application/json Authorization: Bearer dstok_ { ""create_table"": true, ""rows"": [ { ""column1"": ""value1"", ""column2"": ""value2"" }, { ""column1"": ""value3"", ""column2"": ""value4"" } ] } ``` The `create_table` option will cause the table to be created if it doesn't already exist. That means I probably also need a `""pk"": ""...""` column for setting a primary key if the table is being created ... and maybe other options that I invent for this other feature too? - #1882","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425011030,"Create a new table from one or more records, `sqlite-utils` style", https://github.com/simonw/datasette/issues/1855#issuecomment-1302815929,https://api.github.com/repos/simonw/datasette/issues/1855,1302815929,IC_kwDOBm6k_c5Np2S5,9599,simonw,2022-11-04T00:19:10Z,2022-12-03T07:05:51Z,OWNER,Added the tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1871#issuecomment-1302815105,https://api.github.com/repos/simonw/datasette/issues/1871,1302815105,IC_kwDOBm6k_c5Np2GB,9599,simonw,2022-11-04T00:17:23Z,2022-11-04T00:17:23Z,OWNER,"I'll probably enhance it a bit more though, I want to provide a UI that lists all the tables you can explore and lets you click to pre-fill the forms with them. Though at that point what should I do about the other endpoints? Probably list those too. Gets a bit complex, especially with the row-level update and delete endpoints.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1302814693,https://api.github.com/repos/simonw/datasette/issues/1871,1302814693,IC_kwDOBm6k_c5Np1_l,9599,simonw,2022-11-04T00:16:36Z,2022-11-04T00:16:36Z,OWNER,"I can close this issue once I fix it so it no longer hard-codes a potentially invalid example endpoint: https://github.com/simonw/datasette/blob/bcc781f4c50a8870e3389c4e60acb625c34b0317/datasette/templates/api_explorer.html#L24-L26 https://github.com/simonw/datasette/blob/bcc781f4c50a8870e3389c4e60acb625c34b0317/datasette/templates/api_explorer.html#L34-L35","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1881#issuecomment-1302813449,https://api.github.com/repos/simonw/datasette/issues/1881,1302813449,IC_kwDOBm6k_c5Np1sJ,9599,simonw,2022-11-04T00:14:07Z,2022-11-04T00:14:07Z,OWNER,"Tool is now live here: https://latest-1-0-dev.datasette.io/-/permissions Needs root perms, so access this first: https://latest-1-0-dev.datasette.io/login-as-root","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1302812918,https://api.github.com/repos/simonw/datasette/issues/1881,1302812918,IC_kwDOBm6k_c5Np1j2,9599,simonw,2022-11-04T00:13:05Z,2022-11-04T00:13:05Z,OWNER,Has tests now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1863#issuecomment-1302790013,https://api.github.com/repos/simonw/datasette/issues/1863,1302790013,IC_kwDOBm6k_c5Npv99,9599,simonw,2022-11-03T23:32:30Z,2022-11-03T23:32:30Z,OWNER,"I'm not going to allow updates to primary keys. If you need to do that, you can instead delete the record and then insert a new one with the new primary keys you wanted - or maybe use a custom SQL query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1302785086,https://api.github.com/repos/simonw/datasette/issues/1863,1302785086,IC_kwDOBm6k_c5Npuw-,9599,simonw,2022-11-03T23:24:33Z,2022-11-03T23:24:56Z,OWNER,"Thinking more about validation: I'm considering if this should validate that columns which are defined as SQLite foreign keys are being updated to values that exist in those other tables. I like the sound of this. It seems like a sensible default behaviour for Datasette. And it fits with the fact that Datasette treats foreign keys specially elsewhere in the interface.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1302760549,https://api.github.com/repos/simonw/datasette/issues/1863,1302760549,IC_kwDOBm6k_c5Npoxl,9599,simonw,2022-11-03T22:43:04Z,2022-11-03T23:21:31Z,OWNER,"The `id=(int, ...)` thing is weird, but is apparently Pydantic syntax for a required field? https://cs.github.com/starlite-api/starlite/blob/28ddc847c4cb072f0d5d21a9ecd5259711f12ec9/docs/usage/11-data-transfer-objects.md#L161 confirms: > 1. For required fields use a tuple of type + ellipsis, for example `(str, ...)`. > 2. For optional fields use a tuple of type + `None`, for example `(str, None)` > 3. To set a default value use a tuple of type + default value, for example `(str, ""Hello World"")`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1302760382,https://api.github.com/repos/simonw/datasette/issues/1863,1302760382,IC_kwDOBm6k_c5Npou-,9599,simonw,2022-11-03T22:42:47Z,2022-11-03T22:42:47Z,OWNER,"```python print(create_model('document', id=(int, ...), title=(str, None)).schema_json(indent=2)) ``` ```json { ""title"": ""document"", ""type"": ""object"", ""properties"": { ""id"": { ""title"": ""Id"", ""type"": ""integer"" }, ""title"": { ""title"": ""Title"", ""type"": ""string"" } }, ""required"": [ ""id"" ] } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1302759174,https://api.github.com/repos/simonw/datasette/issues/1863,1302759174,IC_kwDOBm6k_c5NpocG,9599,simonw,2022-11-03T22:40:47Z,2022-11-03T22:40:47Z,OWNER,"I'm considering Pydantic for this, see: - https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350 In particular the `create_model()` method: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation This would give me good validation. It would also, weirdly, give me the ability to output JSON schema. Maybe I could have this as the JSON schema for a row? `/db/table/-/json-schema`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1882#issuecomment-1302721916,https://api.github.com/repos/simonw/datasette/issues/1882,1302721916,IC_kwDOBm6k_c5NpfV8,9599,simonw,2022-11-03T21:58:50Z,2022-11-03T21:59:17Z,OWNER,"Mocked up a quick HTML+JavaScript form for creating that JSON structure using some iteration against Copilot prompts: ```html 
    /* JSON format:
{
  ""table"": {
      ""name"": ""my new table"",
      ""columns"": [
          {
              ""name"": ""id"",
              ""type"": ""integer""
          },
          {
              ""name"": ""title"",
              ""type"": ""text""
          }
      ]
     ""pk"": ""id""
  }
}

HTML form with Javascript for creating this JSON:
*/


    Current columns:

      ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350,https://api.github.com/repos/simonw/datasette/issues/1882,1302716350,IC_kwDOBm6k_c5Npd--,9599,simonw,2022-11-03T21:51:14Z,2022-11-03T22:35:54Z,OWNER,"Validating this JSON object is getting a tiny bit complex. I'm tempted to adopt https://pydantic-docs.helpmanual.io/ at this point. The `create_model` example on https://stackoverflow.com/questions/66168517/generate-dynamic-model-using-pydantic/66168682#66168682 is particularly relevant, especially when I work on this issue: - #1863 ```python from pydantic import create_model d = {""strategy"": {""name"": ""test_strat2"", ""periods"": 10}} Strategy = create_model(""Strategy"", **d[""strategy""]) print(Strategy.schema_json(indent=2)) ``` `create_model()`: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1302715662,https://api.github.com/repos/simonw/datasette/issues/1882,1302715662,IC_kwDOBm6k_c5Npd0O,9599,simonw,2022-11-03T21:50:27Z,2022-11-03T21:50:27Z,OWNER,"API design for this: ``` POST /db/-/create Authorization: Bearer xxx Content-Type: application/json { ""table"": { ""name"": ""my new table"", ""columns"": [ { ""name"": ""id"", ""type"": ""integer"" }, { ""name"": ""title"", ""type"": ""text"" } ] ""pk"": ""id"" } } ``` Supported column types are: - `integer` - `text` - `float` (even though SQLite calls it a ""real"") - `blob` This matches my design for `sqlite-utils`: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1843#issuecomment-1302679026,https://api.github.com/repos/simonw/datasette/issues/1843,1302679026,IC_kwDOBm6k_c5NpU3y,9599,simonw,2022-11-03T21:22:42Z,2022-11-03T21:22:42Z,OWNER,Docs for the new `db.close()` method: https://docs.datasette.io/en/latest/internals.html#db-close,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1302678384,https://api.github.com/repos/simonw/datasette/issues/1843,1302678384,IC_kwDOBm6k_c5NpUtw,9599,simonw,2022-11-03T21:21:59Z,2022-11-03T21:21:59Z,OWNER,"I added extra debug info to `/-/threads` to see this for myself: ```diff diff --git a/datasette/app.py b/datasette/app.py index 02bd38f1..16579e28 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -969,6 +969,13 @@ class Datasette: ""threads"": [ {""name"": t.name, ""ident"": t.ident, ""daemon"": t.daemon} for t in threads ], + ""file_connections"": { + db.name: [ + [dict(r) for r in conn.execute(""pragma database_list"").fetchall()] + for conn in db._all_file_connections + ] + for db in self.databases.values() + }, } # Only available in Python 3.7+ if hasattr(asyncio, ""all_tasks""): ``` Output after hitting refresh on a few `/fixtures` tables to ensure more threads started: ``` ""file_connections"": { ""_internal"": [], ""fixtures"": [ [ { ""seq"": 0, ""name"": ""main"", ""file"": ""/Users/simon/Dropbox/Development/datasette/fixtures.db"" } ], [ { ""seq"": 0, ""name"": ""main"", ""file"": ""/Users/simon/Dropbox/Development/datasette/fixtures.db"" } ], [ { ""seq"": 0, ""name"": ""main"", ""file"": ""/Users/simon/Dropbox/Development/datasette/fixtures.db"" } ] ] }, ``` I decided not to ship this feature though as it leaks the names of internal database files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1302634332,https://api.github.com/repos/simonw/datasette/issues/1843,1302634332,IC_kwDOBm6k_c5NpJ9c,9599,simonw,2022-11-03T20:34:56Z,2022-11-03T20:34:56Z,OWNER,"Confirmed that calling `conn.close()` on each SQLite file-based connection is the way to fix this problem. I'm adding a `db.close()` method (sync, not async - I tried async first but it was really hard to cause every thread in the pool to close its threadlocal database connection) which loops through all known open file-based connections and closes them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1302574330,https://api.github.com/repos/simonw/datasette/issues/1843,1302574330,IC_kwDOBm6k_c5No7T6,9599,simonw,2022-11-03T19:30:22Z,2022-11-03T19:30:22Z,OWNER,"This is affecting me a lot at the moment, on my laptop (runs fine in CI). Here's a change to `conftest.py` which highlights the problem - it cause a failure the moment there are more than 5 open files according to `psutil`: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index f4638a14..21d433c1 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,6 +1,7 @@ import httpx import os import pathlib +import psutil import pytest import re import subprocess @@ -192,3 +193,8 @@ def ds_unix_domain_socket_server(tmp_path_factory): yield ds_proc, uds # Shut it down at the end of the pytest session ds_proc.terminate() + + +def pytest_runtest_teardown(item: pytest.Item) -> None: + open_files = psutil.Process().open_files() + assert len(open_files) < 5 ``` The first error I get from this with `pytest --pdb -x` is here: ``` tests/test_api.py ............E >>>>> traceback >>>>> item = def pytest_runtest_teardown(item: pytest.Item) -> None: open_files = psutil.Process().open_files() > assert len(open_files) < 5 E AssertionError: assert 5 < 5 E + where 5 = len([popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfglrt4p2/fixtures.db', fd=14), popenfile(... fd=19), popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmphdi5b250/fixtures.dot.db', fd=20)]) /Users/simon/Dropbox/Development/datasette/tests/conftest.py:200: AssertionError >>>>> entering PDB >>>>> >>>>> PDB post_mortem (IO-capturing turned off) >>>>> > /Users/simon/Dropbox/Development/datasette/tests/conftest.py(200)pytest_runtest_teardown() -> assert len(open_files) < 5 ``` That's this test: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/test_api.py#L656-L673 Which uses this fixture: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L228-L231 Which calls this function: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L105-L122 So now I'm suspicious that, even though the fixture is meant to be session scoped, the way I'm using `with tempfile.TemporaryDirectory() as tmpdir:` is causing a whole load of files to be created and held open which are not later closed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1855#issuecomment-1301646670,https://api.github.com/repos/simonw/datasette/issues/1855,1301646670,IC_kwDOBm6k_c5NlY1O,9599,simonw,2022-11-03T05:11:26Z,2022-11-03T05:11:26Z,OWNER,That still needs comprehensive tests before I land it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1855#issuecomment-1301646493,https://api.github.com/repos/simonw/datasette/issues/1855,1301646493,IC_kwDOBm6k_c5NlYyd,9599,simonw,2022-11-03T05:11:06Z,2022-11-03T05:11:06Z,OWNER,"Built a prototype of the above: ```diff diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 32b0c758..f68aa38f 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -6,8 +6,8 @@ import json import time -@hookimpl(tryfirst=True) -def permission_allowed(datasette, actor, action, resource): +@hookimpl(tryfirst=True, specname=""permission_allowed"") +def permission_allowed_default(datasette, actor, action, resource): async def inner(): if action in ( ""permissions-debug"", @@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource): return inner +@hookimpl(specname=""permission_allowed"") +def permission_allowed_actor_restrictions(actor, action, resource): + if actor is None: + return None + _r = actor.get(""_r"") + if not _r: + # No restrictions, so we have no opinion + return None + action_initials = """".join([word[0] for word in action.split(""-"")]) + # If _r is defined then we use those to further restrict the actor + # Crucially, we only use this to say NO (return False) - we never + # use it to return YES (True) because that might over-ride other + # restrictions placed on this actor + all_allowed = _r.get(""a"") + if all_allowed is not None: + assert isinstance(all_allowed, list) + if action_initials in all_allowed: + return None + # How about for the current database? + if action in (""view-database"", ""view-database-download"", ""execute-sql""): + database_allowed = _r.get(""d"", {}).get(resource) + if database_allowed is not None: + assert isinstance(database_allowed, list) + if action_initials in database_allowed: + return None + # Or the current table? That's any time the resource is (database, table) + if not isinstance(resource, str) and len(resource) == 2: + database, table = resource + table_allowed = _r.get(""t"", {}).get(database, {}).get(table) + # TODO: What should this do for canned queries? + if table_allowed is not None: + assert isinstance(table_allowed, list) + if action_initials in table_allowed: + return None + # This action is not specifically allowed, so reject it + return False + + @hookimpl def actor_from_request(datasette, request): prefix = ""dstok_"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1881#issuecomment-1301645921,https://api.github.com/repos/simonw/datasette/issues/1881,1301645921,IC_kwDOBm6k_c5NlYph,9599,simonw,2022-11-03T05:10:05Z,2022-12-09T01:38:21Z,OWNER,"I'd love to come up with a good short name for the second part of the resource two-tuple, the thing which is usually the name of a table but could also be the name of a SQL view or the name of a canned query. Idea 8th December: why not call it resource? A resource could be a thing that lives inside a database.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301639741,https://api.github.com/repos/simonw/datasette/issues/1881,1301639741,IC_kwDOBm6k_c5NlXI9,9599,simonw,2022-11-03T04:58:21Z,2022-11-03T04:58:21Z,OWNER,"The whole `database_name` or `(database_name, table_name)` tuple for resource is a bit of a code smell. Maybe this is a chance to tidy that up too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301639370,https://api.github.com/repos/simonw/datasette/issues/1881,1301639370,IC_kwDOBm6k_c5NlXDK,9599,simonw,2022-11-03T04:57:21Z,2022-11-03T04:57:21Z,OWNER,"The plugin hook would be called `register_permissions()`, for consistency with `register_routes()` and `register_commands()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301638918,https://api.github.com/repos/simonw/datasette/issues/1881,1301638918,IC_kwDOBm6k_c5NlW8G,9599,simonw,2022-11-03T04:56:06Z,2022-11-03T04:56:06Z,OWNER,"I've also introduced a new concept of a permission abbreviation, which like the permission name needs to be globally unique. That's a problem for plugins - they might just be able to guarantee that their permission long-form name is unique among other plugins (through sensible naming conventions) but the thing where they declare a initial-letters-only abbreviation is far more risky. I think abbreviations are optional - they are provided for core permissions but plugins are advised not to use them. Also Datasette could check that the installed plugins do not provide conflicting permissions on startup and refuse to start if they do.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301638156,https://api.github.com/repos/simonw/datasette/issues/1881,1301638156,IC_kwDOBm6k_c5NlWwM,9599,simonw,2022-11-03T04:54:00Z,2022-11-03T04:54:00Z,OWNER,"If I have the permissions defined like this: ```python PERMISSIONS = ( Permission(""view-instance"", ""vi"", False, False, True), Permission(""view-database"", ""vd"", True, False, True), Permission(""view-database-download"", ""vdd"", True, False, True), Permission(""view-table"", ""vt"", True, True, True), Permission(""view-query"", ""vq"", True, True, True), Permission(""insert-row"", ""ir"", True, True, False), Permission(""delete-row"", ""dr"", True, True, False), Permission(""drop-table"", ""dt"", True, True, False), Permission(""execute-sql"", ""es"", True, False, True), Permission(""permissions-debug"", ""pd"", False, False, False), Permission(""debug-menu"", ""dm"", False, False, False), ) ``` Instead of just calling them by their undeclared names in places like this: ```python await self.ds.permission_allowed( request.actor, ""execute-sql"", database, default=True ) ``` On the one hand I can ditch that confusing `default=True` option - whether a permission is on by default becomes a characteristic of that `Permission()` itself, which feels much neater. On the other hand though, plugins that introduce their own permissions - like https://datasette.io/plugins/datasette-edit-schema - will need a way to register those permissions with Datasette core. Probably another plugin hook.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301635906,https://api.github.com/repos/simonw/datasette/issues/1881,1301635906,IC_kwDOBm6k_c5NlWNC,9599,simonw,2022-11-03T04:48:09Z,2022-11-03T04:48:09Z,OWNER,"I built this prototype on the http://127.0.0.1:8001/-/allow-debug page, which is open to anyone to visit. But... I just realized that using this tool can leak information - you can use it to guess the names of invisible databases and tables and run theoretical permission checks against them. Using the tool also pollutes the list of permission checks that show up on the root-anlo `/-/permissions` page. So.... I'm going to restrict the usage of this tool to users with access to `/-/permissions` and put it on that page instead. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301635340,https://api.github.com/repos/simonw/datasette/issues/1881,1301635340,IC_kwDOBm6k_c5NlWEM,9599,simonw,2022-11-03T04:46:41Z,2022-11-03T04:46:41Z,OWNER,"Built this prototype: ![prototype](https://user-images.githubusercontent.com/9599/199649219-f146e43b-bfb5-45e6-9777-956f21a79887.gif) In building it I realized I needed to know which permissions took a table, a database, both or neither. So I had to bake that into the code. Here's the prototype so far (which includes a prototype of the logic for the `_r` field on actor, see #1855): ```diff diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 32b0c758..f68aa38f 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -6,8 +6,8 @@ import json import time -@hookimpl(tryfirst=True) -def permission_allowed(datasette, actor, action, resource): +@hookimpl(tryfirst=True, specname=""permission_allowed"") +def permission_allowed_default(datasette, actor, action, resource): async def inner(): if action in ( ""permissions-debug"", @@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource): return inner +@hookimpl(specname=""permission_allowed"") +def permission_allowed_actor_restrictions(actor, action, resource): + if actor is None: + return None + _r = actor.get(""_r"") + if not _r: + # No restrictions, so we have no opinion + return None + action_initials = """".join([word[0] for word in action.split(""-"")]) + # If _r is defined then we use those to further restrict the actor + # Crucially, we only use this to say NO (return False) - we never + # use it to return YES (True) because that might over-ride other + # restrictions placed on this actor + all_allowed = _r.get(""a"") + if all_allowed is not None: + assert isinstance(all_allowed, list) + if action_initials in all_allowed: + return None + # How about for the current database? + if action in (""view-database"", ""view-database-download"", ""execute-sql""): + database_allowed = _r.get(""d"", {}).get(resource) + if database_allowed is not None: + assert isinstance(database_allowed, list) + if action_initials in database_allowed: + return None + # Or the current table? That's any time the resource is (database, table) + if not isinstance(resource, str) and len(resource) == 2: + database, table = resource + table_allowed = _r.get(""t"", {}).get(database, {}).get(table) + # TODO: What should this do for canned queries? + if table_allowed is not None: + assert isinstance(table_allowed, list) + if action_initials in table_allowed: + return None + # This action is not specifically allowed, so reject it + return False + + @hookimpl def actor_from_request(datasette, request): prefix = ""dstok_"" diff --git a/datasette/templates/allow_debug.html b/datasette/templates/allow_debug.html index 0f1b30f0..ae43f0f5 100644 --- a/datasette/templates/allow_debug.html +++ b/datasette/templates/allow_debug.html @@ -35,7 +35,7 @@ p.message-warning {

      Use this tool to try out different actor and allow combinations. See Defining permissions with ""allow"" blocks for documentation.

      -
      +

      @@ -55,4 +55,82 @@ p.message-warning { {% if result == ""False"" %}

      Result: deny

      {% endif %} +

      Test permission check

      + +

      This tool lets you simulate an actor and a permission check for that actor.

      + + + +
      +

      + +
      +
      +

      +

      + +

      +

      +
      +
      + +
      + + + + + {% endblock %} diff --git a/datasette/views/special.py b/datasette/views/special.py index 9922a621..d46fc280 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -1,6 +1,8 @@ import json +from datasette.permissions import PERMISSIONS from datasette.utils.asgi import Response, Forbidden from datasette.utils import actor_matches_allow, add_cors_headers +from datasette.permissions import PERMISSIONS from .base import BaseView import secrets import time @@ -138,9 +140,34 @@ class AllowDebugView(BaseView): ""error"": ""\n\n"".join(errors) if errors else """", ""actor_input"": actor_input, ""allow_input"": allow_input, + ""permissions"": PERMISSIONS, }, ) + async def post(self, request): + vars = await request.post_vars() + actor = json.loads(vars[""actor""]) + permission = vars[""permission""] + resource_1 = vars[""resource_1""] + resource_2 = vars[""resource_2""] + resource = [] + if resource_1: + resource.append(resource_1) + if resource_2: + resource.append(resource_2) + resource = tuple(resource) + result = await self.ds.permission_allowed( + actor, permission, resource, default=""USE_DEFAULT"" + ) + return Response.json( + { + ""actor"": actor, + ""permission"": permission, + ""resource"": resource, + ""result"": result, + } + ) + class MessagesDebugView(BaseView): name = ""messages_debug"" ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1855#issuecomment-1301594495,https://api.github.com/repos/simonw/datasette/issues/1855,1301594495,IC_kwDOBm6k_c5NlMF_,9599,simonw,2022-11-03T03:11:17Z,2022-11-03T03:11:17Z,OWNER,"Maybe the way to do this is through a new standard mechanism on the actor: a set of additional restrictions, e.g.: ``` { ""id"": ""root"", ""_r"": { ""a"": [""ir"", ""ur"", ""dr""], ""d"": { ""fixtures"": [""ir"", ""ur"", ""dr""] }, ""t"": { ""fixtures"": { ""searchable"": [""ir""] } } } ``` `""a""` is ""all permissions"" - these apply to everything. `""d""` permissions only apply to the specified database `""t""` permissions only apply to the specified table The way this works is there's a default [permission_allowed(datasette, actor, action, resource)](https://docs.datasette.io/en/stable/plugin_hooks.html#id25) hook which only consults these, and crucially just says NO if those rules do not match. In this way it would apply as an extra layer of permission rules over the defaults (which for this `root` instance would all return yes).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1871#issuecomment-1299607082,https://api.github.com/repos/simonw/datasette/issues/1871,1299607082,IC_kwDOBm6k_c5Ndm4q,9599,simonw,2022-11-02T05:45:31Z,2022-11-02T05:45:31Z,OWNER,"I'm going to add a link to the Datasette API docs for the current running version of Datasette, e.g. to https://docs.datasette.io/en/0.63/json_api.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299600257,https://api.github.com/repos/simonw/datasette/issues/1871,1299600257,IC_kwDOBm6k_c5NdlOB,9599,simonw,2022-11-02T05:36:40Z,2022-11-02T05:36:40Z,OWNER,"The API Explorer should definitely link to the `/-/create-token` page for users who have permission though. And it should probably go in the Datasette application menu?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299599461,https://api.github.com/repos/simonw/datasette/issues/1871,1299599461,IC_kwDOBm6k_c5NdlBl,9599,simonw,2022-11-02T05:35:36Z,2022-11-02T05:36:15Z,OWNER,"Here's a slightly wild idea: what if there was a button on `/-/api` that you could click to turn on ""API explorer mode"" for the rest of the Datasette interface - which sets a cookie, and that cookie means you then see ""API explorer"" links in all sorts of other relevant places in the Datasette UI (maybe tucked away in cog menus). Only reason I don't want to show these to everyone is that I don't think this is a very user-friendly feature: if you don't know what an API is I don't want to expose you to it unnecessarily.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299598570,https://api.github.com/repos/simonw/datasette/issues/1871,1299598570,IC_kwDOBm6k_c5Ndkzq,9599,simonw,2022-11-02T05:34:28Z,2022-11-02T05:34:28Z,OWNER,"This is pretty useful now. Two features I still want to add: - The ability to link to the API explorer such that the form is pre-filled with material from the URL. Need to guard against clickjacking first though, so no-one can link to it in an invisible iframe and trick the user into hitting POST. - Some kind of list of endpoints so people can click links to start using the API explorer. A list of every table the user can write to with each of their `/db/table/-/insert` endpoints for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299597066,https://api.github.com/repos/simonw/datasette/issues/1871,1299597066,IC_kwDOBm6k_c5NdkcK,9599,simonw,2022-11-02T05:32:22Z,2022-11-02T05:32:22Z,OWNER,"Demo of the latest API explorer: ![explorer](https://user-images.githubusercontent.com/9599/199406184-1292df42-25ea-4daf-8b54-ca26170ec1ea.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299388341,https://api.github.com/repos/simonw/datasette/issues/1871,1299388341,IC_kwDOBm6k_c5Ncxe1,9599,simonw,2022-11-02T00:24:28Z,2022-11-02T00:25:00Z,OWNER,"I want JSON syntax highlighting. https://github.com/luyilin/json-format-highlight is an MIT licensed tiny highlighter that looks decent for this. https://unpkg.com/json-format-highlight@1.0.1/dist/json-format-highlight.js","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299349741,https://api.github.com/repos/simonw/datasette/issues/1871,1299349741,IC_kwDOBm6k_c5NcoDt,9599,simonw,2022-11-01T23:22:55Z,2022-11-01T23:22:55Z,OWNER,"It's weird that the API explorer only lets you explore POST APIs. It should probably also let you explore GET APIs, or be renamed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1879#issuecomment-1299102755,https://api.github.com/repos/simonw/datasette/issues/1879,1299102755,IC_kwDOBm6k_c5Nbrwj,9599,simonw,2022-11-01T20:31:37Z,2022-11-01T20:31:37Z,OWNER,And some JavaScript that can spot if Datasette thinks it is being served over HTTP when it's actually being served over HTTPS.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1879#issuecomment-1299102108,https://api.github.com/repos/simonw/datasette/issues/1879,1299102108,IC_kwDOBm6k_c5Nbrmc,9599,simonw,2022-11-01T20:30:54Z,2022-11-01T20:33:06Z,OWNER,One idea: add a `/-/debug` page (or `/-/tips` or `/-/checks`) which shows the incoming requests headers and could even detect if there's an `x-forwarded-host` header that isn't being repeated and show a tip on how to fix that.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1879#issuecomment-1299098458,https://api.github.com/repos/simonw/datasette/issues/1879,1299098458,IC_kwDOBm6k_c5Nbqta,9599,simonw,2022-11-01T20:27:40Z,2022-11-01T20:33:52Z,OWNER,"https://github.com/simonw/datasette-x-forwarded-host/blob/main/datasette_x_forwarded_host/__init__.py could happen in core controlled by: `--setting trust_forwarded_host 1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1879#issuecomment-1299096850,https://api.github.com/repos/simonw/datasette/issues/1879,1299096850,IC_kwDOBm6k_c5NbqUS,9599,simonw,2022-11-01T20:26:12Z,2022-11-01T20:26:12Z,OWNER,"The other relevant plugin here is https://datasette.io/plugins/datasette-x-forwarded-host Maybe that should be rolled into core too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1879#issuecomment-1299090678,https://api.github.com/repos/simonw/datasette/issues/1879,1299090678,IC_kwDOBm6k_c5Nboz2,9599,simonw,2022-11-01T20:20:28Z,2022-11-01T20:20:28Z,OWNER,My first step in debugging these is to install https://datasette.io/plugins/datasette-debug-asgi - but now I'm thinking maybe something like that should be part of core.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1862#issuecomment-1299073433,https://api.github.com/repos/simonw/datasette/issues/1862,1299073433,IC_kwDOBm6k_c5NbkmZ,9599,simonw,2022-11-01T20:04:31Z,2022-11-01T20:04:31Z,OWNER,"It really feels like this should be accompanied by a `/db/-/create` API for creating tables. I had to add that to `sqlite-utils` eventually (initially it only supported creating by passing in an example document): https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425011030,"Create a new table from one or more records, `sqlite-utils` style", https://github.com/simonw/datasette/issues/1878#issuecomment-1299071456,https://api.github.com/repos/simonw/datasette/issues/1878,1299071456,IC_kwDOBm6k_c5NbkHg,9599,simonw,2022-11-01T20:02:43Z,2022-11-01T20:02:43Z,OWNER,"Note that ""update"" is partially covered by the `replace` option to `/-/insert`, added here: - https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1873#issuecomment-1298919552,https://api.github.com/repos/simonw/datasette/issues/1873,1298919552,IC_kwDOBm6k_c5Na_CA,9599,simonw,2022-11-01T18:11:27Z,2022-11-01T18:11:27Z,OWNER,"I forgot to document `ignore` and `replace`. Also I need to add tests that cover: - Forgetting to include a primary key on a non-autoincrement table - Compound primary keys - Rowid only tables with and without rowid specified I think my validation logic here will get caught out by the fact that `rowid` does not show up as a valid column name: https://github.com/simonw/datasette/blob/9bec7c38eb93cde5afb16df9bdd96aea2a5b0459/datasette/views/table.py#L1151-L1160 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1298905135,https://api.github.com/repos/simonw/datasette/issues/1873,1298905135,IC_kwDOBm6k_c5Na7gv,9599,simonw,2022-11-01T17:59:59Z,2022-11-01T17:59:59Z,OWNER,"It's a bit surprising that you can send `""ignore"": true, ""return_rows"": true` and the returned `""inserted""` key will list rows that were NOT inserted (since they were ignored). Three options: 1. Ignore that and document it 2. Fix it so `""inserted""` only returns rows that were actually inserted (bit tricky) 3. Change the name of `""inserted""` to something else I'm picking 3 - I'm going to change it to be called `""rows""` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451,https://api.github.com/repos/simonw/datasette/issues/1873,1298885451,IC_kwDOBm6k_c5Na2tL,9599,simonw,2022-11-01T17:42:20Z,2022-11-01T17:42:20Z,OWNER,"Design decision: ```json { ""rows"": [{""id"": 1, ""title"": ""The title""}], ""ignore"": true } ``` Or `""replace"": true`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298879701,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1298879701,IC_kwDOCGYnMM5Na1TV,9599,simonw,2022-11-01T17:37:13Z,2022-11-01T17:37:13Z,OWNER,"The question I was originally trying to answer here was this: how many rows were actually inserted by that call to `.insert_all()`? I don't know that `.rowcount` would ever be useful here, since the ""correct"" answer depends on other factors - had I determined to ignore or replace records with a primary key that matches an existing record for example? So I think if people need `rowcount` they can get it by using a `cursor` directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298877872,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1298877872,IC_kwDOCGYnMM5Na02w,9599,simonw,2022-11-01T17:35:30Z,2022-11-01T17:35:30Z,OWNER,"This may not make sense. First, `.last_rowid` is a property on table - but that doesn't make sense for `rowcount` since it should clearly be a property on the database itself (you can run a query directly using `db.execute()` without going through a `Table` object). So I tried this prototype: ```diff diff --git a/docs/python-api.rst b/docs/python-api.rst index 206e5e6..78d3a8d 100644 --- a/docs/python-api.rst +++ b/docs/python-api.rst @@ -186,6 +186,15 @@ The ``db.query(sql)`` function executes a SQL query and returns an iterator over # {'name': 'Cleo'} # {'name': 'Pancakes'} +After executing a query the ``db.rowcount`` property on that database instance will reflect the number of rows affected by any insert, update or delete operations performed by that query: + +.. code-block:: python + + db = Database(memory=True) + db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}]) + print(db.rowcount) + # Outputs: 2 + .. _python_api_execute: db.execute(sql, params) diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a06f4b7..c19c2dd 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -294,6 +294,8 @@ class Database: _counts_table_name = ""_counts"" use_counts_table = False + # Number of rows inserted, updated or deleted + rowcount: Optional[int] = None def __init__( self, @@ -480,9 +482,11 @@ class Database: if self._tracer: self._tracer(sql, parameters) if parameters is not None: - return self.conn.execute(sql, parameters) + cursor = self.conn.execute(sql, parameters) else: - return self.conn.execute(sql) + cursor = self.conn.execute(sql) + self.rowcount = cursor.rowcount + return cursor def executescript(self, sql: str) -> sqlite3.Cursor: """""" ``` But this happens: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}])
      >>> db.rowcount -1 ``` Turning on query tracing demonstrates why: ```pycon >>> db = Database(memory=True, tracer=print) PRAGMA recursive_triggers=on; None >>> db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}]) select name from sqlite_master where type = 'view' None select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'view' None CREATE TABLE [dogs] ( [name] TEXT ); None select name from sqlite_master where type = 'view' None INSERT INTO [dogs] ([name]) VALUES (?), (?); ['Cleo', 'Pancakes'] select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'table' None PRAGMA table_info([dogs]) None
      >>> ``` The `.insert_all()` function does a bunch of other queries too, so `.rowcount` is quickly over-ridden by the same result from extra queries that it executed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/datasette/issues/1876#issuecomment-1298856054,https://api.github.com/repos/simonw/datasette/issues/1876,1298856054,IC_kwDOBm6k_c5Navh2,9599,simonw,2022-11-01T17:16:01Z,2022-11-01T17:16:01Z,OWNER,`ta.style.height = ta.scrollHeight + 'px'` is an easy way to do that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1431786951,SQL query should wrap on SQL interrupted screen, https://github.com/simonw/datasette/issues/1876#issuecomment-1298854321,https://api.github.com/repos/simonw/datasette/issues/1876,1298854321,IC_kwDOBm6k_c5NavGx,9599,simonw,2022-11-01T17:14:33Z,2022-11-01T17:14:33Z,OWNER,"I could use a `textarea` here (would need to figure out a neat pattern to expand it to fit the query): ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1431786951,SQL query should wrap on SQL interrupted screen, https://github.com/simonw/datasette/issues/1864#issuecomment-1296403316,https://api.github.com/repos/simonw/datasette/issues/1864,1296403316,IC_kwDOBm6k_c5NRYt0,9599,simonw,2022-10-31T00:39:43Z,2022-10-31T00:39:43Z,OWNER,"It looks like SQLite has features for this already: https://www.sqlite.org/foreignkeys.html#fk_actions > Foreign key ON DELETE and ON UPDATE clauses are used to configure actions that take place when deleting rows from the parent table (ON DELETE), or modifying the parent key values of existing rows (ON UPDATE). A single foreign key constraint may have different actions configured for ON DELETE and ON UPDATE. Foreign key actions are similar to triggers in many ways. On that basis, I'm not going to implement anything additional in the `.../-/delete` endpoint relating to foreign keys. Developers who want special treatment of them can do that with a combination of a plugin (maybe I'll build a `datasette-enable-foreign-keys` plugin) and tables created using those `ON DELETE` clauses.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029275,Delete a single record from an existing table, https://github.com/simonw/datasette/issues/1864#issuecomment-1296402071,https://api.github.com/repos/simonw/datasette/issues/1864,1296402071,IC_kwDOBm6k_c5NRYaX,9599,simonw,2022-10-31T00:37:09Z,2022-10-31T00:37:09Z,OWNER,"I need to think about what happens if you delete a row that is the target of a foreign key from another row. https://www.sqlite.org/foreignkeys.html#fk_enable shows that SQLite will only actively enforce these relationships (e.g. throw an error if you try to delete a row that is referenced by another row) if you first run `PRAGMA foreign_keys = ON;` against the connection. > Foreign key constraints are disabled by default (for backwards compatibility), so must be enabled separately for each [database connection](https://www.sqlite.org/c3ref/sqlite3.html). (Note, however, that future releases of SQLite might change so that foreign key constraints enabled by default. Careful developers will not make any assumptions about whether or not foreign keys are enabled by default but will instead enable or disable them as necessary.) I don't actually believe that the SQLite maintainers will ever make that the default though. Datasette doesn't turn these on at the moment, but it could be turned on by a `prepare_connection()` plugin. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029275,Delete a single record from an existing table, https://github.com/simonw/datasette/issues/1864#issuecomment-1296375536,https://api.github.com/repos/simonw/datasette/issues/1864,1296375536,IC_kwDOBm6k_c5NRR7w,9599,simonw,2022-10-30T23:17:11Z,2022-10-30T23:17:11Z,OWNER,I'm a bit nervous about calling `.delete()` with the `pk_values` - can I be sure they are in the correct order? https://github.com/simonw/datasette/blob/00632ded30e7cf9f0cf9478680645d1dabe269ae/datasette/views/row.py#L188-L190,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029275,Delete a single record from an existing table, https://github.com/simonw/datasette/issues/1864#issuecomment-1296375310,https://api.github.com/repos/simonw/datasette/issues/1864,1296375310,IC_kwDOBm6k_c5NRR4O,9599,simonw,2022-10-30T23:16:19Z,2022-10-30T23:16:19Z,OWNER,Still needs tests that cover compound primary keys and rowid tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029275,Delete a single record from an existing table, https://github.com/simonw/datasette/issues/1874#issuecomment-1296363981,https://api.github.com/repos/simonw/datasette/issues/1874,1296363981,IC_kwDOBm6k_c5NRPHN,9599,simonw,2022-10-30T22:19:47Z,2022-10-30T22:19:47Z,OWNER,Documentation: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,API to drop a table, https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1296358636,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1296358636,IC_kwDOCGYnMM5NRNzs,9599,simonw,2022-10-30T21:52:11Z,2022-10-30T21:52:11Z,OWNER,This could work in a similar way to `db.insert(...).last_rowid`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/datasette/issues/1873#issuecomment-1296343716,https://api.github.com/repos/simonw/datasette/issues/1873,1296343716,IC_kwDOBm6k_c5NRKKk,9599,simonw,2022-10-30T20:24:55Z,2022-10-30T20:24:55Z,OWNER,"I think the key feature I need here is going to be the equivalent of `ignore=True` and `replace=True` for dealing with primary key collisions, see https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite_utils.db.Table.insert","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343317,https://api.github.com/repos/simonw/datasette/issues/1873,1296343317,IC_kwDOBm6k_c5NRKEV,9599,simonw,2022-10-30T20:22:40Z,2022-10-30T20:22:40Z,OWNER,"So maybe they're not actually worth worrying about separately, because they are guaranteed to have a primary key set.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343173,https://api.github.com/repos/simonw/datasette/issues/1873,1296343173,IC_kwDOBm6k_c5NRKCF,9599,simonw,2022-10-30T20:21:54Z,2022-10-30T20:22:20Z,OWNER,"One last case to consider: `WITHOUT ROWID` tables. https://www.sqlite.org/withoutrowid.html > By default, every row in SQLite has a special column, usually called the ""[rowid](https://www.sqlite.org/lang_createtable.html#rowid)"", that uniquely identifies that row within the table. However if the phrase ""WITHOUT ROWID"" is added to the end of a [CREATE TABLE](https://www.sqlite.org/lang_createtable.html) statement, then the special ""rowid"" column is omitted. There are sometimes space and performance advantages to omitting the rowid. > > ... > > Every WITHOUT ROWID table must have a [PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#primkeyconst). An error is raised if a CREATE TABLE statement with the WITHOUT ROWID clause lacks a PRIMARY KEY.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343014,https://api.github.com/repos/simonw/datasette/issues/1873,1296343014,IC_kwDOBm6k_c5NRJ_m,9599,simonw,2022-10-30T20:21:01Z,2022-10-30T20:21:01Z,OWNER,"Actually, for simplicity I'm going to say that you can always set the primary key, even for auto-incrementing primary key columns... but you cannot set it on pure `rowid` columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296342814,https://api.github.com/repos/simonw/datasette/issues/1873,1296342814,IC_kwDOBm6k_c5NRJ8e,9599,simonw,2022-10-30T20:20:05Z,2022-10-30T20:20:05Z,OWNER,"Some notes on what Datasette does already https://latest.datasette.io/fixtures/tags.json?_shape=array returns: ```json [ { ""tag"": ""canine"" }, { ""tag"": ""feline"" } ] ``` That table is defined [like this](https://latest.datasette.io/fixtures/tags): ```sql CREATE TABLE tags ( tag TEXT PRIMARY KEY ); ``` Here's a `rowid` table with no explicit primary key: https://latest.datasette.io/fixtures/binary_data https://latest.datasette.io/fixtures/binary_data.json?_shape=array ```json [ { ""rowid"": 1, ""data"": { ""$base64"": true, ""encoded"": ""FRwCx60F/g=="" } }, { ""rowid"": 2, ""data"": { ""$base64"": true, ""encoded"": ""FRwDx60F/g=="" } }, { ""rowid"": 3, ""data"": null } ] ``` ```sql CREATE TABLE binary_data ( data BLOB ); ``` https://latest.datasette.io/fixtures/simple_primary_key has a text primary key: https://latest.datasette.io/fixtures/simple_primary_key.json?_shape=array ```json [ { ""id"": ""1"", ""content"": ""hello"" }, { ""id"": ""2"", ""content"": ""world"" }, { ""id"": ""3"", ""content"": """" }, { ""id"": ""4"", ""content"": ""RENDER_CELL_DEMO"" }, { ""id"": ""5"", ""content"": ""RENDER_CELL_ASYNC"" } ] ``` ```sql CREATE TABLE simple_primary_key ( id varchar(30) primary key, content text ); ``` https://latest.datasette.io/fixtures/compound_primary_key is a compound primary key. https://latest.datasette.io/fixtures/compound_primary_key.json?_shape=array ```json [ { ""pk1"": ""a"", ""pk2"": ""b"", ""content"": ""c"" }, { ""pk1"": ""a/b"", ""pk2"": "".c-d"", ""content"": ""c"" } ] ``` ```sql CREATE TABLE compound_primary_key ( pk1 varchar(30), pk2 varchar(30), content text, PRIMARY KEY (pk1, pk2) ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296341469,https://api.github.com/repos/simonw/datasette/issues/1873,1296341469,IC_kwDOBm6k_c5NRJnd,9599,simonw,2022-10-30T20:13:50Z,2022-10-30T20:13:50Z,OWNER,"I checked and SQLite itself does allow you to set the `rowid` on that kind of table - it then increments from whatever you inserted: ``` % sqlite3 /tmp/t.db SQLite version 3.39.4 2022-09-07 20:51:41 Enter "".help"" for usage hints. sqlite> create table docs (title text); sqlite> insert into docs (title) values ('one'); sqlite> select rowid, title from docs; 1|one sqlite> insert into docs (rowid, title) values (3, 'three'); sqlite> select rowid, title from docs; 1|one 3|three sqlite> insert into docs (title) values ('another'); sqlite> select rowid, title from docs; 1|one 3|three 4|another ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296341055,https://api.github.com/repos/simonw/datasette/issues/1873,1296341055,IC_kwDOBm6k_c5NRJg_,9599,simonw,2022-10-30T20:11:47Z,2022-10-30T20:12:30Z,OWNER,"If a table has an auto-incrementing primary key, should you be allowed to insert records with an explicit key into it? I'm torn on this one. It's something you can do with direct database access, but it's something I very rarely want to do. I'm inclined to disallow it and say that if you want that you can get it using a writable canned query instead. Likewise, I'm not going to provide a way to set the `rowid` explicitly on a freshly inserted row.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1871#issuecomment-1296339386,https://api.github.com/repos/simonw/datasette/issues/1871,1296339386,IC_kwDOBm6k_c5NRJG6,9599,simonw,2022-10-30T20:03:04Z,2022-10-30T20:03:04Z,OWNER,"I do need to skip CSRF for these API calls. I'm going to start out by doing that using the `skip_csrf()` hook to skip CSRF checks on anything with a `content-type: application/json` request header. ```python @hookimpl def skip_csrf(scope): if scope[""type""] == ""http"": headers = scope.get(""headers"") if dict(headers).get(b'content-type') == b'application/json': return True ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296339205,https://api.github.com/repos/simonw/datasette/issues/1871,1296339205,IC_kwDOBm6k_c5NRJEF,9599,simonw,2022-10-30T20:02:05Z,2022-10-30T20:02:05Z,OWNER,"Realized the API explorer doesn't need the API key piece at all - it can work with standard cookie-based auth. This also reflects how most plugins are likely to use this API, where they'll be adding JavaScript that uses `fetch()` to call the write API directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296131872,https://api.github.com/repos/simonw/datasette/issues/1871,1296131872,IC_kwDOBm6k_c5NQWcg,9599,simonw,2022-10-30T06:27:56Z,2022-10-30T06:27:56Z,OWNER,Initial prototype API explorer is now live at https://latest-1-0-dev.datasette.io/-/api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1873#issuecomment-1296131681,https://api.github.com/repos/simonw/datasette/issues/1873,1296131681,IC_kwDOBm6k_c5NQWZh,9599,simonw,2022-10-30T06:27:12Z,2022-10-30T06:27:12Z,OWNER,Relevant TODO: https://github.com/simonw/datasette/blob/c35859ae3df163406f1a1895ccf9803e933b2d8e/datasette/views/table.py#L1131-L1135,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1872#issuecomment-1296131343,https://api.github.com/repos/simonw/datasette/issues/1872,1296131343,IC_kwDOBm6k_c5NQWUP,9599,simonw,2022-10-30T06:26:01Z,2022-10-30T06:26:01Z,OWNER,"Good spot fixing that! Sorry about this - it was a change in Datasette 0.63 which should have been better called out. My goal for Datasette 1.0 (which I aim to have out by the end of the year) is to introduce a formal process for avoiding problems like this, with very clear documentation when something like this might happen.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428560020,"SITE-BUSTING ERROR: ""render_template() called before await ds.invoke_startup()""", https://github.com/simonw/datasette/issues/1871#issuecomment-1296130073,https://api.github.com/repos/simonw/datasette/issues/1871,1296130073,IC_kwDOBm6k_c5NQWAZ,9599,simonw,2022-10-30T06:20:56Z,2022-10-30T06:20:56Z,OWNER,"That initial prototype looks like this: It currently shows the returned JSON from the API in an `alert()`. Next I should make that part of the page instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296126389,https://api.github.com/repos/simonw/datasette/issues/1871,1296126389,IC_kwDOBm6k_c5NQVG1,9599,simonw,2022-10-30T06:04:48Z,2022-10-30T06:04:48Z,OWNER,"This is even more important now I have pushed: - #1866","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296114136,https://api.github.com/repos/simonw/datasette/issues/1871,1296114136,IC_kwDOBm6k_c5NQSHY,9599,simonw,2022-10-30T05:15:40Z,2022-10-30T05:15:40Z,OWNER,"Host it at `/-/api` It's an input box with a path in and a textarea you can put JSON in, plus a submit button to post the request. It lists the API endpoints you can use - click on a link to populate the form field plus a example. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/pull/1870#issuecomment-1295660092,https://api.github.com/repos/simonw/datasette/issues/1870,1295660092,IC_kwDOBm6k_c5NOjQ8,9599,simonw,2022-10-29T00:25:26Z,2022-10-29T00:25:26Z,OWNER,"Saw your comment here too: https://github.com/simonw/datasette/issues/1480#issuecomment-1271101072 > switching from `immutable=1` to `mode=ro` completely addressed this. see https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 for details. So maybe we need a special case for containers that are intended to be run using Docker - the ones produced by `datasette package` and `datasette publish cloudrun`? Those are cases where the `-i` option should actually be opened in read-only mode, not immutable mode. Maybe a `datasette serve --irw data.db` option for opening a file in immutable-but-actually-read-only mode? Bit ugly though. I should run some benchmarks to figure out if `immutable` really does offer significant performance benefits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/pull/1870#issuecomment-1295657771,https://api.github.com/repos/simonw/datasette/issues/1870,1295657771,IC_kwDOBm6k_c5NOisr,9599,simonw,2022-10-29T00:19:03Z,2022-10-29T00:19:03Z,OWNER,"Just saw your comment here: https://github.com/simonw/datasette/issues/1836#issuecomment-1272357976 > when you are running from docker, you **always** will want to run as `mode=ro` because the same thing that is causing duplication in the inspect layer will cause duplication in the final container read/write layer when `datasette serve` runs. I don't understand this. My mental model of how Docker works is that the image itself is created using `docker build`... but then when the image runs later on (`docker run`) the image itself isn't touched at all. Are you saying that I can build a container, but then when I run it and it does `datasette serve -i data.db ...` it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that `serve` command?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/issues/1866#issuecomment-1295200988,https://api.github.com/repos/simonw/datasette/issues/1866,1295200988,IC_kwDOBm6k_c5NMzLc,9599,simonw,2022-10-28T16:29:55Z,2022-10-28T16:29:55Z,OWNER,"I wonder if there's something clever I could do here within a transaction? Start a transaction. Write out a temporary in-memory table with all of the existing primary keys in the table. Run the bulk insert. Then run `select pk from table where pk not in (select pk from old_pks)` to see what has changed. I don't think that's going to work well for large tables. I'm going to go with not returning inserted rows by default, unless you pass a special option requesting that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1294316640,https://api.github.com/repos/simonw/datasette/issues/1866,1294316640,IC_kwDOBm6k_c5NJbRg,9599,simonw,2022-10-28T01:51:40Z,2022-10-28T01:51:40Z,OWNER,"This needs to support the following: - Rows do not include a primary key - one is assigned by the database - Rows provide their own primary key, any clashes are errors - Rows provide their own primary key, clashes are silently ignored - Rows provide their own primary key, replacing any existing records","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1294306071,https://api.github.com/repos/simonw/datasette/issues/1866,1294306071,IC_kwDOBm6k_c5NJYsX,9599,simonw,2022-10-28T01:37:14Z,2022-10-28T01:37:59Z,OWNER,"Quick crude benchmark: ```python import sqlite3 db = sqlite3.connect("":memory:"") def create_table(db, name): db.execute(f""create table {name} (id integer primary key, title text)"") create_table(db, ""single"") create_table(db, ""multi"") create_table(db, ""bulk"") def insert_singles(titles): inserted = [] for title in titles: cursor = db.execute(f""insert into single (title) values (?)"", [title]) inserted.append((cursor.lastrowid, title)) return inserted def insert_many(titles): db.executemany(f""insert into multi (title) values (?)"", ((t,) for t in titles)) def insert_bulk(titles): db.execute(""insert into bulk (title) values {}"".format( "", "".join(""(?)"" for _ in titles) ), titles) titles = [""title {}"".format(i) for i in range(1, 10001)] ``` Then in iPython I ran these: ``` In [14]: %timeit insert_singles(titles) 23.8 ms ± 535 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) In [13]: %timeit insert_many(titles) 12 ms ± 520 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [12]: %timeit insert_bulk(titles) 2.59 ms ± 25 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) ``` So the bulk insert really is a lot faster - 3ms compared to 24ms for single inserts, so ~8x faster.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1294296767,https://api.github.com/repos/simonw/datasette/issues/1866,1294296767,IC_kwDOBm6k_c5NJWa_,9599,simonw,2022-10-28T01:22:25Z,2022-10-28T01:23:09Z,OWNER,"Nasty catch on this one: I wanted to return the IDs of the freshly inserted rows. But... the `insert_all()` method I was planning to use from `sqlite-utils` doesn't appear to have a way of doing that: https://github.com/simonw/sqlite-utils/blob/529110e7d8c4a6b1bbf5fb61f2e29d72aa95a611/sqlite_utils/db.py#L2813-L2835 SQLite itself added a `RETURNING` statement which might help, but that is only available from version 3.35 released in March 2021: https://www.sqlite.org/lang_returning.html - which isn't commonly available yet. https://latest.datasette.io/-/versions right now shows 3.34, and https://lite.datasette.io/#/-/versions shows 3.27.2 (from Feb 2019). Two options then: 1. Even for bulk inserts do one insert at a time so I can use `cursor.lastrowid` to get the ID of the inserted record. This isn't terrible since SQLite is very fast, but it may still be a big performance hit for large inserts. 2. Don't return the list of inserted rows for bulk inserts 3. Default to not returning the list of inserted rows for bulk inserts, but allow the user to request that - in which case we use the slower path That third option might be the way to go here. I should benchmark first to figure out how much of a difference this actually makes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1294282263,https://api.github.com/repos/simonw/datasette/issues/1866,1294282263,IC_kwDOBm6k_c5NJS4X,9599,simonw,2022-10-28T01:00:42Z,2022-10-28T01:00:42Z,OWNER,"I'm going to set the limit at 1,000 rows inserted at a time. I'll make this configurable using a new `max_insert_rows` setting (for consistency with `max_returned_rows`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1851#issuecomment-1294281451,https://api.github.com/repos/simonw/datasette/issues/1851,1294281451,IC_kwDOBm6k_c5NJSrr,9599,simonw,2022-10-28T00:59:25Z,2022-10-28T00:59:25Z,OWNER,"I'm going to use this endpoint for bulk inserts too, so I'm closing this issue and continuing the work here: - #1866","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1294224185,https://api.github.com/repos/simonw/datasette/issues/1851,1294224185,IC_kwDOBm6k_c5NJEs5,9599,simonw,2022-10-27T23:18:24Z,2022-11-03T23:26:05Z,OWNER,"So new API design is: ``` POST /db/table/-/insert Authorization: Bearer xxx Content-Type: application/json { ""row"": { ""id"": 1, ""name"": ""New record"" } } ``` Returns: ``` 201 Created { ""row"": [{ ""id"": 1, ""name"": ""New record"" }] } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1869#issuecomment-1294181485,https://api.github.com/repos/simonw/datasette/issues/1869,1294181485,IC_kwDOBm6k_c5NI6Rt,9599,simonw,2022-10-27T22:24:37Z,2022-10-27T22:24:37Z,OWNER,"https://docs.datasette.io/en/stable/changelog.html#v0-63 Annotated release notes: https://simonwillison.net/2022/Oct/27/datasette-0-63/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426253476,Release 0.63, https://github.com/simonw/datasette/issues/1786#issuecomment-1294116493,https://api.github.com/repos/simonw/datasette/issues/1786,1294116493,IC_kwDOBm6k_c5NIqaN,9599,simonw,2022-10-27T21:50:12Z,2022-10-27T21:50:12Z,OWNER,Demo in Datasette Lite: https://lite.datasette.io/#/fixtures?sql=select%0A++pk1%2C%0A++pk2%2C%0A++content%2C%0A++sortable%2C%0A++sortable_with_nulls%2C%0A++sortable_with_nulls_2%2C%0A++text%0Afrom%0A++sortable%0Aorder+by%0A++pk1%2C%0A++pk2%0Alimit%0A++101,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1342430983,Adjust height of textarea for no JS case, https://github.com/simonw/datasette/issues/1869#issuecomment-1294105558,https://api.github.com/repos/simonw/datasette/issues/1869,1294105558,IC_kwDOBm6k_c5NInvW,9599,simonw,2022-10-27T21:44:13Z,2022-10-27T21:44:13Z,OWNER,I'm going to do annotated release notes for this one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426253476,Release 0.63, https://github.com/simonw/datasette/issues/1869#issuecomment-1294056552,https://api.github.com/repos/simonw/datasette/issues/1869,1294056552,IC_kwDOBm6k_c5NIbxo,9599,simonw,2022-10-27T21:00:02Z,2022-10-27T21:02:25Z,OWNER,"Those release notes as markdown: ### Features - Now tested against Python 3.11. Docker containers used by `datasette publish` and `datasette package` both now use that version of Python. ([#1853](https://github.com/simonw/datasette/issues/1853)) - `--load-extension` option now supports entrypoints. Thanks, Alex Garcia. ([#1789](https://github.com/simonw/datasette/pull/1789)) - Facet size can now be set per-table with the new `facet_size` table metadata option. ([#1804](https://github.com/simonw/datasette/issues/1804)) - The [truncate_cells_html](https://docs.datasette.io/en/stable/settings.html#setting-truncate-cells-html) setting now also affects long URLs in columns. ([#1805](https://github.com/simonw/datasette/issues/1805)) - The non-JavaScript SQL editor textarea now increases height to fit the SQL query. ([#1786](https://github.com/simonw/datasette/issues/1786)) - Facets are now displayed with better line-breaks in long values. Thanks, Daniel Rech. ([#1794](https://github.com/simonw/datasette/pull/1794)) - The `settings.json` file used in [Configuration directory mode](https://docs.datasette.io/en/stable/settings.html#config-dir) is now validated on startup. ([#1816](https://github.com/simonw/datasette/issues/1816)) - SQL queries can now include leading SQL comments, using `/* ... */` or `-- ...` syntax. Thanks, Charles Nepote. ([#1860](https://github.com/simonw/datasette/issues/1860)) - SQL query is now re-displayed when terminated with a time limit error. ([#1819](https://github.com/simonw/datasette/issues/1819)) - The [inspect data](https://docs.datasette.io/en/stable/performance.html#performance-inspect) mechanism is now used to speed up server startup - thanks, Forest Gregg. ([#1834](https://github.com/simonw/datasette/issues/1834)) - In [Configuration directory mode](https://docs.datasette.io/en/stable/settings.html#config-dir) databases with filenames ending in `.sqlite` or `.sqlite3` are now automatically added to the Datasette instance. ([#1646](https://github.com/simonw/datasette/issues/1646)) - Breadcrumb navigation display now respects the current user's permissions. ([#1831](https://github.com/simonw/datasette/issues/1831)) ### Plugin hooks and internals - The [prepare_jinja2_environment(env, datasette)](https://docs.datasette.io/en/stable/plugin_hooks.html#plugin-hook-prepare-jinja2-environment) plugin hook now accepts an optional `datasette` argument. Hook implementations can also now return an `async` function which will be awaited automatically. ([#1809](https://github.com/simonw/datasette/issues/1809)) - `Database(is_mutable=)` now defaults to `True`. ([#1808](https://github.com/simonw/datasette/issues/1808)) - The [datasette.check_visibility()](https://docs.datasette.io/en/stable/internals.html#datasette-check-visibility) method now accepts an optional `permissions=` list, allowing it to take multiple permissions into account at once when deciding if something should be shown as public or private. This has been used to correctly display padlock icons in more places in the Datasette interface. ([#1829](https://github.com/simonw/datasette/issues/1829)) - Datasette no longer enforces upper bounds on its dependencies. ([#1800](https://github.com/simonw/datasette/issues/1800)) ### Documentation - New tutorial: [Cleaning data with sqlite-utils and Datasette](https://datasette.io/tutorials/clean-data). - Screenshots in the documentation are now maintained using [shot-scraper](https://shot-scraper.datasette.io/), as described in [Automating screenshots for the Datasette documentation using shot-scraper](https://simonwillison.net/2022/Oct/14/automating-screenshots/). ([#1844](https://github.com/simonw/datasette/issues/1844)) - More detailed command descriptions on the [CLI reference](https://docs.datasette.io/en/stable/cli-reference.html#cli-reference) page. ([#1787](https://github.com/simonw/datasette/issues/1787)) - New documentation on [Running Datasette using OpenRC](https://docs.datasette.io/en/stable/deploying.html#deploying-openrc) - thanks, Adam Simpson. ([#1825](https://github.com/simonw/datasette/pull/1825))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426253476,Release 0.63, https://github.com/simonw/datasette/pull/1835#issuecomment-1294049178,https://api.github.com/repos/simonw/datasette/issues/1835,1294049178,IC_kwDOBm6k_c5NIZ-a,9599,simonw,2022-10-27T20:51:30Z,2022-10-27T20:51:30Z,OWNER,"See also: - https://github.com/simonw/datasette/pull/1837","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400121355,use inspect data for hash and file size, https://github.com/simonw/datasette/pull/1837#issuecomment-1294048849,https://api.github.com/repos/simonw/datasette/issues/1837,1294048849,IC_kwDOBm6k_c5NIZ5R,9599,simonw,2022-10-27T20:51:08Z,2022-10-27T20:51:08Z,OWNER,"Yeah this is better, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400431789,Make hash and size a lazy property, https://github.com/simonw/datasette/pull/1839#issuecomment-1294034011,https://api.github.com/repos/simonw/datasette/issues/1839,1294034011,IC_kwDOBm6k_c5NIWRb,9599,simonw,2022-10-27T20:34:37Z,2022-10-27T20:34:37Z,OWNER,@dependabot rebase,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1401155623,Bump black from 22.8.0 to 22.10.0, https://github.com/simonw/datasette/issues/1851#issuecomment-1294012583,https://api.github.com/repos/simonw/datasette/issues/1851,1294012583,IC_kwDOBm6k_c5NIRCn,9599,simonw,2022-10-27T20:11:22Z,2022-10-27T20:11:22Z,OWNER,"And the response to `""inserted"": [{...}]` - it will be the same for bulk inserts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1294012084,https://api.github.com/repos/simonw/datasette/issues/1851,1294012084,IC_kwDOBm6k_c5NIQ60,9599,simonw,2022-10-27T20:10:47Z,2022-10-27T20:10:47Z,OWNER,"I'm going to change the incoming JSON back to `{""row"": {...}}` - no need to POST `{""insert"": ...}` to something with `/-/insert` in the URL already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1294009354,https://api.github.com/repos/simonw/datasette/issues/1851,1294009354,IC_kwDOBm6k_c5NIQQK,9599,simonw,2022-10-27T20:07:42Z,2022-10-27T20:07:42Z,OWNER,"Need to implement the new URL design from: - #1868 This is now going to be `/db/table/-/insert` - and it will eventually handle bulk inserts as well as single inserts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1868#issuecomment-1294008733,https://api.github.com/repos/simonw/datasette/issues/1868,1294008733,IC_kwDOBm6k_c5NIQGd,9599,simonw,2022-10-27T20:07:01Z,2022-10-27T20:07:01Z,OWNER,I'm happy with this `/db/table/-/action` design for the moment. Will review it once I've built it to see if I still like it!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1868#issuecomment-1294008282,https://api.github.com/repos/simonw/datasette/issues/1868,1294008282,IC_kwDOBm6k_c5NIP_a,9599,simonw,2022-10-27T20:06:34Z,2022-10-27T20:06:34Z,OWNER,"I'm going to stick with one `/-/insert` endpoint which handles both single row inserts AND multiple row inserts I think - partly because I don't want to build both `/-/upsert` and `/-/upsert-many`, I'd rather just have `/-/upsert`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1868#issuecomment-1294007024,https://api.github.com/repos/simonw/datasette/issues/1868,1294007024,IC_kwDOBm6k_c5NIPrw,9599,simonw,2022-10-27T20:05:44Z,2022-10-27T20:05:52Z,OWNER,"So given this scheme, the URL design would look like this: - `POST /db/table/-/insert` - insert a single row - `POST /db/table/-/insert-many` - insert multiple rows (might just keep that on `/-/insert` with a JSON array rather than object though) - `POST /db/table/-/drop` - drop a table - `POST /db/table/-/alter` - alter a table - `POST /db/table/-/upsert` - upsert, https://sqlite-utils.datasette.io/en/stable/python-api.html#upserting-data - `POST /db/table/-/create` - could be an endpoint for explicitly creating a table, or should that live at `/db/-/create` instead? And for rows (`pks` here since compound primary keys are supported): - `POST /db/table/pks/-/update` - update row - `POST /db/table/pks/-/delete` - delete row","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1868#issuecomment-1294004308,https://api.github.com/repos/simonw/datasette/issues/1868,1294004308,IC_kwDOBm6k_c5NIPBU,9599,simonw,2022-10-27T20:03:08Z,2022-10-27T20:03:08Z,OWNER,The other option here would be to lean into custom HTTP verbs like `DELETE` and `PATCH`. I'm not sold on those: they've never given me any convincing wins over just using `POST` for the many times I've encountered them in my career to date.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1868#issuecomment-1294003701,https://api.github.com/repos/simonw/datasette/issues/1868,1294003701,IC_kwDOBm6k_c5NIO31,9599,simonw,2022-10-27T20:02:26Z,2022-10-27T20:02:26Z,OWNER,"The problem with the above design is that I want to support a bunch of different actions that can be taken against a table: - insert a single row - insert multiple rows - bulk update rows - rename table - alter table - drop table I could have ALL of those be a `POST /db/table` with different JSON root keys (`{""drop"": true}` for example, but this raises two problems: 1. Server logs that only show `POST /db/table` will be less useful, they won't reveal what action was performed 2. What happens if you send `{""insert"": {""title"": ""New record""}, ""drop"": true}`? Does that return an error, or does it perform both of those actions? This is already slightly confusing in that `POST /db/name-of-query` is the existing API for executing a writable canned query: https://docs.datasette.io/en/stable/sql_queries.html#json-api-for-writable-canned-queries So I'm ready to consider other design options. Initial thoughts on possible designs (for the single row insert case, but could be expanded to cover other verbs): - `POST /db/table?action=insert` - `POST /db/table?nsert` - `POST /db/table/-/insert` I quite like that third one: it feels consistent with the existing `/-/actor` etc pages that Datasette serves already. There's one slight confusion here in that it overlaps with the URL for a row with a primary key of `""-""` - which is currently at `/db/table/-` - but that might be OK. Especially if I say that child pages of rows must theselves use the `/-/` pattern. So to update or delet a row you would use: - `POST /db/table/row/-/update` - `POST /db/table/row/-/delete` So a row with primary key `-` would end up as `/db/table/row/-/-/update` - which I think is OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1851#issuecomment-1293996735,https://api.github.com/repos/simonw/datasette/issues/1851,1293996735,IC_kwDOBm6k_c5NINK_,9599,simonw,2022-10-27T19:54:53Z,2022-10-27T19:54:53Z,OWNER,"Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#inserting-a-single-row ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1860#issuecomment-1293939737,https://api.github.com/repos/simonw/datasette/issues/1860,1293939737,IC_kwDOBm6k_c5NH_QZ,9599,simonw,2022-10-27T18:57:37Z,2022-10-27T18:57:37Z,OWNER,The new code is now live at https://latest.datasette.io/fixtures,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1293928738,https://api.github.com/repos/simonw/datasette/issues/1860,1293928738,IC_kwDOBm6k_c5NH8ki,9599,simonw,2022-10-27T18:46:31Z,2022-10-27T18:46:31Z,OWNER,I think mine has a better pattern for handling `/* ... anything in here that isn't */ ... */`,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1293928230,https://api.github.com/repos/simonw/datasette/issues/1860,1293928230,IC_kwDOBm6k_c5NH8cm,9599,simonw,2022-10-27T18:46:03Z,2022-10-27T18:46:03Z,OWNER,"Here's yours on Debuggex: https://www.debuggex.com/r/HjdJryTy9ezGsuWK ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1293926417,https://api.github.com/repos/simonw/datasette/issues/1860,1293926417,IC_kwDOBm6k_c5NH8AR,9599,simonw,2022-10-27T18:44:20Z,2022-10-27T18:45:21Z,OWNER,"Hah, I just came up with this one - we were clearly working on this at the same time! `^\s*((?:\-\-.*?\n\s*)|(?:\/\*((?!\*\/)[\s\S])*\*\/)\s*)*\s*select\b` https://www.debuggex.com/r/Rbw-UWD9PdOU2GyO ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1866#issuecomment-1293893789,https://api.github.com/repos/simonw/datasette/issues/1866,1293893789,IC_kwDOBm6k_c5NH0Cd,9599,simonw,2022-10-27T18:13:00Z,2022-10-27T18:13:00Z,OWNER,If people care about that kind of thing they could always push all of their inserts to a table called `_tablename` and then atomically rename that once they've uploaded all of the data (assuming I provide an atomic-rename-this-table mechanism).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293892818,https://api.github.com/repos/simonw/datasette/issues/1866,1293892818,IC_kwDOBm6k_c5NHzzS,9599,simonw,2022-10-27T18:12:02Z,2022-10-27T18:12:02Z,OWNER,"There's one catch with batched inserts: if your CLI tool fails half way through you could end up with a partially populated table - since a bunch of batches will have succeeded first. I think that's OK. In the future I may want to come up with a way to run multiple batches of inserts inside a single transaction, but I can ignore that for the first release of this feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293891876,https://api.github.com/repos/simonw/datasette/issues/1866,1293891876,IC_kwDOBm6k_c5NHzkk,9599,simonw,2022-10-27T18:11:05Z,2022-10-27T18:11:05Z,OWNER,Likewise for newline-delimited JSON. While it's tempting to want to accept that as an ingest format (because it's nice to generate and stream) I think it's better to have a client application that can turn a stream of newline-delimited JSON into batched JSON inserts.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293891191,https://api.github.com/repos/simonw/datasette/issues/1866,1293891191,IC_kwDOBm6k_c5NHzZ3,9599,simonw,2022-10-27T18:10:22Z,2022-10-27T18:10:22Z,OWNER,"So for the moment I'm just going to concentrate on the JSON API. I can consider CSV variants later on, or as plugins, or both.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293890684,https://api.github.com/repos/simonw/datasette/issues/1866,1293890684,IC_kwDOBm6k_c5NHzR8,9599,simonw,2022-10-27T18:09:52Z,2022-10-27T18:09:52Z,OWNER,"Should this API accept CSV/TSV etc in addition to JSON? I'm torn on this one. My initial instinct is that it should not - and there should instead be a Datasette client library / CLI tool you can use that knows how to turn CSV into batches of JSON calls for when you want to upload a CSV file. I don't think the usability of `curl https://datasette/db/table -F 'data=@path/to/file.csv' -H 'Authentication: Bearer xxx'` is particularly great compared to something like`datasette client insert https://datasette/ db table file.csv --csv` (where the command version could store API tokens for you too).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293887808,https://api.github.com/repos/simonw/datasette/issues/1866,1293887808,IC_kwDOBm6k_c5NHylA,9599,simonw,2022-10-27T18:07:02Z,2022-10-27T18:07:02Z,OWNER,"Error handling is really important here. What should happen if you submit 100 records and one of them has some kind of validation error? How should that error be reported back to you? I'm inclined to say that it defaults to all-or-nothing in a transaction - but there should be a `""continue_on_error"": true` option (or similar) which causes it to insert the ones that are valid while reporting back the ones that are invalid.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1862#issuecomment-1293857306,https://api.github.com/repos/simonw/datasette/issues/1862,1293857306,IC_kwDOBm6k_c5NHrIa,9599,simonw,2022-10-27T17:38:17Z,2022-10-27T17:38:17Z,OWNER,"Strongly related to: - #1866","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425011030,"Create a new table from one or more records, `sqlite-utils` style", https://github.com/simonw/datasette/issues/1865#issuecomment-1293568194,https://api.github.com/repos/simonw/datasette/issues/1865,1293568194,IC_kwDOBm6k_c5NGkjC,9599,simonw,2022-10-27T13:58:26Z,2022-10-27T13:58:26Z,OWNER,"Here's the issue where I started doing this: - #849","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425682079,Stop syncing main to master, https://github.com/simonw/datasette/issues/1851#issuecomment-1292999579,https://api.github.com/repos/simonw/datasette/issues/1851,1292999579,IC_kwDOBm6k_c5NEZub,9599,simonw,2022-10-27T04:59:06Z,2022-10-27T04:59:12Z,OWNER,"I should probably refactor this to use `sqlite-utils`, since I'm going to want to use that later for the feature that automatically creates tables. Might make it easier to solve the rowid issues too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292997608,https://api.github.com/repos/simonw/datasette/issues/1851,1292997608,IC_kwDOBm6k_c5NEZPo,9599,simonw,2022-10-27T04:54:53Z,2022-10-27T19:05:50Z,OWNER,"I'm going to change the design of this to: ``` { ""insert"": { ""title"" :""..."" } } ``` Renaming `""row""` to `""insert""`. This will be consistent with adding `""drop"": true` for dropping a table, and maybe other verbs like for modifying the schema. The API response will look like this: ```json { ""inserted_row"": { ""id"": 1, ""title"": ""..."" } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292996181,https://api.github.com/repos/simonw/datasette/issues/1851,1292996181,IC_kwDOBm6k_c5NEY5V,9599,simonw,2022-10-27T04:51:47Z,2022-10-27T04:51:47Z,OWNER,Also need a test for invalid JSON (currently triggers a 500 HTML error).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1855#issuecomment-1292962813,https://api.github.com/repos/simonw/datasette/issues/1855,1292962813,IC_kwDOBm6k_c5NEQv9,9599,simonw,2022-10-27T04:31:40Z,2022-10-27T04:31:40Z,OWNER,"My hunch on this is that anyone with that level of complex permissions requirements needs to be using a custom authentication plugin which includes much more concrete token rules, rather than the default signed stateless token implementation that ships with Datasette core.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1855#issuecomment-1292959886,https://api.github.com/repos/simonw/datasette/issues/1855,1292959886,IC_kwDOBm6k_c5NEQCO,9599,simonw,2022-10-27T04:30:07Z,2022-10-27T04:30:07Z,OWNER,"Here's an interesting edge-case to consider: what if a user creates themselves a token for a specific table, then deletes that table, and waits for another user to create a table of the same name... and then uses their previously created token to write to the table that someone else created? Not sure if this is a threat I need to actively consider, but it's worth thinking a little bit about the implications of such a thing - since there will be APIs that allow users to create tables, and there may be cases where people want to have a concept of users ""owning"" specific tables. This is probably something that could be left for plugins to solve, but it still needs to be understood and potentially documented. There may even be a world in which tracking the timestamp at which a table was created becomes useful - because that could then be baked into API tokens, such that a token created BEFORE the table was created does not grant access to that table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1851#issuecomment-1292952121,https://api.github.com/repos/simonw/datasette/issues/1851,1292952121,IC_kwDOBm6k_c5NEOI5,9599,simonw,2022-10-27T04:24:09Z,2022-10-27T04:24:20Z,OWNER,"And come up with a whole bunch of tests for weird table shapes, surprising column names, different types etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292951833,https://api.github.com/repos/simonw/datasette/issues/1851,1292951833,IC_kwDOBm6k_c5NEOEZ,9599,simonw,2022-10-27T04:23:40Z,2022-10-27T04:23:40Z,OWNER,Also need to think about transactions - it should use them!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1850#issuecomment-1292940011,https://api.github.com/repos/simonw/datasette/issues/1850,1292940011,IC_kwDOBm6k_c5NELLr,9599,simonw,2022-10-27T04:01:59Z,2022-10-27T04:01:59Z,OWNER,"Working on that first ""insert row"" implementation: - https://github.com/simonw/datasette/issues/1851 Has made it very clear to me that I should go the whole hog and build the basic form-based interface for this as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1851#issuecomment-1292939146,https://api.github.com/repos/simonw/datasette/issues/1851,1292939146,IC_kwDOBm6k_c5NEK-K,9599,simonw,2022-10-27T04:00:17Z,2022-10-27T04:23:15Z,OWNER,"Documentation for this first draft of the API: https://docs.datasette.io/en/1.0-dev/json_api.html#inserting-a-single-row It currently returns errors as HTML - it needs to return errors as JSON. Also the errors need comprehensive test coverage. I'm also worried about what happens if you use it on a table that doesn't use an integer primary key - need to check that. I think this code may break: https://github.com/simonw/datasette/blob/51c436fed29205721dcf17fa31d7e7090d34ebb8/datasette/views/table.py#L155-L171 Plus will `rowid` tables without an explicit primary key return the `rowid` column? They should.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1858#issuecomment-1292709818,https://api.github.com/repos/simonw/datasette/issues/1858,1292709818,IC_kwDOBm6k_c5NDS-6,9599,simonw,2022-10-26T22:07:04Z,2022-10-26T22:07:04Z,OWNER,"New token design: ```json { ""a"": ""actor-id"", ""t"": ""creation timestamp as integer"", ""d"": ""intended duration in seconds, or blank if no duration set"" } ``` This is in place of the `""e"": ""expiry timestamp""` design I've built so far.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423364990,`max_signed_tokens_ttl` setting for a maximum duration on API tokens, https://github.com/simonw/datasette/issues/1858#issuecomment-1292708227,https://api.github.com/repos/simonw/datasette/issues/1858,1292708227,IC_kwDOBm6k_c5NDSmD,9599,simonw,2022-10-26T22:05:34Z,2022-10-26T22:05:34Z,OWNER,"I just realized this can't easily affect the `datasette create-token` command because it doesn't currently accept the `--setting` option, so it wouldn't know what `max_signed_tokens_ttl` was. More to the point: even if it did, someone could abuse their knowledge of the secret to create a signed non-expiring token even on servers that didn't want to support those. So I actually need to redesign the token format: it needs to store the timestamp when the token was created and the intended duration, NOT the timestamp that the token expires at. Otherwise it's not possible for servers to enforce `max_signed_tokens_ttl` - someone could always create a token with a custom `expires_at` timestamp on it outside of the configured limit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423364990,`max_signed_tokens_ttl` setting for a maximum duration on API tokens, https://github.com/simonw/datasette/issues/1858#issuecomment-1292687774,https://api.github.com/repos/simonw/datasette/issues/1858,1292687774,IC_kwDOBm6k_c5NDNme,9599,simonw,2022-10-26T21:44:57Z,2022-10-26T21:44:57Z,OWNER,"I'm going for consistency with `max_csv_mb` and `max_returned_rows` and `allow_signed_tokens` and `default_cache_ttl`. So `max_signed_tokens_ttl`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423364990,`max_signed_tokens_ttl` setting for a maximum duration on API tokens, https://github.com/simonw/datasette/issues/1860#issuecomment-1292685478,https://api.github.com/repos/simonw/datasette/issues/1860,1292685478,IC_kwDOBm6k_c5NDNCm,9599,simonw,2022-10-26T21:42:35Z,2022-10-26T21:42:35Z,OWNER,"That's deployed to https://latest.datasette.io/ now - some examples: - https://latest.datasette.io/fixtures?sql=--+one+kind+of+comment%0D%0Aselect+*+from+searchable - https://latest.datasette.io/fixtures?sql=%2F*+Multi%0D%0A++line+comment+*%2F%0D%0Aselect+*+from+searchable - https://latest.datasette.io/fixtures?sql=%2F*+Both+kinds+*%2F%0D%0A--+of+comment%0D%0A%2F*+and+more+*%2F%0D%0A--+and+more+and+more%0D%0Aselect+*+from+searchable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1292679567,https://api.github.com/repos/simonw/datasette/issues/1860,1292679567,IC_kwDOBm6k_c5NDLmP,9599,simonw,2022-10-26T21:36:25Z,2022-10-26T21:36:25Z,OWNER,I'm never 100% sure how to tell if a regular expression includes a nasty denial of service attack - are there any inputs that could cause this new regex to execute in quadratic time or similar?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1292678657,https://api.github.com/repos/simonw/datasette/issues/1860,1292678657,IC_kwDOBm6k_c5NDLYB,9599,simonw,2022-10-26T21:35:23Z,2022-10-26T21:35:37Z,OWNER,Here are the new tests - each of these should now work: https://github.com/simonw/datasette/blob/55a709c480a1e7401b4ff6208f37a2cf7c682183/tests/test_utils.py#L170-L175,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1292674919,https://api.github.com/repos/simonw/datasette/issues/1860,1292674919,IC_kwDOBm6k_c5NDKdn,9599,simonw,2022-10-26T21:31:22Z,2022-10-26T21:31:22Z,OWNER,"I'm experimenting with this: ```python # Allow SQL to start with a /* */ or -- comment comment_re = ( # Start of string, then any amount of whitespace r'^(\s*' + # Comment that starts with -- and ends at a newline r'(?:\-\-.*?\n\s*)' + # Comment that starts with /* and ends with */ r'|(?:/\*[\s\S]*?\*/)' + # Whitespace r')*\s*' ) allowed_sql_res = [ re.compile(comment_re + r""select\b""), re.compile(comment_re + r""explain\s+select\b""), re.compile(comment_re + r""explain\s+query\s+plan\s+select\b""), re.compile(comment_re + r""with\b""), re.compile(comment_re + r""explain\s+with\b""), re.compile(comment_re + r""explain\s+query\s+plan\s+with\b""), ] ``` This should allow any number of comments of either type as a suffix to the allowed SQL patterns. Needs extensive unit tests! I'm not massively worried if it has a flaw in it though, since this is part of Datasette's defense in depth: if a non-SELECT query sneaks through it still shouldn't be able to cause any damage as the database connection is read-only or immutable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1292659986,https://api.github.com/repos/simonw/datasette/issues/1860,1292659986,IC_kwDOBm6k_c5NDG0S,9599,simonw,2022-10-26T21:14:26Z,2022-10-26T21:15:22Z,OWNER,"Yeah we should fix this. https://www.sqlite.org/lang_comment.html - SQLite also supports `-- style` comments. I like how explicit the documentation is here: > SQL comments begin with two consecutive ""-"" characters (ASCII 0x2d) and extend up to and including the next newline character (ASCII 0x0a) or until the end of input, whichever comes first. > > C-style comments begin with ""/*"" and extend up to and including the next ""*/"" character pair or until the end of input, whichever comes first. C-style comments can span multiple lines. ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1849#issuecomment-1292654852,https://api.github.com/repos/simonw/datasette/issues/1849,1292654852,IC_kwDOBm6k_c5NDFkE,9599,simonw,2022-10-26T21:08:44Z,2022-10-26T21:08:44Z,OWNER,"Generally though we should expect that people might try to use `render_template(...)` without passing a `request`, so Datasette core should be able to handle this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1849#issuecomment-1292654522,https://api.github.com/repos/simonw/datasette/issues/1849,1292654522,IC_kwDOBm6k_c5NDFe6,9599,simonw,2022-10-26T21:08:20Z,2022-10-26T21:08:20Z,OWNER,"From the stack trace in Sentry: So this happened because a custom plugin tried to render `forbidden.html` without passing in the `request`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1849#issuecomment-1292653219,https://api.github.com/repos/simonw/datasette/issues/1849,1292653219,IC_kwDOBm6k_c5NDFKj,9599,simonw,2022-10-26T21:06:56Z,2022-10-26T21:06:56Z,OWNER,"This was a hit to an authenticated page where the incoming user WAS logged in but did not have permission to view that specific page. Code in question: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/app.py#L634-L640","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1851#issuecomment-1292544296,https://api.github.com/repos/simonw/datasette/issues/1851,1292544296,IC_kwDOBm6k_c5NCqko,9599,simonw,2022-10-26T19:33:34Z,2022-10-26T19:33:34Z,OWNER,That trigger solution is pretty neat!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1855#issuecomment-1291485444,https://api.github.com/repos/simonw/datasette/issues/1855,1291485444,IC_kwDOBm6k_c5M-oEE,9599,simonw,2022-10-26T04:30:34Z,2022-10-26T04:30:34Z,OWNER,"I'm going to delay working on this until after I have some of the write APIs built to try it against: - #1851","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1859#issuecomment-1291484749,https://api.github.com/repos/simonw/datasette/issues/1859,1291484749,IC_kwDOBm6k_c5M-n5N,9599,simonw,2022-10-26T04:29:43Z,2022-10-26T04:29:43Z,OWNER,"Documentation: - https://docs.datasette.io/en/1.0-dev/authentication.html#datasette-create-token - https://docs.datasette.io/en/1.0-dev/cli-reference.html#datasette-create-token","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423369494,datasette create-token CLI command, https://github.com/simonw/datasette/issues/1843#issuecomment-1291467084,https://api.github.com/repos/simonw/datasette/issues/1843,1291467084,IC_kwDOBm6k_c5M-jlM,9599,simonw,2022-10-26T04:03:49Z,2022-10-26T04:03:49Z,OWNER,"This time I'm suspicious that there are open SQLite files tucked away in thread locals hidden inside my thread pool executor: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/database.py#L24 https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/database.py#L204-L214","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1291466613,https://api.github.com/repos/simonw/datasette/issues/1843,1291466613,IC_kwDOBm6k_c5M-jd1,9599,simonw,2022-10-26T04:02:56Z,2022-10-26T04:02:56Z,OWNER,Just saw this error again!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1859#issuecomment-1291439998,https://api.github.com/repos/simonw/datasette/issues/1859,1291439998,IC_kwDOBm6k_c5M-c9-,9599,simonw,2022-10-26T03:15:13Z,2022-10-26T03:15:13Z,OWNER,Reads from `DATASETTE_SECRET` or accepts `--secret` for the signing secret.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423369494,datasette create-token CLI command, https://github.com/simonw/datasette/issues/1859#issuecomment-1291439875,https://api.github.com/repos/simonw/datasette/issues/1859,1291439875,IC_kwDOBm6k_c5M-c8D,9599,simonw,2022-10-26T03:14:58Z,2022-10-26T03:14:58Z,OWNER,"Initial design: datasette create-token Or: datasette create-token --expire-after 10m","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423369494,datasette create-token CLI command, https://github.com/simonw/datasette/issues/1858#issuecomment-1291435464,https://api.github.com/repos/simonw/datasette/issues/1858,1291435464,IC_kwDOBm6k_c5M-b3I,9599,simonw,2022-10-26T03:07:16Z,2022-10-26T03:07:16Z,OWNER,"This setting will disable the ""Token never expires"" option: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423364990,`max_signed_tokens_ttl` setting for a maximum duration on API tokens, https://github.com/simonw/datasette/issues/1852#issuecomment-1291431132,https://api.github.com/repos/simonw/datasette/issues/1852,1291431132,IC_kwDOBm6k_c5M-azc,9599,simonw,2022-10-26T02:59:50Z,2022-10-26T02:59:50Z,OWNER,Documentation: https://docs.datasette.io/en/1.0-dev/authentication.html#api-tokens,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1850#issuecomment-1291430992,https://api.github.com/repos/simonw/datasette/issues/1850,1291430992,IC_kwDOBm6k_c5M-axQ,9599,simonw,2022-10-26T02:59:33Z,2022-10-26T02:59:33Z,OWNER,I started the documentation for the API tokens mechanism here: https://docs.datasette.io/en/1.0-dev/authentication.html#api-tokens,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1857#issuecomment-1291418546,https://api.github.com/repos/simonw/datasette/issues/1857,1291418546,IC_kwDOBm6k_c5M-Xuy,9599,simonw,2022-10-26T02:38:35Z,2022-10-26T02:38:35Z,OWNER,"I'm going to set a convention that an actor signed in via a token should set `""token"": ""something""` as a key. Then the `/-/create-token` view can reject those actors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423347412,Prevent API tokens from using /-/create-token to create more tokens, https://github.com/simonw/datasette/issues/1850#issuecomment-1291417755,https://api.github.com/repos/simonw/datasette/issues/1850,1291417755,IC_kwDOBm6k_c5M-Xib,9599,simonw,2022-10-26T02:36:52Z,2022-10-26T02:36:58Z,OWNER,"I'm going to set a convention that `""token"": ""something""` in an actor means that they were authenticated by a token. `""token"": ""dstok""` for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1291417100,https://api.github.com/repos/simonw/datasette/issues/1850,1291417100,IC_kwDOBm6k_c5M-XYM,9599,simonw,2022-10-26T02:35:32Z,2022-10-26T02:35:32Z,OWNER,"It strikes me that users should NOT be able to use a token to create additional tokens. The current design actually does allow that, since the `dstok_` Bearer token can be used to authenticate calls to the `/-/create-token` page. So I think I need a mechanism whereby that page can only allow access to users authenticated by cookie. Not obvious how to do that though, since Datasette's authentication actor system is designed to abstract that detail away!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1856#issuecomment-1291410747,https://api.github.com/repos/simonw/datasette/issues/1856,1291410747,IC_kwDOBm6k_c5M-V07,9599,simonw,2022-10-26T02:27:05Z,2022-10-26T02:27:05Z,OWNER,"Because of that I think this is a better name: --setting allow_signed_tokens off","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336122,allow_signed_tokens setting for disabling API signed token mechanism, https://github.com/simonw/datasette/issues/1856#issuecomment-1291410331,https://api.github.com/repos/simonw/datasette/issues/1856,1291410331,IC_kwDOBm6k_c5M-Vub,9599,simonw,2022-10-26T02:26:19Z,2022-10-26T02:26:19Z,OWNER,"It's a bit confusing that a setting called `allow_create_tokens` also causes incoming `dstok_` tokens to be ignored. Is it confusing enough that I should pick a different name for the setting though?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336122,allow_signed_tokens setting for disabling API signed token mechanism, https://github.com/simonw/datasette/issues/1856#issuecomment-1291409312,https://api.github.com/repos/simonw/datasette/issues/1856,1291409312,IC_kwDOBm6k_c5M-Veg,9599,simonw,2022-10-26T02:24:49Z,2022-10-26T02:24:49Z,OWNER,"The effect of this setting will be: - `/-/create-tokens` interface is no longer available - Incoming `dstok_` tokens are no longer respected by the following code: https://github.com/simonw/datasette/blob/b29e487bc3fde6418bf45bda7cfed2e081ff03fb/datasette/default_permissions.py#L52-L72","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336122,allow_signed_tokens setting for disabling API signed token mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291406219,https://api.github.com/repos/simonw/datasette/issues/1852,1291406219,IC_kwDOBm6k_c5M-UuL,9599,simonw,2022-10-26T02:19:54Z,2022-10-26T02:59:52Z,OWNER,"I'm going to split the remaining work into separate issues: - [x] #1856 - [ ] #1855 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291397623,https://api.github.com/repos/simonw/datasette/issues/1852,1291397623,IC_kwDOBm6k_c5M-Sn3,9599,simonw,2022-10-26T02:11:40Z,2022-10-26T02:11:40Z,OWNER,"Built a prototype of the `actor_from_request()` hook for this and now: ``` % curl http://127.0.0.1:8001/-/actor.json -H 'Authorization: Bearer dstok_eyJhIjoicm9vdCIsImUiOm51bGx9.6O1OxgNTFkAU6uw7xNcmXYX949A' {""actor"": {""id"": ""root"", ""dstok"": true}} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291392887,https://api.github.com/repos/simonw/datasette/issues/1852,1291392887,IC_kwDOBm6k_c5M-Rd3,9599,simonw,2022-10-26T02:04:48Z,2022-10-26T02:04:48Z,OWNER,"Implemented that `dstok_` prefix and the thing where only the `actor[""id""]` is copied to the `""a""` field.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291290451,https://api.github.com/repos/simonw/datasette/issues/1852,1291290451,IC_kwDOBm6k_c5M94dT,9599,simonw,2022-10-26T00:49:56Z,2022-10-26T00:49:56Z,OWNER,Prefix: `dstok_` - for Datasette signed token.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291289369,https://api.github.com/repos/simonw/datasette/issues/1852,1291289369,IC_kwDOBm6k_c5M94MZ,9599,simonw,2022-10-26T00:47:46Z,2022-10-26T00:47:46Z,OWNER,"The tokens also need something that can be used to differentiate them from alternative token mechanisms that other plugins might provide. Maybe a prefix before the signed value. Prefixes are also useful for scanning to check they were not accidentally committed to source control.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291281243,https://api.github.com/repos/simonw/datasette/issues/1852,1291281243,IC_kwDOBm6k_c5M92Nb,9599,simonw,2022-10-26T00:32:21Z,2022-10-26T00:32:21Z,OWNER,"Rather than duplicating the entire actor into the ""a"" field, maybe just copy the actor ID? Would need to restrict token creation to just actors with an ID set. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291274835,https://api.github.com/repos/simonw/datasette/issues/1852,1291274835,IC_kwDOBm6k_c5M90pT,9599,simonw,2022-10-26T00:20:48Z,2022-10-26T00:22:26Z,OWNER,"Tests failed because I added a view without also adding documentation! I forgot that the deploy still goes out for branches other than `main` even if the tests aren't passing: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/.github/workflows/deploy-latest.yml#L34-L38","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291273609,https://api.github.com/repos/simonw/datasette/issues/1852,1291273609,IC_kwDOBm6k_c5M90WJ,9599,simonw,2022-10-26T00:18:40Z,2022-10-26T00:18:40Z,OWNER,"Another thought about tokens that can act on behalf of the user. Imagine a user has permission to access a table. They create a token that can create that table... but then their permission is revoked. It would be bad if they could still use that token they created earlier to access that table! On that basis, I think the model described above where tokens mainly work to provide an ""act on behalf of this actor"" - but with optional additional constraints - is a good one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291272612,https://api.github.com/repos/simonw/datasette/issues/1852,1291272612,IC_kwDOBm6k_c5M90Gk,9599,simonw,2022-10-26T00:16:53Z,2022-10-26T00:16:53Z,OWNER,Next step: make these tokens actually do something.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291272414,https://api.github.com/repos/simonw/datasette/issues/1852,1291272414,IC_kwDOBm6k_c5M90De,9599,simonw,2022-10-26T00:16:28Z,2022-10-26T00:16:28Z,OWNER,If I'm going to change the naming conventions for settings I should do it before Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291272280,https://api.github.com/repos/simonw/datasette/issues/1852,1291272280,IC_kwDOBm6k_c5M90BY,9599,simonw,2022-10-26T00:16:09Z,2022-10-26T00:46:21Z,OWNER,"Other options: - `--setting default_api_tokens off` - `--setting signed_api_tokens off` - `--setting allow_create_token off` These feel inconsistent because they don't use the `allow_` prefix - but they're also a bit less ugly to look at. I like that last one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291271580,https://api.github.com/repos/simonw/datasette/issues/1852,1291271580,IC_kwDOBm6k_c5M9z2c,9599,simonw,2022-10-26T00:14:49Z,2022-10-26T00:15:06Z,OWNER,"If I'm going to have a setting to disable this feature I need to decide what it will be called. Closest existing setting is this one, since it's for a feature that is turned on by default: datasette mydatabase.db --setting allow_download off So maybe this? datasette mydatabase.db --setting allow_signed_api_tokens off I like `allow_signed_api_tokens` more than `allow_api_tokens` because if you install a plugin such as https://datasette.io/plugins/datasette-auth-tokens then API tokens will work even though you disabled this default signed token feature. `allow_signed_api_tokens` does feel a bit clumsy/verbose though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291270227,https://api.github.com/repos/simonw/datasette/issues/1852,1291270227,IC_kwDOBm6k_c5M9zhT,9599,simonw,2022-10-26T00:12:18Z,2022-10-26T00:12:18Z,OWNER,Demo is now live at https://latest-1-0-dev.datasette.io/-/create-token - visit https://latest-1-0-dev.datasette.io/login-as-root first to sign in.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291269607,https://api.github.com/repos/simonw/datasette/issues/1852,1291269607,IC_kwDOBm6k_c5M9zXn,9599,simonw,2022-10-26T00:11:15Z,2022-10-26T00:11:15Z,OWNER,"If you click ""Create token"" for ""Token never expires"" multiple times you currently get exactly the same token each time, since it's just a signed token containing a copy of your actor dictionary. I'm not sure if I like that. I could give each token a random ID (maybe using `secrets.token_hex()`) such that different tokens have different identities, which would be useful for logging and auditing and maybe even revocation at some point in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291268380,https://api.github.com/repos/simonw/datasette/issues/1852,1291268380,IC_kwDOBm6k_c5M9zEc,9599,simonw,2022-10-26T00:09:06Z,2022-10-26T00:09:06Z,OWNER,"Demo: ![token-demo](https://user-images.githubusercontent.com/9599/197904595-e5651d6c-bafc-4124-b762-71ad94c06ced.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291243333,https://api.github.com/repos/simonw/datasette/issues/1852,1291243333,IC_kwDOBm6k_c5M9s9F,9599,simonw,2022-10-25T23:25:13Z,2022-10-25T23:25:13Z,OWNER,"A `/-/debug-token` page that can take a token and decode it to show you how long until it expires, what actor it represents and the permissions it has will be useful as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291234262,https://api.github.com/repos/simonw/datasette/issues/1852,1291234262,IC_kwDOBm6k_c5M9qvW,9599,simonw,2022-10-25T23:11:23Z,2022-10-25T23:11:23Z,OWNER,I'm going to build an initial `/-/create-token` interface which just bakes a token with the current actor in it and an optional expiry timestamp. I'll try the limited permissions thing later.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291233652,https://api.github.com/repos/simonw/datasette/issues/1852,1291233652,IC_kwDOBm6k_c5M9ql0,9599,simonw,2022-10-25T23:10:20Z,2022-10-25T23:10:44Z,OWNER,"In which case the token would need to duplicate the current `actor` and then add extra constraints. So maybe the token design looks like this: ```json { ""a"": { ""copy_of"": ""actor_creating_token""}, ""p"": { ""t"": ""... the thing designed earlier, with those permissions in it"" }, ""e"": ""integer timestamp when token expires"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291232589,https://api.github.com/repos/simonw/datasette/issues/1852,1291232589,IC_kwDOBm6k_c5M9qVN,9599,simonw,2022-10-25T23:08:37Z,2022-10-25T23:08:37Z,OWNER,"... so maybe there's a way to create a token that inherits the exact permissions of the actor that created the token? That could even be a default mode for tokens, with an option to then further restrict permissions if desired.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291231651,https://api.github.com/repos/simonw/datasette/issues/1852,1291231651,IC_kwDOBm6k_c5M9qGj,9599,simonw,2022-10-25T23:07:17Z,2022-10-25T23:07:17Z,OWNER,"Interesting challenge: what permissions should users be allowed to grant to tokens? Clearly a user should not be able to create a token with a permission that the user themselves does not have. And should there be a permission that allows people to create tokens? I think so.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291227942,https://api.github.com/repos/simonw/datasette/issues/1852,1291227942,IC_kwDOBm6k_c5M9pMm,9599,simonw,2022-10-25T23:01:18Z,2022-10-25T23:01:18Z,OWNER,"Datasette currently defaults to having everything public-readable by default, unless a permission plugin changes that default. In thinking more about this API mechanism, I realized that it might be good to have a mode where Datasette _doesn't_ default to public everything. Maybe `datasette --private` to start it like that? Might even be an opportunity to get rid of the current slightly confusing mechanism where permission checks can announce that they should default to true: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/views/database.py#L152-L154","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1851#issuecomment-1291226367,https://api.github.com/repos/simonw/datasette/issues/1851,1291226367,IC_kwDOBm6k_c5M9oz_,9599,simonw,2022-10-25T22:58:30Z,2022-10-25T22:58:30Z,OWNER,"The `datasette insert` concept included plugin support, with the idea of being able to support things like SpatiaLite files: - #1160 I think this API mechanism is going to be a bit less exciting than that - it will be low-level for inserting rows, and if you want to do something fancier you can use a canned query that feeds incoming GeoJSON to a SpatiaLite function instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/sqlite-utils/issues/505#issuecomment-1291216193,https://api.github.com/repos/simonw/sqlite-utils/issues/505,1291216193,IC_kwDOCGYnMM5M9mVB,9599,simonw,2022-10-25T22:41:16Z,2022-10-25T22:41:16Z,OWNER,Tweeted about it here: https://twitter.com/simonw/status/1585038766678609921,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423182778,Release sqlite-utils 3.30, https://github.com/simonw/sqlite-utils/issues/505#issuecomment-1291203911,https://api.github.com/repos/simonw/sqlite-utils/issues/505,1291203911,IC_kwDOCGYnMM5M9jVH,9599,simonw,2022-10-25T22:21:02Z,2022-10-25T22:21:02Z,OWNER,"- Now tested against Python 3.11. ([#502](https://github.com/simonw/sqlite-utils/issues/502)) - New `table.search_sql(include_rank=True)` option, which adds a `rank` column to the generated SQL. Thanks, Jacob Chapman. ([#480](https://github.com/simonw/sqlite-utils/pull/480)) - Progress bars now display for newline-delimited JSON files using the `--nl` option. Thanks, Mischa Untaga. ([#485](https://github.com/simonw/sqlite-utils/issues/485)) - New `db.close()` method. ([#504](https://github.com/simonw/sqlite-utils/issues/504)) - Conversion functions passed to [table.convert(...)](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-convert) can now return lists or dictionaries, which will be inserted into the database as JSON strings. ([#495](https://github.com/simonw/sqlite-utils/issues/495)) - `sqlite-utils install` and `sqlite-utils uninstall` commands for installing packages into the same virtual environment as `sqlite-utils`, [described here](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install). ([#483](https://github.com/simonw/sqlite-utils/issues/483)) - New [sqlite_utils.utils.flatten()](https://sqlite-utils.datasette.io/en/stable/reference.html#reference-utils-flatten) utility function. ([#500](https://github.com/simonw/sqlite-utils/issues/500)) - Documentation on [using Just](https://sqlite-utils.datasette.io/en/stable/contributing.html#contributing-just) to run tests, linters and build documentation. - Documentation now covers the [Release process](https://sqlite-utils.datasette.io/en/stable/contributing.html#release-process) for this package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423182778,Release sqlite-utils 3.30, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1291170072,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1291170072,IC_kwDOCGYnMM5M9bEY,9599,simonw,2022-10-25T21:36:12Z,2022-10-25T21:36:12Z,OWNER,"I was going to suggest using `db.table(name)` instead of `db[name]` - but it looks like that method will have the same problem: https://github.com/simonw/sqlite-utils/blob/defa2974c6d3abc19be28d6b319649b8028dc966/sqlite_utils/db.py#L497-L506 I could change `sqlite-utils` so `db.table(name)` always returns a table and you need to call `db.view(name)` if you want to access a view - that would require bumping to 4.0 though. I'm not convinced that's the best approach here either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1291167887,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1291167887,IC_kwDOCGYnMM5M9aiP,9599,simonw,2022-10-25T21:33:25Z,2022-10-25T21:33:25Z,OWNER,"I do care about this, but I'm not hugely experienced with types yet so I'm open to suggestions about how to do it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1291166273,https://api.github.com/repos/simonw/sqlite-utils/issues/493,1291166273,IC_kwDOCGYnMM5M9aJB,9599,simonw,2022-10-25T21:31:15Z,2022-10-25T21:31:15Z,OWNER,"Based on the docs here I tried the following too: https://docutils.sourceforge.io/docs/user/smartquotes.html#description - `\--` - `\\--` - `\\-\\-` - `\-\-` But none of them had the desired effect in this particular piece of markup: the :ref:`insert \--convert ` I think because this is text inside a `:ref:` block, not regular text. Consider the following: The \--convert and the :ref:`insert \--convert ` and It's rendered like this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386562662,Tiny typographical error in install/uninstall docs, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291159549,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291159549,IC_kwDOCGYnMM5M9Yf9,9599,simonw,2022-10-25T21:23:01Z,2022-10-25T21:23:01Z,OWNER,"I've decided not to explicitly document this, since it's consistent with how other parts of the library work already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291152433,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291152433,IC_kwDOCGYnMM5M9Wwx,9599,simonw,2022-10-25T21:14:54Z,2022-10-25T21:14:54Z,OWNER,"There is a case where the function can return a dictionary at the moment: `multi=True` ```python table.convert( ""title"", lambda v: {""upper"": v.upper(), ""lower"": v.lower()}, multi=True ) ``` But I think this change is still compatible with that. if you don't use `multi=True` then the return value will be stringified. If you DO use `multi=True` then something like this could work: ```python table.convert( ""title"", lambda v: {""upper"": {""str"": v.upper()}, ""lower"": {""str"": v.lower()}}, multi=True ) ``` This would result in a `upper` and `lower` column, each containing the JSON string `{""str"": ""UPPERCASE""}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291149509,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291149509,IC_kwDOCGYnMM5M9WDF,9599,simonw,2022-10-25T21:12:11Z,2022-10-25T21:12:11Z,OWNER,"This makes sense to me. There are other places in the codebase where JSON is automatically stringified: https://github.com/simonw/sqlite-utils/blob/c7e4308e6f49d929704163531632e558f9646e4a/sqlite_utils/db.py#L2759-L2766 I don't see why the return value from a convert function shouldn't do the same thing. Since this will result in previous errors working, I don't think it warrants a major version bump either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/497#issuecomment-1291146850,https://api.github.com/repos/simonw/sqlite-utils/issues/497,1291146850,IC_kwDOCGYnMM5M9VZi,9599,simonw,2022-10-25T21:09:28Z,2022-10-25T21:09:28Z,OWNER,"Yeah, `table.columns` and `table.columns_dict` are meant to handle this: https://sqlite-utils.datasette.io/en/stable/python-api.html#columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393212964,column_names, https://github.com/simonw/sqlite-utils/issues/504#issuecomment-1291136971,https://api.github.com/repos/simonw/sqlite-utils/issues/504,1291136971,IC_kwDOCGYnMM5M9S_L,9599,simonw,2022-10-25T21:00:29Z,2022-10-25T21:00:29Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.close,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423069384,"db.close() method, calling db.conn.close()", https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291124413,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291124413,IC_kwDOCGYnMM5M9P69,9599,simonw,2022-10-25T20:47:34Z,2022-10-25T20:47:34Z,OWNER,TIL about this: https://til.simonwillison.net/python/os-remove-windows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291122389,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291122389,IC_kwDOCGYnMM5M9PbV,9599,simonw,2022-10-25T20:45:43Z,2022-10-25T20:45:43Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291115986,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291115986,IC_kwDOCGYnMM5M9N3S,9599,simonw,2022-10-25T20:39:24Z,2022-10-25T20:39:24Z,OWNER,"Used `psutil` to confirm that closing a SQLite connection closes the underlying file: https://til.simonwillison.net/python/too-many-open-files-psutil ```pycon >>> import psutil >>> import sqlite3 >>> for f in psutil.Process().open_files(): print(f) ... >>> sqlite3.connect(""/tmp/blah.db"") >>> conn = _ >>> for f in psutil.Process().open_files(): print(f) ... popenfile(path='/private/tmp/blah.db', fd=3) >>> conn.close() >>> for f in psutil.Process().open_files(): print(f) ... >>> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291111357,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291111357,IC_kwDOCGYnMM5M9Mu9,9599,simonw,2022-10-25T20:36:06Z,2022-10-25T20:36:06Z,OWNER,... or maybe Windows doesn't like attempts to remove a file that the process has opened?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11,