Forbidden

Create an API token

This token will allow API access with the same abilities as your current user.

This token will allow API access with the same abilities as your current user, {{ request.actor.id }}

Restrict actions that can be performed using this token

All databases and tables

{{ permission }}

+ + {% for database in databases %} +

All tables in database: {{ database }}

{{ permission }}

+ {% endfor %} +

Specific tables

+ {% for dbt in database_with_tables %} + {% for table in dbt.tables %} +

{{ dbt.database }}: {{ table }}

{{ permission }}

+ {% endfor %} + {% endfor %} +

diff --git a/datasette/views/special.py b/datasette/views/special.py index 30345d14..48357f87 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -231,12 +231,37 @@ class CreateTokenView(BaseView): return await self.render( [""create_token.html""], request, - {""actor"": request.actor}, + { + ""actor"": request.actor, + ""all_permissions"": self.ds.permissions.keys(), + ""database_permissions"": [ + key + for key, value in self.ds.permissions.items() + if value.takes_database + ], + ""table_permissions"": [ + key + for key, value in self.ds.permissions.items() + if value.takes_resource + ], + ""databases"": [k for k in self.ds.databases.keys() if k != ""_internal""], + ""database_with_tables"": [ + { + ""database"": db.name, + ""tables"": await db.table_names(), + } + for db in self.ds.databases.values() + if db.name != ""_internal"" + ], + }, ) async def post(self, request): self.check_permission(request) post = await request.post_vars() + from pprint import pprint + + pprint(post) errors = [] duration = None if post.get(""expire_type""): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1947#issuecomment-1350013016,https://api.github.com/repos/simonw/datasette/issues/1947,1350013016,IC_kwDOBm6k_c5Qd5BY,9599,simonw,2022-12-13T23:16:24Z,2022-12-13T23:17:17Z,OWNER,"Slightly tricky thing here is that it should only show permissions that the user themselves has - on databases and tables that they have permission to access. I have a nasty feeling this may require looping through _everything_ and running every permission check, which could get very expensive if there are plugins involved that do their own storage check to resolve a permission. It's that classic permission system problem: how to efficiently iterate through everything the user has permission to do in one go? Might be that I have to punt on that, and show the user a list of permissions to select that they might not actually have ability for.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1947#issuecomment-1350008636,https://api.github.com/repos/simonw/datasette/issues/1947,1350008636,IC_kwDOBm6k_c5Qd388,9599,simonw,2022-12-13T23:14:33Z,2022-12-13T23:14:33Z,OWNER,"Checkbox interface looks like this. It's not beautiful but it's good enough for the moment: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1947#issuecomment-1350002434,https://api.github.com/repos/simonw/datasette/issues/1947,1350002434,IC_kwDOBm6k_c5Qd2cC,9599,simonw,2022-12-13T23:11:50Z,2022-12-13T23:11:59Z,OWNER,"I think checkboxes will work well. Here's the data I get back from them (as `post_vars()`): ``` {'all:debug-menu': 'on', 'all:insert-row': 'on', 'expire_duration': '', 'expire_type': '', 'table:fixtures:delete-row': 'on', 'table:fixtures:drop-table': 'on', 'table:fixtures:view-query': 'on'} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1493390939,UI to create reduced scope tokens from the `/-/create-token` page, https://github.com/simonw/datasette/issues/1947#issuecomment-1349975255,https://api.github.com/repos/simonw/datasette/issues/1947,1349975255,IC_kwDOBm6k_c5QdvzX,9599,simonw,2022-12-13T23:00:11Z,2022-12-13T23:00:11Z,OWNER,"My `

Restrict actions that can be performed using this token

Restrict actions that can be performed using this token:

All databases and tables:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1940 +/- ## ========================================== + Coverage 92.00% 92.03% +0.02% ========================================== Files 38 38 Lines 5378 5396 +18 ========================================== + Hits 4948 4966 +18 Misses 430 430 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1940?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/permissions.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3Blcm1pc3Npb25zLnB5) | `100.00% <ø> (ø)` | | | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.26% <ø> (ø)` | | | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <ø> (ø)` | | | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `79.20% <ø> (-0.21%)` | :arrow_down: | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `92.57% <ø> (ø)` | | | [datasette/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.47% <100.00%> (+0.04%)` | :arrow_up: | | [datasette/default\_permissions.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `95.20% <100.00%> (+0.39%)` | :arrow_up: | | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1940/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

@pytest.mark.parametrize( ""allow,expected_anon,expected_auth"", [ (None, 200, 200), ({}, 403, 403), ({""id"": ""root""}, 403, 200), ], ) @pytest.mark.parametrize( ""path"", ( ""/"", ""/fixtures"", ""/fixtures/compound_three_primary_keys"", ""/fixtures/compound_three_primary_keys/a,a,a"", ""/fixtures/two"", # Query ), ) def test_view_padlock(allow, expected_anon, expected_auth, path, padlock_client): padlock_client.ds._metadata_local[""allow""] = allow fragment = ""🔒"" anon_response = padlock_client.get(path) > assert expected_anon == anon_response.status E assert 403 == 200 E + where 200 = .status /Users/simon/Dropbox/Development/datasette/tests/test_permissions.py:61: AssertionError >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > /Users/simon/Dropbox/Development/datasette/tests/test_permissions.py(61)test_view_padlock() -> assert expected_anon == anon_response.status (Pdb) anon_response (Pdb) anon_response.status 200 (Pdb) path '/fixtures/compound_three_primary_keys' (Pdb) padlock_client.ds._metadata_ *** AttributeError: 'Datasette' object has no attribute '_metadata_' (Pdb) padlock_client.ds._metadata_local {'databases': {'fixtures': {'queries': {'two': {'sql': 'select 1 + 1', 'name': 'two'}, 'from_async_hook': {'sql': 'select 2', 'name': 'from_async_hook'}, 'from_hook': {'sql': ""select 1, 'null' as actor_id"", 'name': 'from_hook'}}, 'source': None, 'source_url': None, 'license': None, 'license_url': None, 'about': None, 'about_url': None}}, 'allow': {}} (Pdb) allow {} ``` It looks like I've broken the `allow` logic that notices that if there's an `""allow"": {}` on the root then anonymous users should not be allowed to view any pages.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1486011362,register_permissions() plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1347646516,https://api.github.com/repos/simonw/datasette/issues/1939,1347646516,IC_kwDOBm6k_c5QU3Q0,9599,simonw,2022-12-13T02:07:50Z,2022-12-13T02:07:50Z,OWNER,Documentation for the new hook: https://docs.datasette.io/en/latest/plugin_hooks.html#register-permissions-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1345691103,https://api.github.com/repos/simonw/datasette/issues/1939,1345691103,IC_kwDOBm6k_c5QNZ3f,9599,simonw,2022-12-11T23:37:49Z,2022-12-11T23:37:49Z,OWNER,"Idea: a `/-/permissions` introspection endpoint for listing registered permissions ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343872168,https://api.github.com/repos/simonw/datasette/issues/1939,1343872168,IC_kwDOBm6k_c5QGdyo,9599,simonw,2022-12-09T05:29:53Z,2022-12-09T05:29:53Z,OWNER,"I'm going to address those ideas for changes to the `permission_allowed()` in a separate issue. What would it take for the `register_permissions()` hook to be something I'm comfortable landing? I think it's mainly that the list of permissions it provides should Do More Stuff: - Participate in unit tests, in particular this one: https://github.com/simonw/datasette/blob/e539c1c024bc62d88df91d9107cbe37e7f0fe55f/tests/conftest.py#L79-L102 - That new `default` option should be respected - maybe if you omit `default=` from a call to `permission_allowed()` it could fall back on the default from there? - Log a warning if you attempt to check a permission that wasn't registered Then I can use the permissions - in particular their metadata - to help implement his: - #1636","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343858998,https://api.github.com/repos/simonw/datasette/issues/1939,1343858998,IC_kwDOBm6k_c5QGak2,9599,simonw,2022-12-09T05:12:17Z,2022-12-09T05:12:17Z,OWNER,Draft docs for the new plugin hook: https://datasette--1940.org.readthedocs.build/en/1940/plugin_hooks.html#register-permissions-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343856781,https://api.github.com/repos/simonw/datasette/issues/1939,1343856781,IC_kwDOBm6k_c5QGaCN,9599,simonw,2022-12-09T05:10:00Z,2022-12-09T05:10:00Z,OWNER,Made a draft PR so ReadTheDocs would deploy my new documentation somewhere.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343844555,https://api.github.com/repos/simonw/datasette/issues/1939,1343844555,IC_kwDOBm6k_c5QGXDL,9599,simonw,2022-12-09T04:48:28Z,2022-12-09T04:48:28Z,OWNER,"I'm going to try a spike in a branch with `datasette.action_allowed(...)` and a `register_permissions()` plugin hook, to see what they look like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343844112,https://api.github.com/repos/simonw/datasette/issues/1939,1343844112,IC_kwDOBm6k_c5QGW8Q,9599,simonw,2022-12-09T04:47:28Z,2022-12-09T04:47:28Z,OWNER,"I think `action_allowed` is my favourite, even though there's a little bit of concept overlap with `table_actions` and `database_actions`. I never really liked those plugin hook names much to be honest, especially since they are inconsistent with `menu_links`: https://github.com/simonw/datasette/blob/d67f812b7327c7075732688f3df728807503dc58/datasette/hookspecs.py#L123-L135","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343843352,https://api.github.com/repos/simonw/datasette/issues/1939,1343843352,IC_kwDOBm6k_c5QGWwY,9599,simonw,2022-12-09T04:45:50Z,2022-12-09T04:45:50Z,OWNER,"Another option: ```python if await datasette.actor_can(actor, ""insert-data""...) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343842362,https://api.github.com/repos/simonw/datasette/issues/1939,1343842362,IC_kwDOBm6k_c5QGWg6,9599,simonw,2022-12-09T04:43:38Z,2022-12-09T04:43:38Z,OWNER,"Asked ChatGPT for some alternative names, I didn't like any of them: is_permission_granted has_permission check_permission is_action_allowed check_access_permission permission_check validate_permission check_actor_permission verify_permission check_authorization ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343753386,https://api.github.com/repos/simonw/datasette/issues/1939,1343753386,IC_kwDOBm6k_c5QGAyq,9599,simonw,2022-12-09T02:20:20Z,2022-12-09T02:21:01Z,OWNER,"It's also referenced in this plugin hook: ```python @hookspec def permission_allowed(datasette, actor, action, resource): """"""Check if actor is allowed to perform this action - return True, False or None"""""" ``` But more importantly, in these ones: ```python @hookspec def table_actions(datasette, actor, database, table, request): """"""Links for the table actions menu"""""" @hookspec def database_actions(datasette, actor, database, request): """"""Links for the database actions menu"""""" ``` So the word ""action"" is already used within Datasette to refer to those things - which are _almost_ but not quite the same as actions-as-permissions: many of the things that show up in those menus relate to permissions the user has, but not necessarily all of them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343751860,https://api.github.com/repos/simonw/datasette/issues/1939,1343751860,IC_kwDOBm6k_c5QGAa0,9599,simonw,2022-12-09T02:18:11Z,2022-12-09T02:19:23Z,OWNER,"Should I rename ""permission"" to ""action"" elsewhere too? Maybe have a `register_actions(...)` plugin hook instead of adding `register_permissions(...)`? What else could the word ""action"" mean? Currently it's used in the codebase to refer to GitHub Actions, and for code like this: ```python if await self.permission_allowed( actor=actor, action=""view-instance"", default=True ): ``` Which is already revealing the confusion between ""permission"" and ""action"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343751261,https://api.github.com/repos/simonw/datasette/issues/1939,1343751261,IC_kwDOBm6k_c5QGARd,9599,simonw,2022-12-09T02:17:14Z,2022-12-09T02:17:14Z,OWNER,"One option: ```python async def action_allowed(self, actor, action, database=None, resource=None): ``` `action_allowed` fixes the `permission` v.s. `action` thing a bit, and is a new name that doesn't clash with the existing method. I dropped `default` because that's now a property of the permission itself. `table` is now called `resource` and `database` is a separate parameter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343749617,https://api.github.com/repos/simonw/datasette/issues/1939,1343749617,IC_kwDOBm6k_c5QF_3x,9599,simonw,2022-12-09T02:15:54Z,2022-12-09T02:15:54Z,OWNER,"What if I came up with a new method name for this, which could co-exist with the old one while that old one was deprecated?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343744338,https://api.github.com/repos/simonw/datasette/issues/1939,1343744338,IC_kwDOBm6k_c5QF-lS,9599,simonw,2022-12-09T02:08:42Z,2022-12-09T02:08:42Z,OWNER,Extracted a TIL: https://til.simonwillison.net/github/github-code-search-api-uses,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343734812,https://api.github.com/repos/simonw/datasette/issues/1939,1343734812,IC_kwDOBm6k_c5QF8Qc,9599,simonw,2022-12-09T01:57:07Z,2022-12-09T01:57:07Z,OWNER,"This search is better: datasette permission_allowed -user:simonw -path:datasette/** -path:docs/** -path:tests/** language:python That returns 11 results: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+permission_allowed+-user%3Asimonw+-path%3Adatasette%2F**+-path%3Adocs%2F**+-path%3Atests%2F**+language%3Apython 3 are forks of my repos. The rest are all by four users: - [20after4/ddd](https://github.com/20after4/ddd) - [emg110/datasette-graphql](https://github.com/emg110/datasette-graphql) - [next-LI/datasette-csv-importer](https://github.com/next-LI/datasette-csv-importer) - [next-LI/datasette-demo](https://github.com/next-LI/datasette-demo) - [next-LI/datasette-live-config](https://github.com/next-LI/datasette-live-config) - [next-LI/datasette-live-permissions](https://github.com/next-LI/datasette-live-permissions) - [next-LI/datasette-search-all](https://github.com/next-LI/datasette-search-all) - [next-LI/datasette-surveys](https://github.com/next-LI/datasette-surveys) - [next-LI/datasette-write](https://github.com/next-LI/datasette-write) - [rclement/datasette-dashboards](https://github.com/rclement/datasette-dashboards) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343728929,https://api.github.com/repos/simonw/datasette/issues/1939,1343728929,IC_kwDOBm6k_c5QF60h,9599,simonw,2022-12-09T01:48:11Z,2022-12-09T01:52:33Z,OWNER,"This code search shows a bunch of repos I don't know about that would be affected by this change: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+permission_allowed+-user%3Asimonw# These (and likely more): Repositories - [20after4/ddd](https://github.com/20after4/ddd) - [next-LI/datasette-csv-importer](https://github.com/next-LI/datasette-csv-importer) - [digital-land/datasette](https://github.com/digital-land/datasette) - [mroswell/datasette](https://github.com/mroswell/datasette) - [next-LI/datasette-live-config](https://github.com/next-LI/datasette-live-config) - [keladhruv/datasette](https://github.com/keladhruv/datasette) - [RhetTbull/datasette](https://github.com/RhetTbull/datasette) - [chriswedgwood/datasette](https://github.com/chriswedgwood/datasette) - [boan-anbo/datasette](https://github.com/boan-anbo/datasette) - [MattTriano/datasette](https://github.com/MattTriano/datasette) - [incadenza/datasette](https://github.com/incadenza/datasette) - [robdyke/datasette](https://github.com/robdyke/datasette) - [ctb/datasette](https://github.com/ctb/datasette) - [eyeseast/datasette](https://github.com/eyeseast/datasette) - [symbol-management/api-match-audit](https://github.com/symbol-management/api-match-audit) Actually a lot of those are forks of Datasette itself - so maybe this is manageable? Would be nice if I could come up with a GitHub search that excluded any repos with ""datasette"" as their exact name. https://docs.github.com/en/search-github/github-code-search/understanding-github-code-search-syntax#using-qualifiers says: > **Note:** The new code search beta does not currently support regular expressions or partial matching for repository names, so you will have to type the entire repository name (including the user prefix) for the `repo:` qualifier to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343727184,https://api.github.com/repos/simonw/datasette/issues/1939,1343727184,IC_kwDOBm6k_c5QF6ZQ,9599,simonw,2022-12-09T01:45:15Z,2022-12-09T01:45:15Z,OWNER,"Moving the concept of the default for the permission into this registry warrants a redesign of this method anyway: https://github.com/simonw/datasette/blob/e539c1c024bc62d88df91d9107cbe37e7f0fe55f/datasette/app.py#L706","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343724732,https://api.github.com/repos/simonw/datasette/issues/1939,1343724732,IC_kwDOBm6k_c5QF5y8,9599,simonw,2022-12-09T01:40:44Z,2022-12-09T01:43:25Z,OWNER,"```python Permission = collections.namedtuple( ""Permission"", (""name"", ""abbr"", ""takes_database"", ""takes_table"", ""default"") ) ``` I don`t think that design is quite right. - Elsewhere in the code the concept is called an ""action"" rather than a ""permission"" - I think I can stick with the `Permission` name here though, it's pretty clear - `takes_database` - is `takes_` the right verb here? - `takes_table` can also refer to a SQL view or a canned named query A question that was raised by the work in #1938 is whether you should be able to grant a permission like `insert-row` at the instance or database level - and if so, what does that look like? I think you should be able to do that, it doesn't make sense to have to grant it explicitly for every single table. So maybe `takes_table` and `takes_database` are the right names here? But `table` is still bad because it doesn't reflect views and canned queries. One thought is to use `resource` - but that will require a bunch of breaking changes to the existing APIs which treat resource as a tuple. Now's the best time to do that though before Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343722020,https://api.github.com/repos/simonw/datasette/issues/1939,1343722020,IC_kwDOBm6k_c5QF5Ik,9599,simonw,2022-12-09T01:36:05Z,2022-12-09T01:36:16Z,OWNER,"I originally added `permissions.py` for the permission debug tool in https://github.com/simonw/datasette/commit/c51d9246b996a2831c9bd6a1e205f6cb48b9a5f3 - I don't think anything else uses it yet. - #1881","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/issues/1939#issuecomment-1343721522,https://api.github.com/repos/simonw/datasette/issues/1939,1343721522,IC_kwDOBm6k_c5QF5Ay,9599,simonw,2022-12-09T01:35:15Z,2022-12-09T01:35:15Z,OWNER,"One concern I have about this: there are a bunch of existing plugins that do stuff with permissions that won't currently be using this hook. Do I break those plugins, forcing new releases of them for compatibility with Datasette 1.0? Or maybe I keep them working, but until they've upgraded to register their permissions there are things about them that won't work - e.g. you won't be able to configure their permissions in `metadata.yml` until they release something that does this hook. Best thing is probably for me to get this working in core first and then evaluate the impact it would have on existing plugins once I have some running code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511,register_permissions(datasette) plugin hook, https://github.com/simonw/datasette/pull/1938#issuecomment-1347767048,https://api.github.com/repos/simonw/datasette/issues/1938,1347767048,IC_kwDOBm6k_c5QVUsI,9599,simonw,2022-12-13T05:23:18Z,2022-12-13T05:23:18Z,OWNER,"I landed this already: - #1636 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485488236,"""permissions"" blocks in metadata.json/yaml", https://github.com/simonw/datasette/pull/1938#issuecomment-1343449918,https://api.github.com/repos/simonw/datasette/issues/1938,1343449918,IC_kwDOBm6k_c5QE2s-,22429695,codecov[bot],2022-12-08T22:20:10Z,2022-12-08T22:54:08Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1938?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.00**% // Head: **92.01**% // Increases project coverage by **`+0.01%`** :tada: > Coverage data is based on head [(`6e35a6b`)](https://codecov.io/gh/simonw/datasette/pull/1938?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`e539c1c`)](https://codecov.io/gh/simonw/datasette/commit/e539c1c024bc62d88df91d9107cbe37e7f0fe55f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1938 +/- ## ========================================== + Coverage 92.00% 92.01% +0.01% ========================================== Files 38 38 Lines 5378 5386 +8 ========================================== + Hits 4948 4956 +8 Misses 430 430 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1938?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/default\_permissions.py](https://codecov.io/gh/simonw/datasette/pull/1938/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `95.10% <100.00%> (+0.29%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1938?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485488236,"""permissions"" blocks in metadata.json/yaml", https://github.com/simonw/datasette/pull/1938#issuecomment-1343445885,https://api.github.com/repos/simonw/datasette/issues/1938,1343445885,IC_kwDOBm6k_c5QE1t9,9599,simonw,2022-12-08T22:16:03Z,2022-12-08T22:16:03Z,OWNER,Docs: https://datasette--1938.org.readthedocs.build/en/1938/authentication.html#other-permissions-in-metadata,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485488236,"""permissions"" blocks in metadata.json/yaml", https://github.com/simonw/datasette/issues/1937#issuecomment-1347770871,https://api.github.com/repos/simonw/datasette/issues/1937,1347770871,IC_kwDOBm6k_c5QVVn3,9599,simonw,2022-12-13T05:30:43Z,2022-12-13T05:30:43Z,OWNER,"Also you should need `update-row` permission to use the `""replace"": true` option - I should add that rule to `/-/insert` add well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1483320357,/db/-/create API should require insert-rows permission to use row: or rows: option, https://github.com/simonw/datasette/issues/1937#issuecomment-1341848525,https://api.github.com/repos/simonw/datasette/issues/1937,1341848525,IC_kwDOBm6k_c5P-vvN,9599,simonw,2022-12-08T01:34:03Z,2022-12-08T01:34:03Z,OWNER,Check should go somewhere around here: https://github.com/simonw/datasette/blob/dee18ed8ce7af2ab8699bcb5a51a99f48301bc42/datasette/views/database.py#L625-L631,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1483320357,/db/-/create API should require insert-rows permission to use row: or rows: option, https://github.com/simonw/datasette/issues/1936#issuecomment-1341849735,https://api.github.com/repos/simonw/datasette/issues/1936,1341849735,IC_kwDOBm6k_c5P-wCH,9599,simonw,2022-12-08T01:35:54Z,2022-12-08T01:35:54Z,OWNER,"Running that twice produced this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1483250004,Fix /db/table/-/upsert in the API explorer, https://github.com/simonw/datasette/issues/1936#issuecomment-1341849496,https://api.github.com/repos/simonw/datasette/issues/1936,1341849496,IC_kwDOBm6k_c5P-v-Y,9599,simonw,2022-12-08T01:35:35Z,2022-12-08T01:35:35Z,OWNER,"Related bug: you can send `""id"": null` and it works (it should throw an error): ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1483250004,Fix /db/table/-/upsert in the API explorer, https://github.com/simonw/datasette/pull/1935#issuecomment-1340950566,https://api.github.com/repos/simonw/datasette/issues/1935,1340950566,IC_kwDOBm6k_c5P7Ugm,22429695,codecov[bot],2022-12-07T13:14:41Z,2022-12-07T13:14:41Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1935?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **91.73**% // Head: **91.49**% // Decreases project coverage by **`-0.24%`** :warning: > Coverage data is based on head [(`e8ae41e`)](https://codecov.io/gh/simonw/datasette/pull/1935?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`93ababe`)](https://codecov.io/gh/simonw/datasette/commit/93ababe6f7150454d2cf278dae08569e505d2a5b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch has no changes to coverable lines.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1935 +/- ## ========================================== - Coverage 91.73% 91.49% -0.25% ========================================== Files 36 37 +1 Lines 4987 5031 +44 ========================================== + Hits 4575 4603 +28 - Misses 412 428 +16 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1935?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/utils/shutil\_backport.py](https://codecov.io/gh/simonw/datasette/pull/1935/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3NodXRpbF9iYWNrcG9ydC5weQ==) | `9.09% <0.00%> (ø)` | | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1935/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.78% <0.00%> (+0.36%)` | :arrow_up: | | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/1935/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `85.29% <0.00%> (+2.94%)` | :arrow_up: | | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1935/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `93.60% <0.00%> (+3.59%)` | :arrow_up: | | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1935/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.18% <0.00%> (+4.00%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1935?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1481875485,Bump furo from 2022.9.29 to 2022.12.7, https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339844639,https://api.github.com/repos/simonw/sqlite-utils/issues/516,1339844639,IC_kwDOCGYnMM5P3Ggf,122043,briandorsey,2022-12-06T19:08:13Z,2022-12-06T19:08:13Z,NONE,"Reference: tqdm (https://tqdm.github.io/) shows a progress bar when total is known, and falls back to counting units of work done for streams. File input vs. stdin seems like a similar situation. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1479914599,Feature request: output number of ignored/replaced rows for insert command, https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339839767,https://api.github.com/repos/simonw/sqlite-utils/issues/516,1339839767,IC_kwDOCGYnMM5P3FUX,122043,briandorsey,2022-12-06T19:04:17Z,2022-12-06T19:04:17Z,NONE,"Current behavior is different when importing via stdin vs. a file. Imports from a file give a progress bar. For this new request, I'd love to see total imported and total ignored/replaced in both cases. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1479914599,Feature request: output number of ignored/replaced rows for insert command, https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339837520,https://api.github.com/repos/simonw/sqlite-utils/issues/516,1339837520,IC_kwDOCGYnMM5P3ExQ,122043,briandorsey,2022-12-06T19:02:30Z,2022-12-06T19:02:30Z,NONE,"`--verbose` or `--verbosity=ABC` were the flags I looked for. Expected to see them at a global level near `--version`. But only sharing because that's where I looked first, I don't have a strong opinion on the exact wording/location. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1479914599,Feature request: output number of ignored/replaced rows for insert command, https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339834918,https://api.github.com/repos/simonw/sqlite-utils/issues/516,1339834918,IC_kwDOCGYnMM5P3EIm,9599,simonw,2022-12-06T19:00:18Z,2022-12-06T19:00:35Z,OWNER,"Right now the command produces no output at all. Maybe a `--verbose` mode that writes these numbers to standard error (or even standard output since it's an option)? Is there a better name than `--verbose` for this? `--summary` perhaps?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1479914599,Feature request: output number of ignored/replaced rows for insert command, https://github.com/simonw/datasette/pull/1931#issuecomment-1341825314,https://api.github.com/repos/simonw/datasette/issues/1931,1341825314,IC_kwDOBm6k_c5P-qEi,9599,simonw,2022-12-08T01:03:18Z,2022-12-08T01:03:18Z,OWNER,"I broke this test: ``` ds_write = @pytest.mark.asyncio async def test_insert_row(ds_write): token = write_token(ds_write) response = await ds_write.client.post( ""/data/docs/-/insert"", json={""row"": {""title"": ""Test"", ""score"": 1.2, ""age"": 5}}, headers={ ""Authorization"": ""***"".format(token), ""Content-Type"": ""application/json"", }, ) expected_row = {""id"": 1, ""title"": ""Test"", ""score"": 1.2, ""age"": 5} > assert response.status_code == 201 E assert 500 == 201 E + where 500 = .status_code /home/runner/work/datasette/datasette/tests/test_api_write.py:43: AssertionError ----------------------------- Captured stderr call ----------------------------- Traceback (most recent call last): File ""/home/runner/work/datasette/datasette/datasette/app.py"", line 1447, in route_path response = await view(request, send) File ""/home/runner/work/datasette/datasette/datasette/views/base.py"", line 151, in view return await self.dispatch_request(request) File ""/home/runner/work/datasette/datasette/datasette/views/base.py"", line 105, in dispatch_request response = await handler(request) File ""/home/runner/work/datasette/datasette/datasette/views/table.py"", line 1228, in post row_pk_values_for_later = [tuple(row[pk] for pk in pks) for row in rows] File ""/home/runner/work/datasette/datasette/datasette/views/table.py"", line 1228, in row_pk_values_for_later = [tuple(row[pk] for pk in pks) for row in rows] File ""/home/runner/work/datasette/datasette/datasette/views/table.py"", line 1228, in row_pk_values_for_later = [tuple(row[pk] for pk in pks) for row in rows] KeyError: 'id' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/pull/1931#issuecomment-1341821213,https://api.github.com/repos/simonw/datasette/issues/1931,1341821213,IC_kwDOBm6k_c5P-pEd,9599,simonw,2022-12-08T00:58:21Z,2022-12-08T00:58:21Z,OWNER,"In the interests of shipping, I'm going to punt the API explorer to a later issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/pull/1931#issuecomment-1339968514,https://api.github.com/repos/simonw/datasette/issues/1931,1339968514,IC_kwDOBm6k_c5P3kwC,9599,simonw,2022-12-06T20:28:47Z,2022-12-06T20:28:47Z,OWNER,"Should the `""return"": true` mode reflect the order in which the rows were provided when the API was called? I think it should. Since this is small enough to happily fit in Python memory (thanks to the `max_insert_rows` setting) I can load the fresh data from the database and then sort it in Python space before returning it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/pull/1931#issuecomment-1339916064,https://api.github.com/repos/simonw/datasette/issues/1931,1339916064,IC_kwDOBm6k_c5P3X8g,3556,davidbgk,2022-12-06T19:42:45Z,2022-12-06T19:42:45Z,CONTRIBUTOR,"The `""return"": true` option is really nice!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/pull/1931#issuecomment-1339911152,https://api.github.com/repos/simonw/datasette/issues/1931,1339911152,IC_kwDOBm6k_c5P3Wvw,9599,simonw,2022-12-06T19:38:12Z,2022-12-06T19:38:12Z,OWNER,Documentation: https://datasette--1931.org.readthedocs.build/en/1931/json_api.html#upserting-rows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/pull/1931#issuecomment-1339906241,https://api.github.com/repos/simonw/datasette/issues/1931,1339906241,IC_kwDOBm6k_c5P3VjB,22429695,codecov[bot],2022-12-06T19:33:32Z,2022-12-08T01:04:56Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1931?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **90.42**% // Head: **91.77**% // Increases project coverage by **`+1.34%`** :tada: > Coverage data is based on head [(`645be0f`)](https://codecov.io/gh/simonw/datasette/pull/1931?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`cab5b60`)](https://codecov.io/gh/simonw/datasette/commit/cab5b60e09e94aca820dbec5308446a88c99ea3d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 95.55% of modified lines in pull request are covered. > :exclamation: Current head 645be0f differs from pull request most recent head 7cd6fd9. Consider uploading reports for the commit 7cd6fd9 to get more accurate results

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1931 +/- ## ========================================== + Coverage 90.42% 91.77% +1.34% ========================================== Files 36 36 Lines 5057 5019 -38 ========================================== + Hits 4573 4606 +33 + Misses 484 413 -71 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1931?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/1931/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `79.41% <0.00%> (ø)` | | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1931/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `92.44% <97.43%> (+0.20%)` | :arrow_up: | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1931/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.42% <100.00%> (+<0.01%)` | :arrow_up: | | [datasette/default\_permissions.py](https://codecov.io/gh/simonw/datasette/pull/1931/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `94.81% <100.00%> (+0.07%)` | :arrow_up: | | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1931/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.83% <0.00%> (+17.01%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1931?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/pull/1931#issuecomment-1339784569,https://api.github.com/repos/simonw/datasette/issues/1931,1339784569,IC_kwDOBm6k_c5P2315,9599,simonw,2022-12-06T18:16:15Z,2022-12-06T18:17:56Z,OWNER,"Just noticed the insert API returns `{}` when it should return `{""ok"": true}` - will fix that here too. UPDATE: no it did that already, it was just the documentation that was wrong.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/pull/1931#issuecomment-1339768422,https://api.github.com/repos/simonw/datasette/issues/1931,1339768422,IC_kwDOBm6k_c5P2z5m,9599,simonw,2022-12-06T18:04:59Z,2022-12-06T18:04:59Z,OWNER,"I realized this API should require both the `insert-row` AND the `update-row` permissions, since calls to it could do either one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539,/db/table/-/upsert, https://github.com/simonw/datasette/pull/1930#issuecomment-1343360006,https://api.github.com/repos/simonw/datasette/issues/1930,1343360006,IC_kwDOBm6k_c5QEgwG,9599,simonw,2022-12-08T21:12:28Z,2022-12-08T21:12:28Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473664029,Typo in JSON API `Updating a row` documentation, https://github.com/simonw/datasette/pull/1930#issuecomment-1336017976,https://api.github.com/repos/simonw/datasette/issues/1930,1336017976,IC_kwDOBm6k_c5PogQ4,22429695,codecov[bot],2022-12-03T02:30:21Z,2022-12-03T02:30:21Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1930?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **90.42**% // Head: **90.42**% // No change to project coverage :thumbsup: > Coverage data is based on head [(`9928ff1`)](https://codecov.io/gh/simonw/datasette/pull/1930?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`cab5b60`)](https://codecov.io/gh/simonw/datasette/commit/cab5b60e09e94aca820dbec5308446a88c99ea3d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch has no changes to coverable lines.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1930 +/- ## ======================================= Coverage 90.42% 90.42% ======================================= Files 36 36 Lines 5057 5057 ======================================= Hits 4573 4573 Misses 484 484 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1930?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473664029,Typo in JSON API `Updating a row` documentation, https://github.com/simonw/datasette/issues/1929#issuecomment-1339909159,https://api.github.com/repos/simonw/datasette/issues/1929,1339909159,IC_kwDOBm6k_c5P3WQn,9599,simonw,2022-12-06T19:36:23Z,2022-12-06T19:36:23Z,OWNER,https://docs.datasette.io/en/1.0a1/json_api.html 👍 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473659191,Incorrect link from the API explorer to the JSON API documentation, https://github.com/simonw/datasette/issues/1929#issuecomment-1339906969,https://api.github.com/repos/simonw/datasette/issues/1929,1339906969,IC_kwDOBm6k_c5P3VuZ,3556,davidbgk,2022-12-06T19:34:20Z,2022-12-06T19:34:20Z,CONTRIBUTOR,I confirm that it works 👍 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473659191,Incorrect link from the API explorer to the JSON API documentation, https://github.com/simonw/datasette/issues/1929#issuecomment-1339871933,https://api.github.com/repos/simonw/datasette/issues/1929,1339871933,IC_kwDOBm6k_c5P3NK9,9599,simonw,2022-12-06T19:23:48Z,2022-12-06T19:24:17Z,OWNER,"I can do that on this page: https://readthedocs.org/projects/datasette/versions/?version_filter=1.0 I'm making them both active and hidden: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473659191,Incorrect link from the API explorer to the JSON API documentation, https://github.com/simonw/datasette/issues/1929#issuecomment-1339867570,https://api.github.com/repos/simonw/datasette/issues/1929,1339867570,IC_kwDOBm6k_c5P3MGy,9599,simonw,2022-12-06T19:22:47Z,2022-12-06T19:22:47Z,OWNER,"Yeah I should deploy the docs for the alpha versions (the intention is that the docs exactly match the release you are using), thanks for spotting that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473659191,Incorrect link from the API explorer to the JSON API documentation, https://github.com/simonw/datasette/issues/1928#issuecomment-1335966329,https://api.github.com/repos/simonw/datasette/issues/1928,1335966329,IC_kwDOBm6k_c5PoTp5,9599,simonw,2022-12-02T23:47:11Z,2022-12-02T23:47:11Z,OWNER,Wrote about this extensively here: https://simonwillison.net/2022/Dec/2/datasette-write-api/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870889,https://api.github.com/repos/simonw/datasette/issues/1928,1335870889,IC_kwDOBm6k_c5Pn8Wp,9599,simonw,2022-12-02T21:41:09Z,2022-12-02T21:41:09Z,OWNER,"Got it working! https://simon.datasette.cloud/data/hacker_news_posts https://github.com/simonw/scrape-hacker-news-by-domain/blob/main/submit-to-datasette-cloud.sh","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870887,https://api.github.com/repos/simonw/datasette/issues/1928,1335870887,IC_kwDOBm6k_c5Pn8Wn,9599,simonw,2022-12-02T21:25:16Z,2022-12-02T21:25:16Z,OWNER,"Here's the change that should submit data to Datasette Cloud: https://github.com/simonw/scrape-hacker-news-by-domain/commit/848bb7e835a9fb87cd656362591835179cd1dc1b I ran `delete from hacker_news_posts` against my instance so https://simon.datasette.cloud/data/hacker_news_posts is now empty.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870884,https://api.github.com/repos/simonw/datasette/issues/1928,1335870884,IC_kwDOBm6k_c5Pn8Wk,9599,simonw,2022-12-02T21:19:58Z,2022-12-02T21:19:58Z,OWNER,"But until I fix this issue: - https://github.com/simonw/datasette/issues/1927 I need to insert freshly scraped data like this: ```bash export ROWS=$( jq -n --argjson rows ""$(cat simonwillison-net.json)"" \ '{ ""rows"": $rows, ""replace"": true }' ) curl -X POST \ https://simon.datasette.cloud/data/hacker_news_posts/-/insert \ -H ""Content-Type: application/json"" \ -H ""Authorization: Bearer $DS_TOKEN"" \ -d $ROWS ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870883,https://api.github.com/repos/simonw/datasette/issues/1928,1335870883,IC_kwDOBm6k_c5Pn8Wj,9599,simonw,2022-12-02T21:19:10Z,2022-12-02T21:19:10Z,OWNER,"I created the `hacker_news_posts` table like this: ```bash export ROWS=$( jq -n --argjson rows ""$(cat simonwillison-net.json)"" \ '{ ""table"": ""hacker_news_posts"", ""rows"": $rows, ""pk"": ""id"", ""replace"": true }' ) # Use curl to POST some JSON to a URL curl -X POST \ https://simon.datasette.cloud/data/-/create \ -H ""Content-Type: application/json"" \ -H ""Authorization: Bearer $DS_TOKEN"" \ -d $ROWS ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870879,https://api.github.com/repos/simonw/datasette/issues/1928,1335870879,IC_kwDOBm6k_c5Pn8Wf,9599,simonw,2022-12-02T21:18:25Z,2022-12-02T21:18:25Z,OWNER,"This is the SQL view for the atom feed: ```sql CREATE VIEW hacker_news_posts_atom as select id as atom_id, title as atom_title, url, commentsUrl as atom_link, dt || 'Z' as atom_updated, 'Submitter: ' || submitter || ' - ' || points || ' points, ' || numComments || ' comments' as atom_content from hacker_news_posts order by dt desc limit 100; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870877,https://api.github.com/repos/simonw/datasette/issues/1928,1335870877,IC_kwDOBm6k_c5Pn8Wd,9599,simonw,2022-12-02T21:17:50Z,2022-12-02T21:17:50Z,OWNER,https://simon.datasette.cloud/data/hacker_news_posts_atom.atom is working now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262,Hacker News Datasette write demo, https://github.com/simonw/datasette/issues/1927#issuecomment-1339910494,https://api.github.com/repos/simonw/datasette/issues/1927,1339910494,IC_kwDOBm6k_c5P3Wle,9599,simonw,2022-12-06T19:37:39Z,2022-12-06T19:37:39Z,OWNER,"I'll finish this after I land: - #1931 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197,ignore:true/replace:true options for /db/-/create API, https://github.com/simonw/datasette/issues/1927#issuecomment-1336099588,https://api.github.com/repos/simonw/datasette/issues/1927,1336099588,IC_kwDOBm6k_c5Po0ME,9599,simonw,2022-12-03T06:58:14Z,2022-12-03T06:58:14Z,OWNER,I have not yet documented the new `insert` and `replace` options.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197,ignore:true/replace:true options for /db/-/create API, https://github.com/simonw/datasette/issues/1927#issuecomment-1336099533,https://api.github.com/repos/simonw/datasette/issues/1927,1336099533,IC_kwDOBm6k_c5Po0LN,9599,simonw,2022-12-03T06:57:52Z,2022-12-03T06:57:52Z,OWNER,I'm going to push what I have anyway. I'll keep this issue open while I think through the above comment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197,ignore:true/replace:true options for /db/-/create API, https://github.com/simonw/datasette/issues/1927#issuecomment-1336099368,https://api.github.com/repos/simonw/datasette/issues/1927,1336099368,IC_kwDOBm6k_c5Po0Io,9599,simonw,2022-12-03T06:56:36Z,2022-12-03T06:56:36Z,OWNER,"Neither of these options make sense if you didn't pass a `""pk""`. My initial implementation spotted if the `pk` was missing and looked it up from the table, but actually I don't think that makes sense - if you know the table exists and hence don't pass the `pk` you should be using `/-/insert` or `/-/upsert` instead. So maybe this work should expanded to include validation that checks if the table exists already - and if it does, confirms that the primary key (and maybe even the columns) are the same as for that existing table. Of course if you only send `row` or `rows` then checking `columns` doesn't completely make sense - but we could check that the rows you have sent are equal to or a subset of the columns in the table. We could even check the column types as well, as seen in: - #1910 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197,ignore:true/replace:true options for /db/-/create API, https://github.com/simonw/datasette/issues/1927#issuecomment-1335984268,https://api.github.com/repos/simonw/datasette/issues/1927,1335984268,IC_kwDOBm6k_c5PoYCM,9599,simonw,2022-12-03T00:26:26Z,2022-12-03T00:26:26Z,OWNER,"Also: the documentation should clarify that you can call this API multiple times when using the `rows` option. (It will probably grow `""alter"": true` soon too).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197,ignore:true/replace:true options for /db/-/create API, https://github.com/simonw/datasette/issues/1926#issuecomment-1334508062,https://api.github.com/repos/simonw/datasette/issues/1926,1334508062,IC_kwDOBm6k_c5Pivoe,9599,simonw,2022-12-01T22:06:12Z,2022-12-01T22:06:12Z,OWNER,Released: https://pypi.org/project/datasette/1.0a1/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1471969984,Release notes for 1.0a1 (and release it), https://github.com/simonw/datasette/issues/1924#issuecomment-1334165225,https://api.github.com/repos/simonw/datasette/issues/1924,1334165225,IC_kwDOBm6k_c5Phb7p,9599,simonw,2022-12-01T18:15:15Z,2022-12-01T18:15:15Z,OWNER,Updated docs at the bottom of this section: https://docs.datasette.io/en/latest/json_api.html#inserting-rows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470509936,Docs for replace:true and ignore:true options for insert API, https://github.com/simonw/datasette/issues/1924#issuecomment-1333042785,https://api.github.com/repos/simonw/datasette/issues/1924,1333042785,IC_kwDOBm6k_c5PdJ5h,9599,simonw,2022-12-01T02:00:06Z,2022-12-01T02:00:06Z,OWNER,"Looks like I did implement these already: https://github.com/simonw/datasette/blob/9a1536b52a07e32da5900652da1bd7894c58fa9f/tests/test_api_write.py#L270-L273 But forgot to document them!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470509936,Docs for replace:true and ignore:true options for insert API, https://github.com/simonw/datasette/issues/1924#issuecomment-1333025076,https://api.github.com/repos/simonw/datasette/issues/1924,1333025076,IC_kwDOBm6k_c5PdFk0,9599,simonw,2022-12-01T01:37:09Z,2022-12-01T01:37:09Z,OWNER,"Related question: what happens if you attempt to insert rows without either of these settings and 1:10 of them is an integrity error due to an existing primary key? Does the entire batch fail to be inserted? Should it? This may be the point that I need to think hard about how to use transactions here. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470509936,Docs for replace:true and ignore:true options for insert API, https://github.com/simonw/datasette/issues/1924#issuecomment-1333023273,https://api.github.com/repos/simonw/datasette/issues/1924,1333023273,IC_kwDOBm6k_c5PdFIp,9599,simonw,2022-12-01T01:34:48Z,2022-12-01T01:34:48Z,OWNER,"This will enable some very cool Datasette write API demos - for example, Git scrapers that insert-replace the most recent copy of the scraped data to a table somewhere (which can then produce an atom feed with https://datasette.io/plugins/datasette-atom)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470509936,Docs for replace:true and ignore:true options for insert API, https://github.com/simonw/datasette/issues/1923#issuecomment-1332851215,https://api.github.com/repos/simonw/datasette/issues/1923,1332851215,IC_kwDOBm6k_c5PcbIP,9599,simonw,2022-11-30T23:04:56Z,2022-11-30T23:04:56Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470320227,latest.datasette.io Cloud Run deploys failing, https://github.com/simonw/datasette/issues/1923#issuecomment-1332842435,https://api.github.com/repos/simonw/datasette/issues/1923,1332842435,IC_kwDOBm6k_c5PcY_D,9599,simonw,2022-11-30T22:58:33Z,2022-11-30T22:58:33Z,OWNER,https://stackoverflow.com/questions/74490465/github-actions-failing-for-setup-gcloud/74562740#74562740 suggests trying Python 3.9.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470320227,latest.datasette.io Cloud Run deploys failing, https://github.com/simonw/datasette/issues/1923#issuecomment-1332835664,https://api.github.com/repos/simonw/datasette/issues/1923,1332835664,IC_kwDOBm6k_c5PcXVQ,9599,simonw,2022-11-30T22:50:10Z,2022-11-30T22:51:25Z,OWNER,https://stackoverflow.com/questions/74490465/github-actions-failing-for-setup-gcloud/74562526#74562526 suggests setting `version: '318.0.0'` of `google-github-actions/setup-gcloud`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470320227,latest.datasette.io Cloud Run deploys failing, https://github.com/simonw/datasette/issues/1922#issuecomment-1332903011,https://api.github.com/repos/simonw/datasette/issues/1922,1332903011,IC_kwDOBm6k_c5Pcnxj,9599,simonw,2022-11-30T23:45:29Z,2022-11-30T23:45:29Z,OWNER,"That worked for the preflight request - got this now: So it looks like error responses (in this case for permission denied) are missing CORS headers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332855687,https://api.github.com/repos/simonw/datasette/issues/1922,1332855687,IC_kwDOBm6k_c5PccOH,9599,simonw,2022-11-30T23:09:31Z,2022-11-30T23:09:31Z,OWNER,"Still getting a CORS header. I tried it in Chrome, which outputs helpful messages to the console: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332698636,https://api.github.com/repos/simonw/datasette/issues/1922,1332698636,IC_kwDOBm6k_c5Pb14M,9599,simonw,2022-11-30T20:25:50Z,2022-11-30T20:25:50Z,OWNER,"I just shipped this: Access-Control-Allow-Methods: GET, POST, HEAD, OPTIONS I'll try this out on `latest.datasette.io` - but I need to research more to check if this is a safe thing to do or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332689547,https://api.github.com/repos/simonw/datasette/issues/1922,1332689547,IC_kwDOBm6k_c5PbzqL,9599,simonw,2022-11-30T20:16:21Z,2022-11-30T20:16:46Z,OWNER,"That notebook: ```javascript viewof token = Inputs.text({ label: ""Your API token"" }) ``` ```javascript viewof createResponse = Inputs.button(""Create table"", { value: null, reduce: async () => { const response = await fetch( ""https://latest.datasette.io/ephemeral/-/create"", { method: ""POST"", mode: ""cors"", headers: { Authorization: `Bearer {$token}`, ""Content-Type"": ""application/json"" }, body: JSON.stringify({ table: ""my_new_table"", row: { task: ""Demonstrate a JSON creation from another domain"" } }) } ); return await response.json(); } }) ``` Based on this tip: https://talk.observablehq.com/t/best-pattern-for-click-here-to-submit-your-results-to-an-api-backend/7353/3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332688245,https://api.github.com/repos/simonw/datasette/issues/1922,1332688245,IC_kwDOBm6k_c5PbzV1,9599,simonw,2022-11-30T20:15:08Z,2022-11-30T20:15:08Z,OWNER,"Still getting a CORS error: My hunch is this is because I'm not sending `Access-Control-Allow-Methods: GET,HEAD,POST`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332585861,https://api.github.com/repos/simonw/datasette/issues/1922,1332585861,IC_kwDOBm6k_c5PbaWF,9599,simonw,2022-11-30T18:43:46Z,2022-11-30T18:43:46Z,OWNER,"Here's what Django Rest Framework does: https://github.com/encode/django-rest-framework/blob/1ae812ea209392ad76cc5d2f35f9f7fb337f63e4/rest_framework/views.py#L514-L521 ```python def options(self, request, *args, **kwargs): """""" Handler method for HTTP 'OPTIONS' request. """""" if self.metadata_class is None: return self.http_method_not_allowed(request, *args, **kwargs) data = self.metadata_class().determine_metadata(request, self) return Response(data, status=status.HTTP_200_OK) ``` That default `determine_metadata` method looks like this: https://github.com/encode/django-rest-framework/blob/1ae812ea209392ad76cc5d2f35f9f7fb337f63e4/rest_framework/metadata.py#L61-L71 ```python def determine_metadata(self, request, view): metadata = OrderedDict() metadata['name'] = view.get_view_name() metadata['description'] = view.get_view_description() metadata['renders'] = [renderer.media_type for renderer in view.renderer_classes] metadata['parses'] = [parser.media_type for parser in view.parser_classes] if hasattr(view, 'get_serializer'): actions = self.determine_actions(request, view) if actions: metadata['actions'] = actions return metadata ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332580395,https://api.github.com/repos/simonw/datasette/issues/1922,1332580395,IC_kwDOBm6k_c5PbZAr,9599,simonw,2022-11-30T18:38:22Z,2022-11-30T18:38:22Z,OWNER,"> [@simon](https://fedi.simonwillison.net/@simon) IMO, it should always be a 2XX series response, typically with no content & an extra `Allow` header with a list of HTTP verbs it responds to. https://mastodon.social/@daniellindsley/109434186252099323","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332572453,https://api.github.com/repos/simonw/datasette/issues/1922,1332572453,IC_kwDOBm6k_c5PbXEl,9599,simonw,2022-11-30T18:30:38Z,2022-11-30T18:30:54Z,OWNER,Started a conversation about how OPTIONS should work on Mastodon: https://fedi.simonwillison.net/@simon/109434148676475291,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332561813,https://api.github.com/repos/simonw/datasette/issues/1922,1332561813,IC_kwDOBm6k_c5PbUeV,9599,simonw,2022-11-30T18:20:05Z,2022-11-30T18:20:05Z,OWNER,"Weird, GitHub reply with a 404! ``` ~ % curl -X OPTIONS https://github.com/ -i HTTP/2 404 server: GitHub.com date: Wed, 30 Nov 2022 18:19:39 GMT content-type: text/html; charset=utf-8 content-length: 0 strict-transport-security: max-age=31536000; includeSubdomains; preload x-frame-options: deny x-content-type-options: nosniff x-xss-protection: 0 referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubusercontent.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events; font-src github.githubassets.com; form-action 'self' github.com gist.github.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com objects-origin.githubusercontent.com secured-user-images.githubusercontent.com/ opengraph.githubassets.com github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; worker-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/ vary: Accept-Encoding, Accept, X-Requested-With x-github-request-id: DD6B:5ACA:102E8A6:1164A99:63879EBB ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332561059,https://api.github.com/repos/simonw/datasette/issues/1922,1332561059,IC_kwDOBm6k_c5PbUSj,9599,simonw,2022-11-30T18:19:20Z,2022-11-30T18:19:20Z,OWNER,"Two test failures: ``` ____________________________ test_homepage_options _____________________________ [gw0] linux -- Python 3.11.0 /opt/hostedtoolcache/Python/3.11.0/x64/bin/python app_client = def test_homepage_options(app_client): response = app_client.get(""/"", method=""OPTIONS"") > assert response.status == 405 E assert 200 == 405 E + where 200 = .status /home/runner/work/datasette/datasette/tests/test_html.py:58: AssertionError ______________________ test_client_methods[options-/-405] ______________________ [gw1] linux -- Python 3.11.0 /opt/hostedtoolcache/Python/3.11.0/x64/bin/python datasette = method = 'options', path = '/', expected_status = 405 @pytest.mark.asyncio @pytest.mark.parametrize( ""method,path,expected_status"", [ (""get"", ""/"", 200), (""options"", ""/"", 405), (""head"", ""/"", 200), (""put"", ""/"", 405), (""patch"", ""/"", 405), (""delete"", ""/"", 405), ], ) async def test_client_methods(datasette, method, path, expected_status): client_method = getattr(datasette.client, method) response = await client_method(path) assert isinstance(response, httpx.Response) > assert response.status_code == expected_status E assert 200 == 405 E + where 200 = .status_code /home/runner/work/datasette/datasette/tests/test_internals_datasette_client.py:29: AssertionError =============================== warnings summary =============================== tests/test_cli.py::test_inspect_cli_writes_to_file tests/test_cli.py::test_inspect_cli /home/runner/work/datasette/datasette/datasette/cli.py:163: DeprecationWarning: There is no current event loop loop = asyncio.get_event_loop() tests/test_cli_serve_get.py: 2 warnings tests/test_cli.py: 12 warnings tests/test_crossdb.py: 1 warning /home/runner/work/datasette/datasette/datasette/cli.py:591: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(ds.invoke_startup()) tests/test_cli_serve_get.py: 2 warnings tests/test_cli.py: 12 warnings tests/test_crossdb.py: 1 warning /home/runner/work/datasette/datasette/datasette/cli.py:594: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(check_databases(ds)) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========================== short test summary info ============================ FAILED tests/test_html.py::test_homepage_options - assert 200 == 405 + where 200 = .status FAILED tests/test_internals_datasette_client.py::test_client_methods[options-/-405] - assert 200 == 405 + where 200 = .status_code ====== 2 failed, 1195 passed, 1 skipped, 32 warnings in 191.06s (0:03:11) ====== Error: Process completed with exit code 1. ``` On reading https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/OPTIONS I feel like I should be a bit more thoughtful about how I treat OPTIONS - maybe it should work for every URL on the site, but return a `204 No Content` header? Comparing a few different sites: ``` ~ % curl -X OPTIONS https://www.google.com/ -i HTTP/2 405 allow: GET, HEAD date: Wed, 30 Nov 2022 18:18:15 GMT content-type: text/html; charset=UTF-8 server: gws content-length: 1592 x-xss-protection: 0 x-frame-options: SAMEORIGIN alt-svc: h3="":443""; ma=2592000,h3-29="":443""; ma=2592000,h3-Q050="":443""; ma=2592000,h3-Q046="":443""; ma=2592000,h3-Q043="":443""; ma=2592000,quic="":443""; ma=2592000; v=""46,43"" Error 405 (Method Not Allowed)!!1

405. That’s an error.

The request method OPTIONS is inappropriate for the URL /. That’s all we know. ~ % curl -X OPTIONS https://www.mozilla.org/ -i HTTP/2 405 content-type: text/html; charset=utf-8 content-length: 0 server: meinheld/1.0.2 date: Wed, 30 Nov 2022 18:18:38 GMT allow: GET, HEAD x-frame-options: DENY content-security-policy: child-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com www.googletagmanager.com www.google-analytics.com www.youtube-nocookie.com trackertest.org www.surveygizmo.com accounts.firefox.com accounts.firefox.com.cn www.youtube.com; connect-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com www.googletagmanager.com www.google-analytics.com region1.google-analytics.com logs.convertexperiments.com 1003350.metrics.convertexperiments.com 1003343.metrics.convertexperiments.com sentry.prod.mozaws.net o1069899.sentry.io o1069899.ingest.sentry.io https://accounts.firefox.com/ stage.cjms.nonprod.cloudops.mozgcp.net cjms.services.mozilla.com; frame-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com www.googletagmanager.com www.google-analytics.com www.youtube-nocookie.com trackertest.org www.surveygizmo.com accounts.firefox.com accounts.firefox.com.cn www.youtube.com; script-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com 'unsafe-inline' 'unsafe-eval' www.googletagmanager.com www.google-analytics.com tagmanager.google.com www.youtube.com s.ytimg.com cdn-3.convertexperiments.com app.convert.com data.track.convertexperiments.com 1003350.track.convertexperiments.com 1003343.track.convertexperiments.com; img-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com data: mozilla.org www.googletagmanager.com www.google-analytics.com adservice.google.com adservice.google.de adservice.google.dk creativecommons.org cdn-3.convertexperiments.com logs.convertexperiments.com images.ctfassets.net ad.doubleclick.net; style-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com 'unsafe-inline' app.convert.com; default-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com; font-src 'self' cache-control: max-age=600 expires: Wed, 30 Nov 2022 18:28:38 GMT x-backend-server: bedrock-prod-web-b95bc569d-grd25.iowa-a strict-transport-security: max-age=31536000 x-content-type-options: nosniff x-xss-protection: 1; mode=block referrer-policy: strict-origin-when-cross-origin via: 1.1 google, 1.1 6c90b631453c435bd0022caa657b67e8.cloudfront.net (CloudFront) x-cache: Error from cloudfront x-amz-cf-pop: SFO5-P2 x-amz-cf-id: A6-9mLztaE2tz840CbV9bXYiBMZRKEamDj6jGGEl1U7sg8egWfsDqg== ~ % curl -X OPTIONS https://example.com -i HTTP/2 200 allow: OPTIONS, GET, HEAD, POST cache-control: max-age=604800 content-type: text/html; charset=UTF-8 date: Wed, 30 Nov 2022 18:18:59 GMT expires: Wed, 07 Dec 2022 18:18:59 GMT server: EOS (vny/0451) content-length: 0 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332504654,https://api.github.com/repos/simonw/datasette/issues/1922,1332504654,IC_kwDOBm6k_c5PbGhO,9599,simonw,2022-11-30T17:27:39Z,2022-11-30T17:27:39Z,OWNER,I'll test this once it's deployed to https://latest.datasette.io/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332493004,https://api.github.com/repos/simonw/datasette/issues/1922,1332493004,IC_kwDOBm6k_c5PbDrM,9599,simonw,2022-11-30T17:18:10Z,2022-11-30T17:18:10Z,OWNER,"Here's why: https://github.com/simonw/datasette/blob/4ddd77e51254bda3bac990ea662bac2e6b29c5e0/datasette/views/base.py#L71-L79 That's code in `BaseView` - but it turns out the code that adds CORS headers is in the `DataView` subclass of that (which the various write API endpoints do not use). https://github.com/simonw/datasette/blob/4ddd77e51254bda3bac990ea662bac2e6b29c5e0/datasette/views/base.py#L158-L162","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1922#issuecomment-1332492092,https://api.github.com/repos/simonw/datasette/issues/1922,1332492092,IC_kwDOBm6k_c5PbDc8,9599,simonw,2022-11-30T17:17:21Z,2022-11-30T17:17:21Z,OWNER,I tried running `fetch()` with a POST from a separate domain and got a browser error because it did a GET against the `/db/-/create` endpoint and the 405 method not supported response did not include the CORS headers.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742,Make sure CORS works for write APIs, https://github.com/simonw/datasette/issues/1919#issuecomment-1331657404,https://api.github.com/repos/simonw/datasette/issues/1919,1331657404,IC_kwDOBm6k_c5PX3q8,9599,simonw,2022-11-30T05:19:43Z,2022-11-30T05:19:43Z,OWNER,"This is the test: https://github.com/simonw/datasette/blob/8404b21556d133c89eda4bd1bf5335ed9a0785d6/tests/test_api_write.py#L342-L401 I'm suspicious that there's a timing error of some sort but I can't think what it might be.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469062686,Intermittent `test_delete_row` test failure , https://github.com/simonw/datasette/issues/1918#issuecomment-1331658629,https://api.github.com/repos/simonw/datasette/issues/1918,1331658629,IC_kwDOBm6k_c5PX3-F,9599,simonw,2022-11-30T05:21:51Z,2022-11-30T05:21:51Z,OWNER,"Much better: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469044738,API explorer should list mutable databases first, https://github.com/simonw/datasette/issues/1917#issuecomment-1331644751,https://api.github.com/repos/simonw/datasette/issues/1917,1331644751,IC_kwDOBm6k_c5PX0lP,9599,simonw,2022-11-30T04:59:22Z,2022-11-30T04:59:22Z,OWNER,"Yeah it looks like I introduced this bug here: https://github.com/simonw/datasette/commit/fb7e70d5e72a951efe4b29ad999d8915c032d021","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469043836,Don't allow writable API to edit the `_memory` database, https://github.com/simonw/datasette/issues/1917#issuecomment-1331644078,https://api.github.com/repos/simonw/datasette/issues/1917,1331644078,IC_kwDOBm6k_c5PX0au,9599,simonw,2022-11-30T04:58:06Z,2022-11-30T04:58:06Z,OWNER,"The problem might actually be here: https://github.com/simonw/datasette/blob/9f5321ff1eca58c469a45cc406d7eb5ad05accbd/datasette/app.py#L280-L281 `is_mutable` defaults to `True`, so this line should probably be: ```python self.add_database(Database(self, is_mutable=False, is_memory=True), name=""_memory"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469043836,Don't allow writable API to edit the `_memory` database, https://github.com/simonw/datasette/issues/1916#issuecomment-1331651721,https://api.github.com/repos/simonw/datasette/issues/1916,1331651721,IC_kwDOBm6k_c5PX2SJ,9599,simonw,2022-11-30T05:10:27Z,2022-11-30T05:10:27Z,OWNER,"They should return 405 method not allowed with an `{""ok"":false, ""error"": ""Method not allowed""}` body.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469015001,GET requests against POST endpoints should not 500 error, https://github.com/simonw/datasette/issues/1915#issuecomment-1331479606,https://api.github.com/repos/simonw/datasette/issues/1915,1331479606,IC_kwDOBm6k_c5PXMQ2,9599,simonw,2022-11-30T00:09:06Z,2022-11-30T00:09:06Z,OWNER,One last feature: I want to show an indication on the table page that the table has X seconds left to live.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531,Interactive demo of Datasette 1.0 write APIs, https://github.com/simonw/datasette/issues/1915#issuecomment-1331479328,https://api.github.com/repos/simonw/datasette/issues/1915,1331479328,IC_kwDOBm6k_c5PXMMg,9599,simonw,2022-11-30T00:08:41Z,2022-11-30T00:08:41Z,OWNER,Five minute has now passed and https://latest.datasette.io/ephemeral/new_table is gone.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531,Interactive demo of Datasette 1.0 write APIs, https://github.com/simonw/datasette/issues/1915#issuecomment-1331478611,https://api.github.com/repos/simonw/datasette/issues/1915,1331478611,IC_kwDOBm6k_c5PXMBT,9599,simonw,2022-11-30T00:07:37Z,2022-11-30T00:07:37Z,OWNER,"Then I created an API token at https://latest.datasette.io/-/create-token and ran this: ``` curl -XPOST 'https://latest.datasette.io/ephemeral/new_table/-/insert' \ -H 'Authorization: Bearer xxx' \ -H 'Content-Type: application/json' \ -d '{""row"": {""name"": ""NAME""}}' ``` And it inserted a row into https://latest.datasette.io/ephemeral/new_table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531,Interactive demo of Datasette 1.0 write APIs, https://github.com/simonw/datasette/issues/1915#issuecomment-1331476246,https://api.github.com/repos/simonw/datasette/issues/1915,1331476246,IC_kwDOBm6k_c5PXLcW,9599,simonw,2022-11-30T00:04:35Z,2022-11-30T00:08:24Z,OWNER,"The new https://github.com/simonw/datasette-ephemeral-tables plugin is live now: https://latest.datasette.io/ephemeral - you have to navigate through https://latest.datasette.io/login-as-root first It work! I created a table using https://latest.datasette.io/-/api#path=%2Fephemeral%2F-%2Fcreate&json=%7B%0A++%22table%22%3A+%22new_table%22%2C%0A++%22columns%22%3A+%5B%0A++++%7B%0A++++++%22name%22%3A+%22id%22%2C%0A++++++%22type%22%3A+%22integer%22%0A++++%7D%2C%0A++++%7B%0A++++++%22name%22%3A+%22name%22%2C%0A++++++%22type%22%3A+%22text%22%0A++++%7D%0A++%5D%2C%0A++%22pk%22%3A+%22id%22%0A%7D&method=POST The table should vanish in a few minutes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531,Interactive demo of Datasette 1.0 write APIs, https://github.com/simonw/datasette/issues/1915#issuecomment-1331432223,https://api.github.com/repos/simonw/datasette/issues/1915,1331432223,IC_kwDOBm6k_c5PXAsf,9599,simonw,2022-11-29T23:06:17Z,2022-11-29T23:06:17Z,OWNER,To (slightly) discourage abuse I'm going to make the demo database only visible to the root user - so people can't create tables with rude names and have them show to the public on https://latest.datasette.io/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531,Interactive demo of Datasette 1.0 write APIs, https://github.com/simonw/datasette/issues/1915#issuecomment-1331331082,https://api.github.com/repos/simonw/datasette/issues/1915,1331331082,IC_kwDOBm6k_c5PWoAK,9599,simonw,2022-11-29T21:24:59Z,2022-11-29T21:34:53Z,OWNER,Maybe a plugin called `datasette-temporary-tables` or `datasette-demo-tables` or `datasette-demo-database`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531,Interactive demo of Datasette 1.0 write APIs, https://github.com/simonw/datasette/issues/1914#issuecomment-1347801679,https://api.github.com/repos/simonw/datasette/issues/1914,1347801679,IC_kwDOBm6k_c5QVdJP,9599,simonw,2022-12-13T06:15:54Z,2022-12-13T06:15:54Z,OWNER,"Should make sure that every API that returns an object as the top level (that's almost all of them) includes `""ok"": true` to indicate no errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468689139,Finalize design of JSON for Datasette 1.0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331238029,https://api.github.com/repos/simonw/datasette/issues/1913,1331238029,IC_kwDOBm6k_c5PWRSN,9599,simonw,2022-11-29T20:10:35Z,2022-11-29T20:10:35Z,OWNER,"Released: - https://pypi.org/project/datasette/1.0a0/ - https://docs.datasette.io/en/latest/changelog.html#a0-2022-11-29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331226346,https://api.github.com/repos/simonw/datasette/issues/1913,1331226346,IC_kwDOBm6k_c5PWObq,9599,simonw,2022-11-29T20:00:16Z,2022-11-29T20:00:36Z,OWNER,"Looks like a fix is coming: https://github.com/pypa/twine/issues/940#issuecomment-1331225509 > > Note that `must_decode` was defined in `pkg_info/_compat.py`, and was thus never an API: before 1.9.0, it was only imported and used in `pkginfo/distribution.py'. > > Nevertheless, I will push out a 1.9.1 release of `pkginfo` which restores a deprecated compatibility alias.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331225277,https://api.github.com/repos/simonw/datasette/issues/1913,1331225277,IC_kwDOBm6k_c5PWOK9,9599,simonw,2022-11-29T19:59:14Z,2022-11-29T19:59:34Z,OWNER,I deleted the tag and tried creating a new release. Now running here: https://github.com/simonw/datasette/actions/runs/3577554546,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331216652,https://api.github.com/repos/simonw/datasette/issues/1913,1331216652,IC_kwDOBm6k_c5PWMEM,9599,simonw,2022-11-29T19:54:22Z,2022-11-29T19:54:22Z,OWNER,Filed a bug report here: https://bugs.launchpad.net/pkginfo/+bug/1998249,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331208206,https://api.github.com/repos/simonw/datasette/issues/1913,1331208206,IC_kwDOBm6k_c5PWKAO,9599,simonw,2022-11-29T19:51:31Z,2022-11-29T19:51:31Z,OWNER,https://pypi.org/project/pkginfo/#history - 1.9.0 came out 39 minutes ago!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331207334,https://api.github.com/repos/simonw/datasette/issues/1913,1331207334,IC_kwDOBm6k_c5PWJym,9599,simonw,2022-11-29T19:50:37Z,2022-11-29T19:50:37Z,OWNER,"https://pypi.org/project/setuptools/65.6.3/ came out most recently - 23rd November (wheel and twine are older). No search results at all for that error message. This is very weird, I would have expected it to have been reported by now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331205613,https://api.github.com/repos/simonw/datasette/issues/1913,1331205613,IC_kwDOBm6k_c5PWJXt,9599,simonw,2022-11-29T19:48:52Z,2022-11-29T19:48:52Z,OWNER,https://github.com/simonw/datasette/blob/07aad511769da9242260c850e8d975cbd8c29552/.github/workflows/publish.yml#L52-L61,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331204360,https://api.github.com/repos/simonw/datasette/issues/1913,1331204360,IC_kwDOBm6k_c5PWJEI,9599,simonw,2022-11-29T19:47:40Z,2022-11-29T19:47:40Z,OWNER,"... but the last step of the deploy failed, when it was meant to push to PyPI! ``` Uploading distributions to https://upload.pypi.org/legacy/ Traceback (most recent call last): File ""/opt/hostedtoolcache/Python/3.11.0/x64/bin/twine"", line 8, in sys.exit(main()) ^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/__main__.py"", line 33, in main error = cli.dispatch(sys.argv[1:]) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/cli.py"", line 123, in dispatch return main(args.args) ^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py"", line 198, in main return upload(upload_settings, parsed_args.dists) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py"", line 123, in upload packages_to_upload = [ ^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py"", line 124, in _make_package(filename, signatures, upload_settings) for filename in uploads ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py"", line 77, in _make_package package = package_file.PackageFile.from_filename(filename, upload_settings.comment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/package.py"", line 96, in from_filename meta = DIST_TYPES[dtype](filename) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/wheel.py"", line 42, in __init__ self.extractMetadata() File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/pkginfo/distribution.py"", line 121, in extractMetadata self.parse(data) File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/wheel.py"", line 89, in parse fp = io.StringIO(distribution.must_decode(data)) ^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'pkginfo.distribution' has no attribute 'must_decode'. Did you mean: '_must_decode'? Error: Process completed with exit code 1. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/issues/1913#issuecomment-1331203997,https://api.github.com/repos/simonw/datasette/issues/1913,1331203997,IC_kwDOBm6k_c5PWI-d,9599,simonw,2022-11-29T19:47:13Z,2022-11-29T19:47:13Z,OWNER,"Weird, retrying the tests DID get them to pass. https://github.com/simonw/datasette/actions/runs/3577355358/jobs/6016518244","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401,Release Datasette 1.0a0, https://github.com/simonw/datasette/pull/1912#issuecomment-1331196531,https://api.github.com/repos/simonw/datasette/issues/1912,1331196531,IC_kwDOBm6k_c5PWHJz,9599,simonw,2022-11-29T19:39:10Z,2022-11-29T19:39:10Z,OWNER,"Annoyingly it looks like I can't rebase this one, and I don't want to squash-merge and lose the commits, so I'm going to do a regular merge instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468592292,Merge 1.0-dev (with initial write API) back into main, https://github.com/simonw/datasette/issues/1911#issuecomment-1331135709,https://api.github.com/repos/simonw/datasette/issues/1911,1331135709,IC_kwDOBm6k_c5PV4Td,9599,simonw,2022-11-29T18:50:58Z,2022-11-29T18:50:58Z,OWNER,Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468519699,`/db/-/create` should support creating tables with compound primary keys, https://github.com/simonw/datasette/issues/1911#issuecomment-1331120055,https://api.github.com/repos/simonw/datasette/issues/1911,1331120055,IC_kwDOBm6k_c5PV0e3,9599,simonw,2022-11-29T18:36:01Z,2022-11-29T18:36:01Z,OWNER,"Current API design: ``` POST //-/create ``` ```json { ""table"": ""name_of_new_table"", ""columns"": [ { ""name"": ""id"", ""type"": ""integer"" }, { ""name"": ""title"", ""type"": ""text"" } ], ""pk"": ""id"" } ``` I'm going to add a new `""pks""` key which is a list, and can be used in place of `""pk""`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468519699,`/db/-/create` should support creating tables with compound primary keys, https://github.com/simonw/datasette/issues/1909#issuecomment-1329624931,https://api.github.com/repos/simonw/datasette/issues/1909,1329624931,IC_kwDOBm6k_c5PQHdj,9599,simonw,2022-11-28T19:19:26Z,2022-11-28T19:19:26Z,OWNER,The list of states here is a good example of somewhere this might be useful: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=state&_facet_size=max,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1466952626,Option to sort facets alphabetically, https://github.com/simonw/sqlite-utils/pull/515#issuecomment-1539077777,https://api.github.com/repos/simonw/sqlite-utils/issues/515,1539077777,IC_kwDOCGYnMM5bvHaR,9599,simonw,2023-05-08T21:27:10Z,2023-05-08T21:27:10Z,OWNER,I should have spotted this PR before I shipped my own fix! https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194930,"upsert new rows with constraints, fixes #514", https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539100300,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539100300,IC_kwDOCGYnMM5bvM6M,9599,simonw,2023-05-08T21:50:51Z,2023-05-08T21:50:51Z,OWNER,Seeing as `sqlite-utils` doesn't currently provide mechanisms for adding `check` constraints like this I'm going to leave this - I'm happy with the fix I put in for the `not null` constraints.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539099703,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539099703,IC_kwDOCGYnMM5bvMw3,9599,simonw,2023-05-08T21:50:06Z,2023-05-08T21:50:06Z,OWNER,"Applying the fix from the PR here doesn't fix the above problem either: - https://github.com/simonw/sqlite-utils/pull/515 So it looks like these kinds of `check` constraints currently aren't compatible with how `upsert()` works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539094287,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539094287,IC_kwDOCGYnMM5bvLcP,9599,simonw,2023-05-08T21:44:11Z,2023-05-08T21:44:11Z,OWNER,"OK, this fails silently: ```python import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute('''CREATE TABLE employees ( id INTEGER PRIMARY KEY, name TEXT, age INTEGER, salary REAL, CHECK (salary is not null and salary > 0) );''') db[""employees""].upsert({""id"": 1, ""name"": ""Bob""}, pk=""id"") list(db[""employees""].rows) ```` It outputs: ```python [] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539079507,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539079507,IC_kwDOCGYnMM5bvH1T,9599,simonw,2023-05-08T21:28:37Z,2023-05-08T21:28:37Z,OWNER,"> This means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted. Huh... on that basis, it's possible my fix in https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304 is incomplete. I only covered the 'not null' case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539078429,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539078429,IC_kwDOCGYnMM5bvHkd,9599,simonw,2023-05-08T21:27:40Z,2023-05-08T21:27:40Z,OWNER,"Dupe of: - #538","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/datasette/issues/1905#issuecomment-1320721241,https://api.github.com/repos/simonw/datasette/issues/1905,1320721241,IC_kwDOBm6k_c5OuJtZ,9599,simonw,2022-11-19T01:12:05Z,2022-11-19T01:12:05Z,OWNER,Used it to deploy this: https://fivethirtyeight.datasettes.com/-/versions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1456012874,`publish heroku` failing due to old Python version, https://github.com/simonw/datasette/issues/1905#issuecomment-1320706596,https://api.github.com/repos/simonw/datasette/issues/1905,1320706596,IC_kwDOBm6k_c5OuGIk,9599,simonw,2022-11-19T00:41:44Z,2022-11-19T00:41:44Z,OWNER,"Tested a deploy: ``` % datasette publish heroku fixtures.db -n datasette-issue-1905 › Warning: heroku update available from 7.63.0 to 7.66.4. › Warning: heroku update available from 7.63.0 to 7.66.4. › Warning: heroku update available from 7.63.0 to 7.66.4. Creating datasette-issue-1905... done › Warning: heroku update available from 7.63.0 to 7.66.4. ▸ Couldn't detect GNU tar. Builds could fail due to decompression errors ▸ See https://devcenter.heroku.com/articles/platform-api-deploying-slugs#create-slug-archive ▸ Please install it, or specify the '--tar' option ▸ Falling back to node's built-in compressor -----> Building on the Heroku-22 stack -----> Determining which buildpack to use for this app -----> Python app detected -----> Using Python version specified in runtime.txt -----> Installing python-3.11.0 -----> Installing pip 22.3.1, setuptools 63.4.3 and wheel 0.37.1 -----> Installing SQLite3 -----> Installing requirements with pip Collecting datasette Downloading datasette-0.63.1-py3-none-any.whl (231 kB) ... -----> Running post-compile hook -----> Discovering process types Procfile declares types -> web -----> Compressing... Done: 28M -----> Launching... Released v3 https://datasette-issue-1905.herokuapp.com/ deployed to Heroku Starting November 28th, 2022, free Heroku Dynos, free Heroku Postgres, and free Heroku Data for Redis® will no longer be available. If you have apps using any of these resources, you must upgrade to paid plans by this date to ensure your apps continue to run and to retain your data. For students, we will announce a new program by the end of September. Learn more at https://blog.heroku.com/next-chapter ``` I had to then pay for the dino because I'd run out of free hours. https://datasette-issue-1905.herokuapp.com/-/versions shows: ```json { ""python"": { ""version"": ""3.11.0"", ""full"": ""3.11.0 (main, Oct 24 2022, 21:34:02) [GCC 11.2.0]"" }, ""datasette"": { ""version"": ""0.63.1"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1456012874,`publish heroku` failing due to old Python version, https://github.com/simonw/datasette/issues/1905#issuecomment-1320689643,https://api.github.com/repos/simonw/datasette/issues/1905,1320689643,IC_kwDOBm6k_c5OuB_r,9599,simonw,2022-11-19T00:17:19Z,2022-11-19T00:41:54Z,OWNER,"The tests don't cover this bit at the moment. Would be easier to write tests if there was a `--generate-dir` option as seen in https://datasette.io/plugins/datasette-publish-vercel","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1456012874,`publish heroku` failing due to old Python version, https://github.com/simonw/datasette/issues/1905#issuecomment-1320678715,https://api.github.com/repos/simonw/datasette/issues/1905,1320678715,IC_kwDOBm6k_c5Ot_U7,9599,simonw,2022-11-19T00:02:28Z,2022-11-19T00:02:28Z,OWNER,This is a strong argument for extracting the Heroku support out to a plugin - it would allow this to be fixed with a plugin release without needing to push a full release of Datasette itself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1456012874,`publish heroku` failing due to old Python version, https://github.com/simonw/datasette/issues/1903#issuecomment-1321262142,https://api.github.com/repos/simonw/datasette/issues/1903,1321262142,IC_kwDOBm6k_c5OwNw-,9599,simonw,2022-11-20T22:35:01Z,2022-11-20T22:35:01Z,OWNER,A want to call this `datasette/exceptions.py` inspired by Takahē: https://github.com/andrewgodwin/takahe/blob/f491fdb56e2de9200e14b855b5576009ca99dfa5/core/exceptions.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1455928469,Refactor all error classes into a datasette.exceptions module, https://github.com/simonw/datasette/issues/1903#issuecomment-1320614541,https://api.github.com/repos/simonw/datasette/issues/1903,1320614541,IC_kwDOBm6k_c5OtvqN,9599,simonw,2022-11-18T22:47:41Z,2022-11-18T22:47:41Z,OWNER,"When I do this it's important to update the documentation for `resolve_database()` and the like: https://github.com/simonw/datasette/blob/ee64130fa8a5ff4a24791916c696e10cf2375102/docs/internals.rst#L594","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1455928469,Refactor all error classes into a datasette.exceptions module, https://github.com/simonw/datasette/issues/1901#issuecomment-1319627012,https://api.github.com/repos/simonw/datasette/issues/1901,1319627012,IC_kwDOBm6k_c5Op-kE,9599,simonw,2022-11-18T07:07:03Z,2022-11-18T07:07:03Z,OWNER,"Here's the full list of 10 plugin releases for this issue: * [datasette-search-all 1.1.1](https://github.com/simonw/datasette-search-all/releases/tag/1.1.1) * [datasette-ripgrep 0.7.1](https://github.com/simonw/datasette-ripgrep/releases/tag/0.7.1) * [datasette-socrata 0.3.1](https://github.com/simonw/datasette-socrata/releases/tag/0.3.1) * [datasette-configure-fts 1.1.1](https://github.com/simonw/datasette-configure-fts/releases/tag/1.1.1) * [datasette-edit-templates 0.2](https://github.com/simonw/datasette-edit-templates/releases/tag/0.2) * [datasette-copyable 0.3.2](https://github.com/simonw/datasette-copyable/releases/tag/0.3.2) * [datasette-public 0.2.1](https://github.com/simonw/datasette-public/releases/tag/0.2.1) * [datasette-import-table 0.3.1](https://github.com/simonw/datasette-import-table/releases/tag/0.3.1) * [datasette-indieauth 1.2.2](https://github.com/simonw/datasette-indieauth/releases/tag/1.2.2) * [datasette-edit-schema 0.5.2](https://github.com/simonw/datasette-edit-schema/releases/tag/0.5.2)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319623911,https://api.github.com/repos/simonw/datasette/issues/1901,1319623911,IC_kwDOBm6k_c5Op9zn,9599,simonw,2022-11-18T07:02:56Z,2022-11-18T07:02:56Z,OWNER,That's all of them!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319588163,https://api.github.com/repos/simonw/datasette/issues/1901,1319588163,IC_kwDOBm6k_c5Op1FD,9599,simonw,2022-11-18T06:05:11Z,2022-11-18T06:05:11Z,OWNER,"For `datasette-copyable` I want to show breadcrumbs that take database/instance permissions into account, so I'm removing `{% block nav %}` entirely and replacing it with this: ```html+jinja {% block crumbs %} {{ crumbs.nav(request=request, database=database, table=table) }} {% endblock %} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319528359,https://api.github.com/repos/simonw/datasette/issues/1901,1319528359,IC_kwDOBm6k_c5Opmen,9599,simonw,2022-11-18T04:27:00Z,2022-11-18T04:27:00Z,OWNER,Also `datasette-indieauth` https://github.com/simonw/datasette-indieauth/blob/a08ce67ddad6098b1240adbeff37d040e4df53b1/datasette_indieauth/templates/indieauth.html#L5-L10,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319525520,https://api.github.com/repos/simonw/datasette/issues/1901,1319525520,IC_kwDOBm6k_c5OplyQ,9599,simonw,2022-11-18T04:21:14Z,2022-11-18T07:22:37Z,OWNER,This search helps too: [https://ripgrep.datasette.io/-/ripgrep?pattern=%7B%25+block+nav&literal=on&ignore=on&glob=%21datasette%2F**](https://ripgrep.datasette.io/-/ripgrep?pattern=%7B%25+block+nav&literal=on&ignore=on&glob=%21datasette%2F**),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319493475,https://api.github.com/repos/simonw/datasette/issues/1901,1319493475,IC_kwDOBm6k_c5Opd9j,9599,simonw,2022-11-18T03:19:25Z,2022-11-18T07:03:03Z,OWNER,"Other plugins this looks like it will affect: - [x] `datasette-ripgrep` https://github.com/simonw/datasette-ripgrep/blob/03446464420130368582022eeb5944993f64ec8f/datasette_ripgrep/templates/ripgrep.html#L37-L42 - [x] `datasette-socrata` https://github.com/simonw/datasette-socrata/blob/32fb256a461bf0e790eca10bdc7dd9d96c20f7c4/datasette_socrata/templates/datasette_socrata_error.html#L5-L10 - [x] `datasette-configure-fts` https://github.com/simonw/datasette-configure-fts/blob/eca742e5d4b9190fc22d68bc0a406c575e6d09a0/datasette_configure_fts/templates/configure_fts_database.html#L9-L14 - [x] `datasette-edit-templates` https://github.com/simonw/datasette-edit-templates/blob/f772aff4a2a4080c949746668a8ec6302dbeb0d9/datasette_edit_templates/templates/edit_template.html#L17-L23 - [x] `datasette-copyable` https://github.com/simonw/datasette-copyable/blob/204d5c912a8d48c49155c67fba7339d4bb26ab9a/datasette_copyable/templates/copyable.html#L36-L43 - [x] `datasette-public` https://github.com/simonw/datasette-public/blob/32b6a0ba53bd5714b6b41eddd8705b213c105efc/datasette_public/templates/public_table_change_privacy.html#L5-L11 - [x] `datasette-import-table` https://github.com/simonw/datasettecloud-datasette/blob/37d0fe525c6649c1aec3d1ee8bc35a684570e87f/templates/import_data.html#L5-L10 - [x] `datasette-edit-schema` (three places) - [x] `datasette-indieauth` https://github.com/simonw/datasette-indieauth/blob/a08ce67ddad6098b1240adbeff37d040e4df53b1/datasette_indieauth/templates/indieauth.html#L5-L10","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319483555,https://api.github.com/repos/simonw/datasette/issues/1901,1319483555,IC_kwDOBm6k_c5Opbij,9599,simonw,2022-11-18T03:02:35Z,2022-11-18T03:02:35Z,OWNER,Looks like this issue could affect a bunch of other plugins too: https://cs.github.com/?scopeName=All+repos&scope=&q=%3Cp+class%3D%22crumbs%22%3E+user%3Asimonw,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1901#issuecomment-1319482791,https://api.github.com/repos/simonw/datasette/issues/1901,1319482791,IC_kwDOBm6k_c5OpbWn,9599,simonw,2022-11-18T03:01:36Z,2022-11-18T03:01:36Z,OWNER,"Good catch. Looks like that bug was introduced by this change: https://github.com/simonw/datasette/commit/1a5e5f2aa951e5bd731067a49819efba68fbe8ef From: - https://github.com/simonw/datasette/issues/1831 The search all plugin includes this code which interacts poorly with that refactor: https://github.com/simonw/datasette-search-all/blob/847b55c368a285e4567627029624d7872ee75cac/datasette_search_all/templates/search_all.html#L31-L36 ```html+jinja {% block nav %}

home

{{ super() }} {% endblock %} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1453813400,"Some plugins show ""home"" breadcrumbs twice in the top left", https://github.com/simonw/datasette/issues/1900#issuecomment-1319664697,https://api.github.com/repos/simonw/datasette/issues/1900,1319664697,IC_kwDOBm6k_c5OqHw5,419145,rdmurphy,2022-11-18T07:59:36Z,2022-11-18T08:00:38Z,NONE,"Okay, my final observations for the night! I've been pushing and pulling the various levers in `utils/__init__.py` to see what makes this work without hard-coding in something for `arm64` and it seems that if I change `/usr/lib/x86_64-linux-gnu/mod_spatialite.so` [here](https://github.com/simonw/datasette/blob/3ecd131e57add427d847b614c920c9624bb2e66b/datasette/utils/__init__.py#L407) to just `mod_spatialite` it's happy. Unfortunately cannot audit that for `x86_64`, but maybe that's a solution that'd be cross-arch compatible? It seems like it's the hard-coding of that path that's tripping it up. (It was actually [this comment from back in 2018 in an entirely unrelated repo](https://github.com/pelias/docker/pull/28#issuecomment-433168462) that nudged me to try this, ha.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319641636,https://api.github.com/repos/simonw/datasette/issues/1900,1319641636,IC_kwDOBm6k_c5OqCIk,419145,rdmurphy,2022-11-18T07:27:26Z,2022-11-18T07:27:26Z,NONE,"Can confirm that my `uname -a` returns something different at the end: ``` root:xnu-8792.41.9~2/RELEASE_ARM64_T6000 arm64 ``` I'm in `arm64` land, you're in `x86_64`. I am admittedly very fuzzy on how this factors into Docker these days. Honestly thought this was one of the things Docker was suppose to help address. 🤔","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319639462,https://api.github.com/repos/simonw/datasette/issues/1900,1319639462,IC_kwDOBm6k_c5OqBmm,419145,rdmurphy,2022-11-18T07:24:19Z,2022-11-18T07:24:19Z,NONE,"Is it, uh, possible we are on different architectures? 😅 I'm using an Apple M1 Pro. I jumped into a bash shell of an unmodified `python:3.11.0-slim-bullseye` container and manually ran `apt-get update` and installed `libsqlite3-mod-spatialite`. I don't end up with with `mod_spatialite.so` in `/usr/lib/x86_64-linux-gnu/` — _mine_ is in `/usr/lib/aarch64-linux-gnu/`. [I swapped that directory in here](https://github.com/simonw/datasette/blob/3db37e9a21f774d6c387fd04bf1e4c870554209e/datasette/utils/__init__.py#L407) in a local copy of `datasette` and poof — it worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319631421,https://api.github.com/repos/simonw/datasette/issues/1900,1319631421,IC_kwDOBm6k_c5Op_o9,9599,simonw,2022-11-18T07:13:00Z,2022-11-18T07:13:00Z,OWNER,"You get: ``` => [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye 0.9s => [internal] load build context 2.3s => => transferring context: 72.38MB 2.3s => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c 0.0s ``` I get: ``` => [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye 1.0s => [internal] load build context 0.0s => => transferring context: 705B 0.0s => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c 0.0s ``` Both the image name and the hash are _exactly_ the same. So why are you getting an error while mine works OK? For my machine: ``` ~ % docker --version Docker version 20.10.12, build e91ed57 ~ % uname -a Darwin Simons-MacBook-Pro-2.local 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:54 PDT 2022; root:xnu-8792.41.9~2/RELEASE_X86_64 x86_64 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319629469,https://api.github.com/repos/simonw/datasette/issues/1900,1319629469,IC_kwDOBm6k_c5Op_Kd,9599,simonw,2022-11-18T07:10:17Z,2022-11-18T07:10:17Z,OWNER,This is so weird! What version of Datasette do you get from `datasette --version` there - and what's your Docker version / operating system version?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319596087,https://api.github.com/repos/simonw/datasette/issues/1900,1319596087,IC_kwDOBm6k_c5Op3A3,419145,rdmurphy,2022-11-18T06:16:33Z,2022-11-18T06:16:33Z,NONE,"Interesting! So I tried this locally using your copy of `nps-spatialite.db` and I got the same error. 🤔 ``` ❯ datasette package nps-spatialite.db --spatialite [+] Building 27.5s (10/10) FINISHED => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 622B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye 0.9s => [internal] load build context 2.3s => => transferring context: 72.38MB 2.3s => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c 0.0s => [2/6] COPY . /app 0.1s => [3/6] WORKDIR /app 0.0s => [4/6] RUN apt-get update && apt-get install -y python3-dev gcc libsqlite3-mod-spatialite && rm -rf /var/lib/apt/lists/* 18.5s => [5/6] RUN pip install -U datasette 4.9s => ERROR [6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json 0.7s ------ > [6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json: #10 0.681 Traceback (most recent call last): #10 0.681 File ""/usr/local/bin/datasette"", line 8, in #10 0.681 sys.exit(cli()) #10 0.681 ^^^^^ #10 0.681 File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 1130, in __call__ #10 0.682 return self.main(*args, **kwargs) #10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^ #10 0.682 File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 1055, in main #10 0.682 rv = self.invoke(ctx) #10 0.682 ^^^^^^^^^^^^^^^^ #10 0.682 File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 1657, in invoke #10 0.682 return _process_result(sub_ctx.command.invoke(sub_ctx)) #10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #10 0.682 File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 1404, in invoke #10 0.682 return ctx.invoke(self.callback, **ctx.params) #10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #10 0.682 File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 760, in invoke #10 0.682 return __callback(*args, **kwargs) #10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^^ #10 0.683 File ""/usr/local/lib/python3.11/site-packages/datasette/cli.py"", line 164, in inspect #10 0.683 inspect_data = loop.run_until_complete(inspect_(files, sqlite_extensions)) #10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #10 0.683 File ""/usr/local/lib/python3.11/asyncio/base_events.py"", line 650, in run_until_complete #10 0.683 return future.result() #10 0.683 ^^^^^^^^^^^^^^^ #10 0.683 File ""/usr/local/lib/python3.11/site-packages/datasette/cli.py"", line 179, in inspect_ #10 0.683 counts = await database.table_counts(limit=3600 * 1000) #10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #10 0.683 File ""/usr/local/lib/python3.11/site-packages/datasette/database.py"", line 304, in table_counts #10 0.683 for table in await self.table_names(): #10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^ #10 0.683 File ""/usr/local/lib/python3.11/site-packages/datasette/database.py"", line 342, in table_names #10 0.683 results = await self.execute( #10 0.683 ^^^^^^^^^^^^^^^^^^^ #10 0.683 File ""/usr/local/lib/python3.11/site-packages/datasette/database.py"", line 267, in execute #10 0.683 results = await self.execute_fn(sql_operation_in_thread) #10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #10 0.683 File ""/usr/local/lib/python3.11/site-packages/datasette/database.py"", line 213, in execute_fn #10 0.683 return await asyncio.get_event_loop().run_in_executor( #10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #10 0.683 File ""/usr/local/lib/python3.11/concurrent/futures/thread.py"", line 58, in run #10 0.683 result = self.fn(*self.args, **self.kwargs) #10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #10 0.683 File ""/usr/local/lib/python3.11/site-packages/datasette/database.py"", line 209, in in_thread #10 0.683 self.ds._prepare_connection(conn, self.name) #10 0.683 File ""/usr/local/lib/python3.11/site-packages/datasette/app.py"", line 593, in _prepare_connection #10 0.683 conn.execute(""SELECT load_extension(?)"", [extension]) #10 0.683 sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory ------ executor failed running [/bin/sh -c datasette inspect nps-spatialite.db --inspect-file inspect-data.json]: exit code: 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319583703,https://api.github.com/repos/simonw/datasette/issues/1900,1319583703,IC_kwDOBm6k_c5Opz_X,9599,simonw,2022-11-18T05:58:31Z,2022-11-18T05:58:31Z,OWNER,Could you provide full steps to reproduce plus a SpatiaLite database file that triggered this for you? I'm not able to recreate the problem.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319583281,https://api.github.com/repos/simonw/datasette/issues/1900,1319583281,IC_kwDOBm6k_c5Opz4x,9599,simonw,2022-11-18T05:57:44Z,2022-11-18T05:57:44Z,OWNER,"Did you use the `--spatialite` option? I just tried this: datasette package nps-spatialite.db It built the image OK (I didn't see the error you reported), but running the container failed with an error: ``` /tmp % docker run -p 8001:8001 7298e8e6bbfb Usage: datasette serve [OPTIONS] [FILES]... Try 'datasette serve --help' for help. Error: It looks like you're trying to load a SpatiaLite database without first loading the SpatiaLite module. Read more: https://docs.datasette.io/en/stable/spatialite.html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319582239,https://api.github.com/repos/simonw/datasette/issues/1900,1319582239,IC_kwDOBm6k_c5Opzof,9599,simonw,2022-11-18T05:55:38Z,2022-11-18T05:55:38Z,OWNER,"Trying this out locally with this 69MB SpatiaLite file I happened to have lying around (from testing `shapefile-to-sqlite` a while ago): https://static.simonwillison.net/static/2022/nps-spatialite.db ``` % datasette package nps-spatialite.db --spatialite ... => [2/6] COPY . /app 0.4s => [3/6] WORKDIR /app 0.0s => [4/6] RUN apt-get update && apt-get install -y python3-dev gcc libsqlite3-mod-spatialite && rm -rf /var/lib/apt/lists/* 29.6s => [5/6] RUN pip install -U datasette 12.0s => [6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json 2.6s => exporting to image 3.0s => => exporting layers 3.0s => => writing image sha256:4dfef1c373c5c057ef7ac22344f834d522acef24313a1b25d2eba9e500066b8f 0.0s ``` And then: docker run -p 8001:8001 4dfef1c373c5 This worked fine for me. I ran `datasette package` using Datasette 0.63.1.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319574972,https://api.github.com/repos/simonw/datasette/issues/1900,1319574972,IC_kwDOBm6k_c5Opx28,9599,simonw,2022-11-18T05:41:28Z,2022-11-18T05:41:28Z,OWNER,Oh this is with `datasette package`? That should work. Will investigate.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1900#issuecomment-1319571220,https://api.github.com/repos/simonw/datasette/issues/1900,1319571220,IC_kwDOBm6k_c5Opw8U,9599,simonw,2022-11-18T05:34:35Z,2022-11-18T05:34:35Z,OWNER,Which Docker image are you using here? It looks like it's missing SpatiaLite from the image.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452572348,datasette package --spatialite throws error during build, https://github.com/simonw/datasette/issues/1899#issuecomment-1319642338,https://api.github.com/repos/simonw/datasette/issues/1899,1319642338,IC_kwDOBm6k_c5OqCTi,9599,simonw,2022-11-18T07:28:28Z,2022-11-18T07:28:28Z,OWNER,Demo: https://latest.datasette.io/fixtures,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452495049,Clicking within the CodeMirror area below the SQL (i.e. when there's only a single line) doesn't cause the editor to get focused , https://github.com/simonw/datasette/issues/1899#issuecomment-1319584553,https://api.github.com/repos/simonw/datasette/issues/1899,1319584553,IC_kwDOBm6k_c5Op0Mp,9599,simonw,2022-11-18T06:00:10Z,2022-11-18T06:01:50Z,OWNER,"I can't actually remember where that `min-height: 70px` came from. I just tried without it and it seems fine - especially since any time you add a newline in the editor it increases its height to fit. I ran this in the DevTools console: ```javascript document.querySelector('.cm-editor').style.minHeight = 'none'; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452495049,Clicking within the CodeMirror area below the SQL (i.e. when there's only a single line) doesn't cause the editor to get focused , https://github.com/simonw/datasette/issues/1899#issuecomment-1318897922,https://api.github.com/repos/simonw/datasette/issues/1899,1318897922,IC_kwDOBm6k_c5OnMkC,95570,bgrins,2022-11-17T16:32:42Z,2022-11-17T16:32:42Z,CONTRIBUTOR,Another idea would be to just not set a min-height and allow the 1 line input to be 1 line heigh,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452495049,Clicking within the CodeMirror area below the SQL (i.e. when there's only a single line) doesn't cause the editor to get focused , https://github.com/simonw/datasette/issues/1899#issuecomment-1317873458,https://api.github.com/repos/simonw/datasette/issues/1899,1317873458,IC_kwDOBm6k_c5OjScy,95570,bgrins,2022-11-17T00:31:07Z,2022-11-17T00:31:07Z,CONTRIBUTOR,"This is one way to fix it ```patch r.html diff --git a/datasette/static/cm-editor-6.0.1.js b/datasette/static/cm-editor-6.0.1.js index c1fd2ab..68cf398 100644 --- a/datasette/static/cm-editor-6.0.1.js +++ b/datasette/static/cm-editor-6.0.1.js @@ -22,7 +22,14 @@ export function editorFromTextArea(textarea, conf = {}) { // https://github.com/codemirror/lang-sql#user-content-sqlconfig.tables let view = new EditorView({ doc: textarea.value, + extensions: [ + EditorView.theme({ + "".cm-content"": { + // Height on cm-content ensures the editor is focusable by clicking beyond the height of the text + minHeight: ""70px"", + }, + }), keymap.of([ { key: ""Shift-Enter"", diff --git a/datasette/templates/_codemirror.html b/datasette/templates/_codemirror.html index dea4710..c4629ae 100644 --- a/datasette/templates/_codemirror.html +++ b/datasette/templates/_codemirror.html @@ -4,7 +4,6 @@ .cm-editor { resize: both; overflow: hidden; - min-height: 70px; width: 80%; border: 1px solid #ddd; } ``` I don't love it but it seems to work for the default case. You can still retrigger the bug by resizing the editor to be > 70px high. The other approach would be to listen for a click on that empty region and move focus to the editor, or something","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452495049,Clicking within the CodeMirror area below the SQL (i.e. when there's only a single line) doesn't cause the editor to get focused , https://github.com/simonw/datasette/pull/1898#issuecomment-1319642535,https://api.github.com/repos/simonw/datasette/issues/1898,1319642535,IC_kwDOBm6k_c5OqCWn,9599,simonw,2022-11-18T07:28:45Z,2022-11-18T07:28:45Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452485922,Use DOMContentLoaded instead of load event for CodeMirror initialization, https://github.com/simonw/datasette/pull/1898#issuecomment-1317870550,https://api.github.com/repos/simonw/datasette/issues/1898,1317870550,IC_kwDOBm6k_c5OjRvW,22429695,codecov[bot],2022-11-17T00:27:55Z,2022-11-17T00:27:55Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1898?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.55**% // Head: **92.55**% // No change to project coverage :thumbsup: > Coverage data is based on head [(`5eb4ea4`)](https://codecov.io/gh/simonw/datasette/pull/1898?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`00e233d`)](https://codecov.io/gh/simonw/datasette/commit/00e233d7a7f6443cb95fb5227c23580c48551cad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch has no changes to coverable lines.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1898 +/- ## ======================================= Coverage 92.55% 92.55% ======================================= Files 35 35 Lines 4432 4432 ======================================= Hits 4102 4102 Misses 330 330 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1898?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452485922,Use DOMContentLoaded instead of load event for CodeMirror initialization, https://github.com/simonw/datasette/issues/1897#issuecomment-1319570586,https://api.github.com/repos/simonw/datasette/issues/1897,1319570586,IC_kwDOBm6k_c5Opwya,9599,simonw,2022-11-18T05:33:20Z,2022-11-18T05:33:20Z,OWNER,"One of the big changes still left to do for Datasette 1.0 is to unify the JSON representation with the context psssed to the templates (via an `?_extra=` mechanism to add extra context needed by the HTML templates), because a goal for 1.0 is for the template context to be a documented API contract such that custom templates won't break with future releases. As such I expect to do quite a bit of refactoring and cleanup on how the template context works later on.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1319533445,https://api.github.com/repos/simonw/datasette/issues/1897,1319533445,IC_kwDOBm6k_c5OpnuF,95570,bgrins,2022-11-18T04:38:03Z,2022-11-18T04:38:03Z,CONTRIBUTOR,Are you tracking the change to send the JSON over to the frontend separately or was that part of this? Something like this is probably pretty close https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25#diff-0c93232bfd5477eeac96382e52769108b41433d960d5277ffcccf2f464e60abdR9,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1319478811,https://api.github.com/repos/simonw/datasette/issues/1897,1319478811,IC_kwDOBm6k_c5OpaYb,9599,simonw,2022-11-18T02:53:57Z,2022-11-18T02:53:57Z,OWNER,"I decided to just go for the view names, not their columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1319477721,https://api.github.com/repos/simonw/datasette/issues/1897,1319477721,IC_kwDOBm6k_c5OpaHZ,9599,simonw,2022-11-18T02:51:40Z,2022-11-18T02:51:40Z,OWNER,Views aren't currently available in the `_internal` schema.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1319435374,https://api.github.com/repos/simonw/datasette/issues/1897,1319435374,IC_kwDOBm6k_c5OpPxu,9599,simonw,2022-11-18T01:33:30Z,2022-11-18T01:33:30Z,OWNER,"Just noticed that this isn't including views, which it should.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1319401843,https://api.github.com/repos/simonw/datasette/issues/1897,1319401843,IC_kwDOBm6k_c5OpHlz,9599,simonw,2022-11-18T00:42:03Z,2022-11-18T00:42:23Z,OWNER,"This function works even if the SQLite JSON functions are not available: ```python async def _table_columns(datasette, database_name): internal = datasette.get_database(""_internal"") result = await internal.execute( ""select table_name, name from columns where database_name = ?"", [database_name], ) table_columns = {} for row in result.rows: table_columns.setdefault(row[""table_name""], []).append(row[""name""]) return table_columns ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1317840727,https://api.github.com/repos/simonw/datasette/issues/1897,1317840727,IC_kwDOBm6k_c5OjKdX,9599,simonw,2022-11-16T23:57:52Z,2022-11-16T23:57:52Z,OWNER,In terms of permissions: if you have `execute-sql` permission for a database then it's OK for you to see the table columns for that database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1317839781,https://api.github.com/repos/simonw/datasette/issues/1897,1317839781,IC_kwDOBm6k_c5OjKOl,9599,simonw,2022-11-16T23:56:47Z,2022-11-16T23:56:47Z,OWNER,I'm going to call this `table_columns` in the template context (because `schema` might mean `CREATE TABLE ...`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1897#issuecomment-1317838892,https://api.github.com/repos/simonw/datasette/issues/1897,1317838892,IC_kwDOBm6k_c5OjKAs,9599,simonw,2022-11-16T23:55:42Z,2022-11-16T23:55:42Z,OWNER,"Here's where the schema is hard-coded at the moment: https://github.com/simonw/datasette/blob/00e233d7a7f6443cb95fb5227c23580c48551cad/datasette/templates/_codemirror_foot.html#L2-L7 I figured out how to extract that data from the `_internal` table in this comment: https://github.com/simonw/datasette/pull/1893#issuecomment-1317475720 Although that used JSON functions which may (in a real edge-case) not be available in the version of SQLite that Datasette is running on, so probably going to use a regular SQL query and then assemble the JSON separately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452457263,Serve schema JSON to the SQL editor to enable autocomplete, https://github.com/simonw/datasette/issues/1896#issuecomment-1320616559,https://api.github.com/repos/simonw/datasette/issues/1896,1320616559,IC_kwDOBm6k_c5OtwJv,9599,simonw,2022-11-18T22:51:14Z,2022-11-18T22:51:14Z,OWNER,New methods are documented here: https://docs.datasette.io/en/1.0-dev/internals.html#resolve-database-request,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452364777,Extract logic for resolving a URL to a database / table / row, https://github.com/simonw/datasette/issues/1896#issuecomment-1320588299,https://api.github.com/repos/simonw/datasette/issues/1896,1320588299,IC_kwDOBm6k_c5OtpQL,9599,simonw,2022-11-18T22:16:59Z,2022-11-18T22:17:06Z,OWNER,"Found myself needing an `await db.view_exists()` method for this, similar to the existing `await db.table_exists()` one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452364777,Extract logic for resolving a URL to a database / table / row, https://github.com/simonw/datasette/issues/1896#issuecomment-1317757949,https://api.github.com/repos/simonw/datasette/issues/1896,1317757949,IC_kwDOBm6k_c5Oi2P9,9599,simonw,2022-11-16T22:27:47Z,2022-11-18T21:48:29Z,OWNER,"Open question: should `resolve_table()` know how to identify named canned queries too? I think not, at least for the moment. Feels a bit too specialist to expose in a documented API.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452364777,Extract logic for resolving a URL to a database / table / row, https://github.com/simonw/datasette/issues/1896#issuecomment-1317757112,https://api.github.com/repos/simonw/datasette/issues/1896,1317757112,IC_kwDOBm6k_c5Oi2C4,9599,simonw,2022-11-16T22:26:52Z,2022-11-16T22:33:13Z,OWNER,"Some ideas from walking the dog: Challenge: standard URL routing of request to database/table/row Standardize on the named components of the URL patterns - `database`, `table`, `pks` Async function that takes the request and the Datasette instance and returns a Resolved instance with: ``` .level - database or table or row (better name?) .database - the name of the database .db - the database object .table - the name of the table (or view) .is_view perhaps? .pk_values if it's a row ``` Should this attempt to resolve names queries too? ``` .where_sql - the where fragment you use .where_params - accompanying dictionary await datasette.resolve_request(request) ``` Or even better three methods: ```python datasette.resolve_database(request) datasette.resolve_table(request) datasette.resolve_row(request) ``` These can be typed correctly Methods raise `NotFound` if not found","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1452364777,Extract logic for resolving a URL to a database / table / row, https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316530539,https://api.github.com/repos/simonw/sqlite-utils/issues/512,1316530539,IC_kwDOCGYnMM5OeKlr,9599,simonw,2022-11-16T07:49:50Z,2022-11-16T07:49:50Z,OWNER,Tests passed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450952393,mypy failures in CI, https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316447182,https://api.github.com/repos/simonw/sqlite-utils/issues/512,1316447182,IC_kwDOCGYnMM5Od2PO,9599,simonw,2022-11-16T06:32:31Z,2022-11-16T06:32:31Z,OWNER,"Test failed again: https://github.com/simonw/sqlite-utils/actions/runs/3476950474/jobs/5812663096 `E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/s/spatialite/libsqlite3-mod-spatialite_4.3.0a-6build1_amd64.deb Unable to connect to azure.archive.ubuntu.com:http:` That looks like an intermittent error. I'll try running it again in the morning.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450952393,mypy failures in CI, https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316437748,https://api.github.com/repos/simonw/sqlite-utils/issues/512,1316437748,IC_kwDOCGYnMM5Odz70,9599,simonw,2022-11-16T06:24:31Z,2022-11-16T06:24:31Z,OWNER,"``` sqlite-utils % pipx run no_implicit_optional . Calculating full-repo metadata... Executing codemod... 11.43s 98% complete, 0.24s estimated for 5 files to go... ``` Then: ``` Finished codemodding 239 files! - Transformed 239 files successfully. - Skipped 0 files. - Failed to codemod 0 files. - 0 warnings were generated. ``` Here's the diff: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a06f4b7..e819d17 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -297,12 +297,12 @@ class Database: def __init__( self, - filename_or_conn: Union[str, pathlib.Path, sqlite3.Connection] = None, + filename_or_conn: Optional[Union[str, pathlib.Path, sqlite3.Connection]] = None, memory: bool = False, - memory_name: str = None, + memory_name: Optional[str] = None, recreate: bool = False, recursive_triggers: bool = True, - tracer: Callable = None, + tracer: Optional[Callable] = None, use_counts_table: bool = False, ): assert (filename_or_conn is not None and (not memory and not memory_name)) or ( @@ -341,7 +341,7 @@ class Database: self.conn.close() @contextlib.contextmanager - def tracer(self, tracer: Callable = None): + def tracer(self, tracer: Optional[Callable] = None): """""" Context manager to temporarily set a tracer function - all executed SQL queries will be passed to this. @@ -378,7 +378,7 @@ class Database: def register_function( self, - fn: Callable = None, + fn: Optional[Callable] = None, deterministic: bool = False, replace: bool = False, name: Optional[str] = None, @@ -879,7 +879,7 @@ class Database: pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, hash_id: Optional[str] = None, hash_id_columns: Optional[Iterable[str]] = None, @@ -1129,7 +1129,7 @@ class Database: sql += "" [{}]"".format(name) self.execute(sql) - def init_spatialite(self, path: str = None) -> bool: + def init_spatialite(self, path: Optional[str] = None) -> bool: """""" The ``init_spatialite`` method will load and initialize the SpatiaLite extension. The ``path`` argument should be an absolute path to the compiled extension, which @@ -1182,7 +1182,7 @@ class Queryable: def count_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, ) -> int: """""" @@ -1213,12 +1213,12 @@ class Queryable: def rows_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, - order_by: str = None, + order_by: Optional[str] = None, select: str = ""*"", - limit: int = None, - offset: int = None, + limit: Optional[int] = None, + offset: Optional[int] = None, ) -> Generator[dict, None, None]: """""" Iterate over every row in this table or view that matches the specified where clause. @@ -1251,11 +1251,11 @@ class Queryable: def pks_and_rows_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, - order_by: str = None, - limit: int = None, - offset: int = None, + order_by: Optional[str] = None, + limit: Optional[int] = None, + offset: Optional[int] = None, ) -> Generator[Tuple[Any, Dict], None, None]: """""" Like ``.rows_where()`` but returns ``(pk, row)`` pairs - ``pk`` can be a single value or tuple. @@ -1345,7 +1345,7 @@ class Table(Queryable): pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, batch_size: int = 100, hash_id: Optional[str] = None, @@ -1545,7 +1545,7 @@ class Table(Queryable): pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, hash_id: Optional[str] = None, hash_id_columns: Optional[Iterable[str]] = None, @@ -2464,7 +2464,7 @@ class Table(Queryable): columns: Optional[Iterable[str]] = None, limit: Optional[int] = None, offset: Optional[int] = None, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, quote: bool = False, ) -> Generator[dict, None, None]: @@ -2527,7 +2527,7 @@ class Table(Queryable): def delete_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, analyze: bool = False, ) -> ""Table"": ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450952393,mypy failures in CI, https://github.com/simonw/datasette/pull/1893#issuecomment-1317837416,https://api.github.com/repos/simonw/datasette/issues/1893,1317837416,IC_kwDOBm6k_c5OjJpo,9599,simonw,2022-11-16T23:54:02Z,2022-11-16T23:54:02Z,OWNER,"I'm going to tackle #1897 in the next few minutes. Tests failed due to Prettier check, just pushed a fix so it would ignore `.bundle.js` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317834838,https://api.github.com/repos/simonw/datasette/issues/1893,1317834838,IC_kwDOBm6k_c5OjJBW,95570,bgrins,2022-11-16T23:50:58Z,2022-11-16T23:50:58Z,CONTRIBUTOR,"Should we empty out the fixture schema to avoid fixture autocomplete showing up on live databases in the interim, or are you planning to tackle #1897 shortly?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317831555,https://api.github.com/repos/simonw/datasette/issues/1893,1317831555,IC_kwDOBm6k_c5OjIOD,9599,simonw,2022-11-16T23:47:13Z,2022-11-16T23:47:13Z,OWNER,I'll open a follow-up issue to fix the schema.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317831425,https://api.github.com/repos/simonw/datasette/issues/1893,1317831425,IC_kwDOBm6k_c5OjIMB,9599,simonw,2022-11-16T23:47:05Z,2022-11-16T23:47:05Z,OWNER,"OK, let's do it! Thanks so much for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317829214,https://api.github.com/repos/simonw/datasette/issues/1893,1317829214,IC_kwDOBm6k_c5OjHpe,9599,simonw,2022-11-16T23:44:36Z,2022-11-16T23:44:36Z,OWNER,Deployed that to https://datasette-pr-1893.vercel.app/fixtures - looks good to me!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317805482,https://api.github.com/repos/simonw/datasette/issues/1893,1317805482,IC_kwDOBm6k_c5OjB2q,95570,bgrins,2022-11-16T23:18:17Z,2022-11-16T23:18:17Z,CONTRIBUTOR,Alright with https://github.com/simonw/datasette/pull/1893/commits/f254be4b38936e95e7a7f25866e7c6b0520db96f we should be getting autocomplete on fixture data. Give that a test and see what you think,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317797044,https://api.github.com/repos/simonw/datasette/issues/1893,1317797044,IC_kwDOBm6k_c5Oi_y0,9599,simonw,2022-11-16T23:08:34Z,2022-11-16T23:08:34Z,OWNER,"> I can push up a commit that uses the static fixtures schema for testing, but given that the query used to generate it is authed we would still need some work to make that work on live data, right? Yeah, push that up. I'm happy to wire in the query right after we land this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317789308,https://api.github.com/repos/simonw/datasette/issues/1893,1317789308,IC_kwDOBm6k_c5Oi958,95570,bgrins,2022-11-16T22:59:57Z,2022-11-16T22:59:57Z,CONTRIBUTOR,"I can push up a commit that uses the static fixtures schema for testing, but given that the query used to generate it is authed we would still need some work to make that work on live data, right? Ideally it could come down to db and query views directly to avoid waiting on an extra xhr and managing that state change.On Nov 16, 2022, at 2:16 PM, Simon Willison ***@***.***> wrote: Honestly I'm not too bothered if table names with weird characters don't work correctly here - I care about those in the Datasette fixtures.db database because Datasette aims to support ANY valid SQLite database, so I need stuff in the test suite that includes weird edge cases like this. But I would hope very few people actually create tables with spaces in their names, so it's not a huge concern to me if autocompletion doesn't work properly for those. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317746206,https://api.github.com/repos/simonw/datasette/issues/1893,1317746206,IC_kwDOBm6k_c5OizYe,9599,simonw,2022-11-16T22:17:24Z,2022-11-16T22:17:24Z,OWNER,Deployed 0a649e8f78c23e8db6869442eeb0dfe36a5443da: https://datasette-pr-1893.vercel.app/fixtures,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317744563,https://api.github.com/repos/simonw/datasette/issues/1893,1317744563,IC_kwDOBm6k_c5Oiy-z,9599,simonw,2022-11-16T22:16:03Z,2022-11-16T22:16:03Z,OWNER,"Honestly I'm not too bothered if table names with weird characters don't work correctly here - I care about those in the Datasette `fixtures.db` database because Datasette aims to support ANY valid SQLite database, so I need stuff in the test suite that includes weird edge cases like this. But I would hope very few people actually create tables with spaces in their names, so it's not a huge concern to me if autocompletion doesn't work properly for those.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317715580,https://api.github.com/repos/simonw/datasette/issues/1893,1317715580,IC_kwDOBm6k_c5Oir58,95570,bgrins,2022-11-16T21:49:51Z,2022-11-16T21:49:51Z,CONTRIBUTOR,"I think the table completion still has some quirks to work out. Something like ``` schema: { ""[123_starts_with_digits]"": [""content""], } ``` Seems to work alright, although it will append it after any other numbers you've started typing - so you end up with `select * from 12[123_starts_with_digits]` if you typed ""12"" to get the completion to appear. This might just be an issue with numeric names, I haven't tested it in a lot of detail. You can do ``` searchable: [ { label: ""name with . and spaces"", apply: ""[name with . and spaces]"", }, ""pk"", ""text1"", ""text2"", ], ``` Which is pretty neat and will show the non-escaped string but complete to the escaped one. You can't easily do that with the table names themselves (you can pass a `tables` array like so https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/src/sql.ts#L121 but it will overwrite the columns from the schema ). It's buggy enough (bad output for these unusual table names) that I'd suggest that work gets moved into a follow up to the upgrade to 6. That would give space to sort out how to deliver that to the view directly, figure out where name escaping should happen, and have overall testing to uncover bugs and fix papercuts before enabling it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317681193,https://api.github.com/repos/simonw/datasette/issues/1893,1317681193,IC_kwDOBm6k_c5Oijgp,95570,bgrins,2022-11-16T21:19:13Z,2022-11-16T21:19:13Z,CONTRIBUTOR,"Alright, added Cmd+Enter to submit (Ctrl+Enter on Windows as well bc of using Meta-Enter on codemirror). We can make that MacOS only by changing the combo to Cmd+Enter specifically but I think it's probably fine to have both.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317522323,https://api.github.com/repos/simonw/datasette/issues/1893,1317522323,IC_kwDOBm6k_c5Oh8uT,95570,bgrins,2022-11-16T18:59:49Z,2022-11-16T18:59:49Z,CONTRIBUTOR,Or I guess you could return only the escaped table name and then we could derive the unescaped from the client side (removing the outer `[]` when present),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317520304,https://api.github.com/repos/simonw/datasette/issues/1893,1317520304,IC_kwDOBm6k_c5Oh8Ow,95570,bgrins,2022-11-16T18:58:43Z,2022-11-16T18:58:43Z,CONTRIBUTOR,Nice. And is it possible to include another field which is an escaped table name (only when necessary) - i.e. `[123_starts_with_digits]`. Or is that easy enough to derive on the client? I'm thinking we'd map those to Completion objects so that CM would show the non escaped text but complete to escaped.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317475720,https://api.github.com/repos/simonw/datasette/issues/1893,1317475720,IC_kwDOBm6k_c5OhxWI,9599,simonw,2022-11-16T18:25:16Z,2022-11-16T18:25:16Z,OWNER,"Here's a query that returns the exact JSON we need to pass to the schema: https://latest.datasette.io/_internal?sql=with+inner+as+%28%0D%0A++select%0D%0A++++table_name%2C%0D%0A++++json_group_array%28name%29+as+table_columns%0D%0A++from%0D%0A++++columns%0D%0A++where%0D%0A++++database_name+%3D+%3Adatabase%0D%0A++group+by%0D%0A++++table_name%0D%0A%29%0D%0Aselect%0D%0A++json_group_object%28table_name%2C+table_columns%29%0D%0Afrom%0D%0A++inner&database=fixtures ```sql with inner as ( select table_name, json_group_array(name) as table_columns from columns where database_name = :database group by table_name ) select json_group_object(table_name, table_columns) from inner ``` Returns (after pretty-printing): ```json { ""123_starts_with_digits"": [ ""content"" ], ""Table With Space In Name"": [ ""content"", ""pk"" ], ""attraction_characteristic"": [ ""name"", ""pk"" ], ""binary_data"": [ ""data"" ], ""complex_foreign_keys"": [ ""f1"", ""f2"", ""f3"", ""pk"" ], ""compound_primary_key"": [ ""content"", ""pk1"", ""pk2"" ], ""compound_three_primary_keys"": [ ""content"", ""pk1"", ""pk2"", ""pk3"" ], ""custom_foreign_key_label"": [ ""foreign_key_with_custom_label"", ""pk"" ], ""facet_cities"": [ ""id"", ""name"" ], ""facetable"": [ ""_city_id"", ""_neighborhood"", ""complex_array"", ""created"", ""distinct_some_null"", ""n"", ""on_earth"", ""pk"", ""planet_int"", ""state"", ""tags"" ], ""foreign_key_references"": [ ""foreign_key_compound_pk1"", ""foreign_key_compound_pk2"", ""foreign_key_with_blank_label"", ""foreign_key_with_label"", ""foreign_key_with_no_label"", ""pk"" ], ""infinity"": [ ""value"" ], ""no_primary_key"": [ ""a"", ""b"", ""c"", ""content"" ], ""primary_key_multiple_columns"": [ ""content"", ""content2"", ""id"" ], ""primary_key_multiple_columns_explicit_label"": [ ""content"", ""content2"", ""id"" ], ""roadside_attraction_characteristics"": [ ""attraction_id"", ""characteristic_id"" ], ""roadside_attractions"": [ ""address"", ""latitude"", ""longitude"", ""name"", ""pk"", ""url"" ], ""searchable"": [ ""name with . and spaces"", ""pk"", ""text1"", ""text2"" ], ""searchable_fts"": [ ""__langid"", ""docid"", ""name with . and spaces"", ""searchable_fts"", ""text1"", ""text2"" ], ""searchable_fts_docsize"": [ ""docid"", ""size"" ], ""searchable_fts_segdir"": [ ""end_block"", ""idx"", ""leaves_end_block"", ""level"", ""root"", ""start_block"" ], ""searchable_fts_segments"": [ ""block"", ""blockid"" ], ""searchable_fts_stat"": [ ""id"", ""value"" ], ""searchable_tags"": [ ""searchable_id"", ""tag"" ], ""select"": [ ""and"", ""group"", ""having"", ""json"" ], ""simple_primary_key"": [ ""content"", ""id"" ], ""sortable"": [ ""content"", ""pk1"", ""pk2"", ""sortable"", ""sortable_with_nulls"", ""sortable_with_nulls_2"", ""text"" ], ""table/with/slashes.csv"": [ ""content"", ""pk"" ], ""tags"": [ ""tag"" ], ""units"": [ ""distance"", ""frequency"", ""pk"" ] } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317465874,https://api.github.com/repos/simonw/datasette/issues/1893,1317465874,IC_kwDOBm6k_c5Ohu8S,9599,simonw,2022-11-16T18:21:17Z,2022-11-16T18:21:54Z,OWNER,"I was worrying about the server-side overhead of collecting together all of the tables and column names for databases that might have hundreds of tables... but then I remember that I built the `_internal` table precisely for this kind of thing - so gathering all of that data should still only be a single SQL query against an in-memory database. https://latest.datasette.io/login-as-root and then visit this page for an example query: https://latest.datasette.io/_internal?sql=select%0D%0A++database_name%2C%0D%0A++table_name%2C%0D%0A++json_group_array%28name%29%0D%0Afrom%0D%0A++columns%0D%0Awhere%0D%0A++database_name+%21%3D+%27_internal%27%0D%0Agroup+by%0D%0A++database_name%2C%0D%0A++table_name ```sql select database_name, table_name, json_group_array(name) from columns where database_name != '_internal' group by database_name, table_name ``` database_name | table_name | json_group_array(name) -- | -- | -- extra_database | searchable | [""pk"",""text1"",""text2""] extra_database | searchable_fts | [""__langid"",""content"",""docid"",""searchable_fts"",""text1"",""text2""] extra_database | searchable_fts_content | [""c0text1"",""c1text2"",""c2content"",""docid""] extra_database | searchable_fts_segdir | [""end_block"",""idx"",""leaves_end_block"",""level"",""root"",""start_block""] extra_database | searchable_fts_segments | [""block"",""blockid""] fixtures | 123_starts_with_digits | [""content""] fixtures | Table With Space In Name | [""content"",""pk""] fixtures | attraction_characteristic | [""name"",""pk""] fixtures | binary_data | [""data""] fixtures | complex_foreign_keys | [""f1"",""f2"",""f3"",""pk""] fixtures | compound_primary_key | [""content"",""pk1"",""pk2""] ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317456909,https://api.github.com/repos/simonw/datasette/issues/1893,1317456909,IC_kwDOBm6k_c5OhswN,9599,simonw,2022-11-16T18:17:39Z,2022-11-16T18:17:39Z,OWNER,"Tiny feature request (since you're in this code already) - I keep hitting Command+Enter on my macOS keyboard to submit the query, but the correct shortcut is Shift+Enter. Would be great if both worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317452541,https://api.github.com/repos/simonw/datasette/issues/1893,1317452541,IC_kwDOBm6k_c5Ohrr9,9599,simonw,2022-11-16T18:15:52Z,2022-11-16T18:15:52Z,OWNER,"Deployed latest copy with: ``` datasette publish vercel fixtures.db \ --project datasette-pr-1893 \ --about 'PR 1893' \ --about_url https://github.com/simonw/datasette/pull/1893 \ --scope datasette \ --branch eccb1c6c781d69d8ec3c542ef65c78a4a0927a7c ``` https://datasette-pr-1893.vercel.app/fixtures ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317449610,https://api.github.com/repos/simonw/datasette/issues/1893,1317449610,IC_kwDOBm6k_c5Ohq-K,9599,simonw,2022-11-16T18:14:28Z,2022-11-16T18:14:28Z,OWNER,"> I'm thinking of also adding `count` to the list since that's a common thing people would want to autocomplete. I notice BQ console highlights `count` in the same manner as other keywords like `select` as well. Huh, yeah we should definitely have `count` - surprised it's not on the list on https://www.sqlite.org/lang_keywords.html which is why we didn't get it from the GPT-3 generated schema.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317329157,https://api.github.com/repos/simonw/datasette/issues/1893,1317329157,IC_kwDOBm6k_c5OhNkF,95570,bgrins,2022-11-16T16:46:52Z,2022-11-16T16:46:52Z,CONTRIBUTOR,"> > > UI issue I see on the autocomplete popup with overlapping icon & text. Screenshot's from Firefox, it seems even a little more pronounced on Safari I checked and if I empty out app.css the bug goes away, so there's some kind of inheritance issue there. It's hard to debug bc the autocomplete popup goes away on blur (i.e. when trying to inspect it in devtools), but at least it's narrowed down a bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317326406,https://api.github.com/repos/simonw/datasette/issues/1893,1317326406,IC_kwDOBm6k_c5OhM5G,95570,bgrins,2022-11-16T16:45:09Z,2022-11-16T16:45:09Z,CONTRIBUTOR,"For escaped table names it looks like we could pass a Completion object (https://codemirror.net/docs/ref/#autocomplete) instead of a string which would allow the non escaped name to be a label and then the escaped name to actually complete in the editor, which might help with some of the funkiness I was seeing w/ completion","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317314064,https://api.github.com/repos/simonw/datasette/issues/1893,1317314064,IC_kwDOBm6k_c5OhJ4Q,95570,bgrins,2022-11-16T16:36:46Z,2022-11-16T16:36:46Z,CONTRIBUTOR,"With ```patch diff --git a/datasette/templates/_codemirror_foot.html b/datasette/templates/_codemirror_foot.html index ed709b3..74fe18e 100644 --- a/datasette/templates/_codemirror_foot.html +++ b/datasette/templates/_codemirror_foot.html @@ -7,7 +7,11 @@ sqlFormat.hidden = false; } if (sqlInput) { - var editor = (window.editor = cm.editorFromTextArea(sqlInput)); + var editor = (window.editor = cm.editorFromTextArea(sqlInput, { + schema: { + compound_three_primary_keys: [""pk1"", ""pk2"", ""pk3"", ""content""], + }, + })); ``` we get table autocompletion and column completion if you name the table in the query (see screencast). I do see bugs with escaped table names like `""'123_starts_with_digits'"": [""col1"", ""col2""]` or `""[123_starts_with_digits]"": [""col1"", ""col2""]` where it doesn't seem to pick up the column names though. I think it needs some further testing and debugging. https://user-images.githubusercontent.com/95570/202238521-e613b4e2-ba92-4418-9068-fc022edaee93.mp4 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1317281292,https://api.github.com/repos/simonw/datasette/issues/1893,1317281292,IC_kwDOBm6k_c5OhB4M,95570,bgrins,2022-11-16T16:19:16Z,2022-11-16T16:19:16Z,CONTRIBUTOR,"Ha, nice idea! Updating the dialect with that list. I'm thinking of also adding `count` to the list since that's a common thing people would want to autocomplete. I notice BQ console highlights `count` in the same manner as other keywords like `select` as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316412234,https://api.github.com/repos/simonw/datasette/issues/1893,1316412234,IC_kwDOBm6k_c5OdttK,9599,simonw,2022-11-16T06:00:39Z,2022-11-16T06:01:36Z,OWNER,"Should note though that this is a classic example of GPT-3 making stuff up in places. > current: Returns the current date, time, or timestamp `select current` throws an error for me: https://latest.datasette.io/_memory?sql=select+current `select current_date, current_time, current_timestamp` works though: https://latest.datasette.io/_memory?sql=select+current_date%2C+current_time%2C+current_timestamp So let's drop `current` from the list. I'm OK with it though, I think it's likely good enough for the first attempt at this. We should drop `temp` and `temporary` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316401895,https://api.github.com/repos/simonw/datasette/issues/1893,1316401895,IC_kwDOBm6k_c5OdrLn,9599,simonw,2022-11-16T05:50:40Z,2022-11-16T05:50:40Z,OWNER,"So I think our dialect (at least to start with) should be: ``` keywords: ""and as asc between by case cast current current_date current_time current_timestamp desc distinct each else escape except exists explain filter first for from full generated group having if in index inner intersect into isnull join last left like limit not null or order outer over pragma primary query raise range regexp right rollback row select set table temp temporary then to union unique using values view virtual when where"", // https://www.sqlite.org/datatype3.html types: ""null integer real text blob"", builtin: """" ``` I left `builtin` blank here because I don't think we need any of those things at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316400688,https://api.github.com/repos/simonw/datasette/issues/1893,1316400688,IC_kwDOBm6k_c5Odq4w,9599,simonw,2022-11-16T05:49:31Z,2022-11-16T05:49:31Z,OWNER,"OK, I decided to play around with GPT-3 for this. https://beta.openai.com/playground I prompted: ``` For each keyword say it it can be used in a SQLite select statement: and as asc between by case cast cross current current_date current_time current_timestamp desc detach distinct each else end escape except exclude exclusive exists explain fail filter first following for foreign from full generated glob group groups having if ignore immediate in index indexed initially inner insert instead intersect into is isnull join key last left like limit match materialized natural no not nothing notnull null nulls of offset on or order others outer over partition plan pragma preceding primary query raise range recursive references regexp reindex release rename replace restrict returning right rollback row rows savepoint select set table temp temporary then ties to transaction trigger unbounded union unique update using vacuum values view virtual when where window with without - and: yes - as: yes - asc: yes - between: yes - by: yes - case: yes - cast: yes - current: yes - current_date: yes - current_time: yes - current_timestamp: yes - desc: yes - detach: no - distinct: yes - each: yes - else: yes - end: yes - escape: yes - except: no - exclude: yes - exclusive: no ``` Having given it these examples I ran it a few times: ``` - exists: yes - explain: no - fail: no - filter: yes - first: yes - following: no - for: yes - foreign: no - from: yes - full: no - generated: no - glob: yes - group: yes - groups: no - having: yes - if: yes - ignore: no - immediate: no - in: yes - index: yes - indexed: no - initially: no - inner: no - insert: no - instead: no - intersect: no - into: yes - is: yes - isnull: yes - join: yes - key: no - last: yes - left: yes - like: yes - limit: yes - match: no - materialized: no -natural: yes - no: no - not: no - nothing: no - notnull: yes - null: yes - nulls: yes - of: no - offset: yes - on: yes - or: yes - order: yes - others: yes - outer: no - over: no - partition: no - pragma: no - preceding: no - primary: no - query: no - raise: no - range: no - recursive: no - references: no - regexp: yes - reindex: no - release: no - rename: no - replace: no - restrict: no - returning: yes - right: yes - rollback: no - row: yes - rows: no - savepoint: no - select: yes - set: no - table: yes - temp: yes - temporary: yes - then: yes - ties: no - to: yes - transaction: no - trigger: no - unbounded: no - union: yes - unique: yes - update: no - using: yes - vacuum: no - values: yes - view: yes - virtual: no - when: yes - where: yes - window: no - with: yes - without: no ``` I added this: ``` A space separated list of all of the keywords for which the answer was yes: ``` It replied: ``` and as asc between by case cast current current_date current_time current_timestamp desc distinct each else escape except exists explain filter first for from full generated group having if in index inner intersect into isnull join last left like limit not null or order outer over pragma primary query raise range regexp right rollback row select set table temp temporary then to union unique using values view virtual when where ``` Then I tried doing this: ``` Produce a short summary of how each of those are used: - and: ``` This kind of worked too! It ended up defining every keyword, not just the ones for which the ""is it used in select"" was ""yes"", but it gave me this list: > - and: Used to combine multiple conditions in a WHERE clause > - as: Rename a column or table in the output of a query > - asc: Sort results in ascending order > - between: Used in a WHERE clause to specify a range > - by: Specifies how to sort the results of a query > - case: Used to create different outputs based on conditions > - cast: Converts a value from one data type to another > - current: Returns the current date, time, or timestamp > - current_date: Returns the current date > - current_time: Returns the current time > - current_timestamp: Returns the current timestamp > - desc: Sort results in descending order > - distinct: Selects only unique values > - each: Used with aggregate functions > - else: Used with the CASE statement > - escape: Used to escape special characters in a string > - except: Returns all rows from the first query that are not in the second query > - exists: Used with subqueries > - explain: Analyzes a query and returns information about it > - fail: Used with the ON CONFLICT clause > - filter: Used to specify conditions for aggregate functions > - first: Used to specify conditions for aggregate functions > - following: Used with window functions > - for: Used to specify conditions for aggregate functions > - foreign: Creates a foreign key > - from: Specifies the table(s) to select data from > - full: Used with outer joins > - generated: Used with generated columns > - glob: Used to match a string with a pattern > - group: Groups together rows that have the same values > - groups: Used with aggregate functions > - having: Specifies conditions for groups > - if: Used with the ON CONFLICT clause > - ignore: Used with the ON CONFLICT clause > - immediate: Used with the ON CONFLICT clause > - in: Used to check if a value is in a list > - index: Creates an index on a column or columns > - indexed: Used with the ON CONFLICT clause > - initially: Used with the ON CONFLICT clause > - inner: Creates an inner join > - insert: Inserts new data into a table > - instead: Used with triggers > - intersect: Returns all rows that are in both queries > - into: Specifies the table to insert data into > - is: Used to compare values > - isnull: Checks if a value is null > - join: Joins two or more tables together > - key: Creates a primary key or unique key > - last: Used to specify conditions for aggregate functions > - left: Creates a left join > - like: Checks if a value matches a pattern > - limit: Specifies the maximum number of rows to return > - not: Negates a condition > - null: Specifies a null value > - nulls: Used with the ORDER BY clause > - of: Used with window functions > - offset: Specifies the number of rows to skip > - on: Specifies conditions for a join > - or: Checks if either condition is true > - order: Specifies how to sort the results of a query > - others: Used with the CASE statement > - outer: Creates an outer join > - over: Used with window functions > - partition: Used with window functions > - pragma: Used to query or change various internal settings > - preceding: Used with window functions > - primary: Creates a primary key > - query: Used with the ON CONFLICT clause > - raise: Used with the ON CONFLICT clause > - range: Used with window functions > - recursive: Used with common table expressions > - references: Creates a foreign key > - regexp: Checks if a value matches a regular expression > - reindex: Recreates an index > - release: Used with savepoints > - rename: Changes the name of a table or column > - replace: Inserts new data into a table or updates existing data > - restrict: Used with savepoints > - returning: Returns data from inserted or updated rows > - right: Creates a right join > - rollback: Rolls back a transaction > - row: Used with aggregate functions > - select: Selects data from a table > - set: Changes the value of a column > - table: Specifies the table(s) to select data from > - temp: Creates a temporary table > - temporary: Creates a temporary table > - then: Used with the CASE statement > - ties: Used with the ORDER BY clause > - to: Used with common table expressions > - transaction: Starts a transaction > - trigger: Creates a trigger > - unbounded: Used with window functions > - union: Returns all rows from both queries > - unique: Creates a unique constraint > - update: Updates existing data in a table > - using: Specifies the table(s) to join > - vacuum: Optimizes a database > - values: Specifies the values to insert into a table > - view: Creates a view > - virtual: Creates a virtual table > - when: Used with the CASE statement > - where: Specifies conditions for a query > - window: Used with window functions > - with: Specifies common table expressions used in a query > - without: Used with indexes","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316387382,https://api.github.com/repos/simonw/datasette/issues/1893,1316387382,IC_kwDOBm6k_c5Odno2,95570,bgrins,2022-11-16T05:33:55Z,2022-11-16T05:33:55Z,CONTRIBUTOR,I added a commit to make our own dialect at https://github.com/simonw/datasette/pull/1893/commits/e273fc8ed5341bdf0b622e722d761bd2acc30a90. Pulled in the full list of keywords from https://www.sqlite.org/lang_keywords.html but haven't gone through and pruned it to only include common select keywords. @simonw you'll have better knowledge than me on that - do you want to take a first shot at narrowing that down to the set that people will be using in the editor?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316340865,https://api.github.com/repos/simonw/datasette/issues/1893,1316340865,IC_kwDOBm6k_c5OdcSB,9599,simonw,2022-11-16T04:49:30Z,2022-11-16T04:49:43Z,OWNER,"> The main issue is that we don't pass the relevant table data down to QueryView. If you can come up with a static example JSON data structure example that does the right thing, I'm happy to refactor QueryView to make that available to the template - or even have a separate `fetch()` that grabs just the data needed for the autocomplete as a separate hit when the page loads (whichever has better performance implications). I'm working a fair amount in the view classes at the moment so adding this to that work would make sense. ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316339035,https://api.github.com/repos/simonw/datasette/issues/1893,1316339035,IC_kwDOBm6k_c5Odb1b,95570,bgrins,2022-11-16T04:47:11Z,2022-11-16T04:47:11Z,CONTRIBUTOR,"> Have you ever seen CodeMirror correctly auto-completing columns? I'm not entirely sure I believe that the feature works anywhere else. I was thinking of the BigQuery console, like But they must be doing something pretty custom & appears to be using Monaco anyway. I suspect some kind of lower level autocomplete integration could make this work, but if the table completion is a good-enough starting point I think it's not too hard. The main issue is that we don't pass the relevant table data down to QueryView.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316320521,https://api.github.com/repos/simonw/datasette/issues/1893,1316320521,IC_kwDOBm6k_c5OdXUJ,95570,bgrins,2022-11-16T04:29:23Z,2022-11-16T04:29:23Z,CONTRIBUTOR," UI issue I see on the autocomplete popup with overlapping icon & text. Screenshot's from Firefox, it seems even a little more pronounced on Safari","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316318961,https://api.github.com/repos/simonw/datasette/issues/1893,1316318961,IC_kwDOBm6k_c5OdW7x,95570,bgrins,2022-11-16T04:27:51Z,2022-11-16T04:27:51Z,CONTRIBUTOR,"> The resize handle doesn't appear on Mobile Safari on iPhone - I don't think that particularly matters though. > > The textarea does get a weird border around it when focused on iPhone though. The default focus styles appear to be ``` .c1.cm-editor.cm-focused { outline: 1px dotted #212121; } ``` Which I also see on desktop. Would be nice to changed to whatever the default UA textarea styles are to blend in better but I wouldn't recommend removing it entirely - just to keep the visual indication that the element is focused. Maybe followup material to have a theming pass","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316297347,https://api.github.com/repos/simonw/datasette/issues/1893,1316297347,IC_kwDOBm6k_c5OdRqD,22429695,codecov[bot],2022-11-16T04:05:12Z,2022-11-16T23:27:45Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1893?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.55**% // Head: **92.55**% // No change to project coverage :thumbsup: > Coverage data is based on head [(`f254be4`)](https://codecov.io/gh/simonw/datasette/pull/1893?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`6f610e1`)](https://codecov.io/gh/simonw/datasette/commit/6f610e1d94b7b8ec605b5b7fcb01537f6adf9c5b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch has no changes to coverable lines.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1893 +/- ## ======================================= Coverage 92.55% 92.55% ======================================= Files 35 35 Lines 4432 4432 ======================================= Hits 4102 4102 Misses 330 330 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1893?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316294156,https://api.github.com/repos/simonw/datasette/issues/1893,1316294156,IC_kwDOBm6k_c5OdQ4M,9599,simonw,2022-11-16T04:00:12Z,2022-11-16T04:00:12Z,OWNER,Have you ever seen CodeMirror correctly auto-completing columns? I'm not entirely sure I believe that the feature works anywhere else.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316293353,https://api.github.com/repos/simonw/datasette/issues/1893,1316293353,IC_kwDOBm6k_c5OdQrp,9599,simonw,2022-11-16T03:59:03Z,2022-11-16T03:59:03Z,OWNER,"Deployed a fresh copy: ``` datasette publish vercel fixtures.db \ --branch b7b2942b13f9ea09cfa9f8c73e2869b9bd2349ae \ --project datasette-pr-1893 \ --about 'PR 1893' \ --about_url https://github.com/simonw/datasette/pull/1893 \ --scope datasette ``` https://datasette-pr-1893.vercel.app/fixtures","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316256386,https://api.github.com/repos/simonw/datasette/issues/1893,1316256386,IC_kwDOBm6k_c5OdHqC,95570,bgrins,2022-11-16T03:18:06Z,2022-11-16T03:18:06Z,CONTRIBUTOR,"> If you can get a version of this working with table and column autocompletion just using a static JavaScript object in the source code with the right tables and columns, I'm happy to take on the work of turning that static object into something that Datasette includes in the page itself with all of the correct values. This version ""sort of"" works when on the main database page where the template passes the relevant data https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25 by doing this and passing that into the `schema` object: ``` let TABLES_DATA = []; {% if tables is defined %} TABLES_DATA = {{ tables | tojson(indent=2) }}; {% endif %} // Turn into an object, shaped like https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/test/test-complete.ts#L27. const TABLES_SCHEMA = Object.fromEntries( new Map( TABLES_DATA.map((table) => { return [table.name, table.columns]; }) ).entries() ); ``` But there are a number of papercuts with it - it's not escaping table names with spaces (likely be fixable from the data being passed into the view) but mainly it doesn't seem to autocomplete columns. I think it might only want to do it when you first type the table name from my read of https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/test/test-complete.ts#L37. It's possible I'm just passing something wrong, but it may end up being something that needs feature work upstream. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316253186,https://api.github.com/repos/simonw/datasette/issues/1893,1316253186,IC_kwDOBm6k_c5OdG4C,9599,simonw,2022-11-16T03:16:36Z,2022-11-16T03:16:36Z,OWNER,Yeah I haven't written this down anywhere but Datasette definitely has an undocumented preference for lower-case SQL.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316243602,https://api.github.com/repos/simonw/datasette/issues/1893,1316243602,IC_kwDOBm6k_c5OdEiS,95570,bgrins,2022-11-16T03:11:46Z,2022-11-16T03:11:46Z,CONTRIBUTOR,Was just reviewing the SQL options and there's an [upperCaseKeywords](https://github.com/codemirror/lang-sql#user-content-sqlconfig.uppercasekeywords) if we'd rather have SELECT vs select. Datasette seems to prefer lowercase so probably best to keep it as-is,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316236448,https://api.github.com/repos/simonw/datasette/issues/1893,1316236448,IC_kwDOBm6k_c5OdCyg,9599,simonw,2022-11-16T03:04:57Z,2022-11-16T03:04:57Z,OWNER,If you rebase from `main` you should get the fix for that test failure.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316232588,https://api.github.com/repos/simonw/datasette/issues/1893,1316232588,IC_kwDOBm6k_c5OdB2M,9599,simonw,2022-11-16T03:00:04Z,2022-11-16T03:00:04Z,OWNER,"Oops, the tests are failing because of a test failure I introduced here: - #1890","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316231560,https://api.github.com/repos/simonw/datasette/issues/1893,1316231560,IC_kwDOBm6k_c5OdBmI,9599,simonw,2022-11-16T02:59:00Z,2022-11-16T02:59:00Z,OWNER,"The resize handle doesn't appear on Mobile Safari on iPhone - I don't think that particularly matters though. The textarea does get a weird border around it when focused on iPhone though. Focused: ![BF34E8FB-E35C-4CAB-9BFB-8EEF7E29B16C_1_201_a](https://user-images.githubusercontent.com/9599/202072748-c85bab94-a039-4ed6-8185-3cac25c78ed3.jpeg) Not focused: ![31A5CF38-D540-4A1A-8A7D-E29453D150F4_1_201_a](https://user-images.githubusercontent.com/9599/202072744-d9f0ea62-13b7-46ff-afe1-6d88d7fb8b53.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316227073,https://api.github.com/repos/simonw/datasette/issues/1893,1316227073,IC_kwDOBm6k_c5OdAgB,9599,simonw,2022-11-16T02:54:22Z,2022-11-16T02:54:32Z,OWNER,"If you can get a version of this working with table and column autocompletion just using a static JavaScript object in the source code with the right tables and columns, I'm happy to take on the work of turning that static object into something that Datasette includes in the page itself with all of the correct values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316141764,https://api.github.com/repos/simonw/datasette/issues/1893,1316141764,IC_kwDOBm6k_c5OcrrE,9599,simonw,2022-11-16T01:26:59Z,2022-11-16T01:26:59Z,OWNER,"Resizing works great for me - and the page automatically sizes the editor to fit an existing query, e.g. on https://datasette-pr-1893.vercel.app/fixtures?sql=select+id%2C+content%2C+content2%0D%0A++from+primary_key_multiple_columns_explicit_label%0D%0A++order+by+id%0D%0A++limit+101","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316137982,https://api.github.com/repos/simonw/datasette/issues/1893,1316137982,IC_kwDOBm6k_c5Ocqv-,9599,simonw,2022-11-16T01:23:47Z,2022-11-16T01:23:47Z,OWNER,"Autocomplete here looks promising (I've wanted that to work for years!), but it does currently show a whole bunch of suggestions which aren't part of the SQLite SQL dialect: ![autocomplete](https://user-images.githubusercontent.com/9599/202060211-51ec9f45-bc52-459a-a729-27fc2faadff9.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316135244,https://api.github.com/repos/simonw/datasette/issues/1893,1316135244,IC_kwDOBm6k_c5OcqFM,9599,simonw,2022-11-16T01:21:41Z,2022-11-16T01:21:41Z,OWNER,"I just deployed a demo instance like this (using the commit hash from this PR): ```bash datasette publish vercel fixtures.db \ --branch 544f7025900b78f63c34b9985522271ba5fd9c0f \ --project datasette-pr-1893 \ --scope datasette \ --about 'PR 1893' \ --about_url https://github.com/simonw/datasette/pull/1893 ``` Here's the result: https://datasette-pr-1893.vercel.app/fixtures","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316041828,https://api.github.com/repos/simonw/datasette/issues/1893,1316041828,IC_kwDOBm6k_c5OcTRk,95570,bgrins,2022-11-15T23:51:35Z,2022-11-15T23:51:35Z,CONTRIBUTOR,"I experimented with autocompleting the actual schema in https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25, but it would need some work (current problems with it listed in the commit message there)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1315869946,https://api.github.com/repos/simonw/datasette/issues/1893,1315869946,IC_kwDOBm6k_c5ObpT6,95570,bgrins,2022-11-15T21:12:38Z,2022-11-15T21:12:38Z,CONTRIBUTOR,https://github.com/Sphinxxxx/cm-resize isn't compatible with 6. There's a suggestion to try using CSS resize in https://discuss.codemirror.net/t/resizing-codemirror-6/3265/2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1315869040,https://api.github.com/repos/simonw/datasette/issues/1893,1315869040,IC_kwDOBm6k_c5ObpFw,95570,bgrins,2022-11-15T21:11:42Z,2022-11-15T21:11:42Z,CONTRIBUTOR,"extraKeys is done - Shift+Enter is added in the helper function, and it appears that the Tab behavior now defaults to what the `Tab: false` setting was doing (allowing it to escape to the form)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1315853097,https://api.github.com/repos/simonw/datasette/issues/1893,1315853097,IC_kwDOBm6k_c5OblMp,95570,bgrins,2022-11-15T20:55:40Z,2022-11-15T20:55:40Z,CONTRIBUTOR,Should also minify the bundled output,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/issues/1892#issuecomment-1315814786,https://api.github.com/repos/simonw/datasette/issues/1892,1315814786,IC_kwDOBm6k_c5Obb2C,4399499,ocdtrekkie,2022-11-15T20:14:38Z,2022-11-15T20:14:38Z,NONE,"I have no particular point, I just want to say being around for the 1.0 release of Datasette seems historic and legendary to witness.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450312343,Merge 1.0-dev branch back to main, https://github.com/simonw/datasette/issues/1892#issuecomment-1315805498,https://api.github.com/repos/simonw/datasette/issues/1892,1315805498,IC_kwDOBm6k_c5ObZk6,9599,simonw,2022-11-15T20:05:30Z,2022-11-15T20:05:30Z,OWNER,"One slight concern: https://latest.datasette.io/ will increasingly reflect a version that isn't the most recent production release. I might setup https://stable.datasette.io/ as a demo instance of the most recent non-alpha release to compensate for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450312343,Merge 1.0-dev branch back to main, https://github.com/simonw/datasette/issues/1892#issuecomment-1315804535,https://api.github.com/repos/simonw/datasette/issues/1892,1315804535,IC_kwDOBm6k_c5ObZV3,9599,simonw,2022-11-15T20:04:38Z,2022-11-15T20:04:38Z,OWNER,"I'll do this after the 1.0a0 release: - #1708","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450312343,Merge 1.0-dev branch back to main, https://github.com/simonw/datasette/issues/1891#issuecomment-1331181922,https://api.github.com/repos/simonw/datasette/issues/1891,1331181922,IC_kwDOBm6k_c5PWDli,9599,simonw,2022-11-29T19:23:41Z,2022-11-29T19:23:41Z,OWNER,https://github.com/simonw/datasette/blob/4d49a5a39739476e1ada43f70a0029abcef07977/docs/changelog.rst#10a0-2022-11-29,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450303205,1.0a0 release notes, https://github.com/simonw/datasette/issues/1891#issuecomment-1331143292,https://api.github.com/repos/simonw/datasette/issues/1891,1331143292,IC_kwDOBm6k_c5PV6J8,9599,simonw,2022-11-29T18:57:40Z,2022-11-29T18:57:40Z,OWNER,I'm going to keep these short - they'll mostly be links to the documentation for the new features.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450303205,1.0a0 release notes, https://github.com/simonw/datasette/issues/1891#issuecomment-1331140747,https://api.github.com/repos/simonw/datasette/issues/1891,1331140747,IC_kwDOBm6k_c5PV5iL,9599,simonw,2022-11-29T18:55:42Z,2022-11-29T18:55:42Z,OWNER,"All features for the alpha are complete now. Release notes should be based on these commits: https://github.com/simonw/datasette/compare/0.63.2...6bda2257868a2cbd70b84b7a86a5bcb47dcc4874","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450303205,1.0a0 release notes, https://github.com/simonw/datasette/issues/1891#issuecomment-1320625260,https://api.github.com/repos/simonw/datasette/issues/1891,1320625260,IC_kwDOBm6k_c5OtyRs,9599,simonw,2022-11-18T23:01:03Z,2022-11-18T23:01:48Z,OWNER,I think this actually needs to include a whole section of the documentation about the road to 1.0 - what to expect (planned breaking changes) etc. I can add that to the https://docs.datasette.io/en/stable/contributing.html page perhaps - or even create a Roadmap page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450303205,1.0a0 release notes, https://github.com/simonw/datasette/issues/1890#issuecomment-1317889323,https://api.github.com/repos/simonw/datasette/issues/1890,1317889323,IC_kwDOBm6k_c5OjWUr,536941,fgregg,2022-11-17T00:47:36Z,2022-11-17T00:47:36Z,CONTRIBUTOR,amazing! thanks @simonw ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1316262169,https://api.github.com/repos/simonw/datasette/issues/1890,1316262169,IC_kwDOBm6k_c5OdJEZ,9599,simonw,2022-11-16T03:22:40Z,2022-11-16T03:22:40Z,OWNER,"Actually this works as it should in desktop Safari: ![autocomplete-safari](https://user-images.githubusercontent.com/9599/202075764-fbc4b4c8-c92f-4f69-81fd-84002de5aea7.gif) I'm going to just put up with the weird behaviour in Mobile Safari.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1316242752,https://api.github.com/repos/simonw/datasette/issues/1890,1316242752,IC_kwDOBm6k_c5OdEVA,9599,simonw,2022-11-16T03:10:52Z,2022-11-16T03:12:47Z,OWNER,"https://bugs.webkit.org/show_bug.cgi?id=201768 - "" Datalist option's label not used"" - marked as RESOLVED FIXED on March 31st 2020. The commit: https://trac.webkit.org/changeset/259330/webkit And here's the test mirrored on GitHub: https://cs.github.com/qtwebkit/webkit-mirror/blob/cc3fcd0b4bad1f7cf77c26e34aa01d16618d6d5e/LayoutTests/fast/forms/datalist/datalist-option-labels.html?q=datalist-option-labels.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1316240839,https://api.github.com/repos/simonw/datasette/issues/1890,1316240839,IC_kwDOBm6k_c5OdD3H,9599,simonw,2022-11-16T03:09:11Z,2022-11-16T03:09:11Z,OWNER,"Here's a polyfill for ``: https://github.com/mfranzke/datalist-polyfill It shouldn't be necessary now that Safari has shipped support (apparently added in https://developer.apple.com/documentation/safari-release-notes/safari-12_1-release-notes#3130314 Safari 12.1 in March 2019). But it does look like Safari doesn't support differing `label` and `value` attributes, though documentation about this is hard to come by.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1316233532,https://api.github.com/repos/simonw/datasette/issues/1890,1316233532,IC_kwDOBm6k_c5OdCE8,9599,simonw,2022-11-16T03:00:58Z,2022-11-16T03:00:58Z,OWNER,"Oops, introduced a test failure: ``` def test_table_html_foreign_key_facets(app_client): response = app_client.get( ""/fixtures/foreign_key_references?_facet=foreign_key_with_blank_label"" ) assert response.status == 200 > assert ( '

' ""- 1

"" ) in response.text E assert '

- 1

' in '\n\n\n fixtures: foreign_key_references: 2 rows\n \n\n\n\n\n' E + where '\n\n\n fixtures: foreign_key_references: 2 rows\n \n\n\n\n\n' = .text ``` Need to fix this test: https://github.com/simonw/datasette/blob/eac028d3f77aa5473a5fcf59240635a1bca80f7d/tests/test_table_html.py#L616-L624","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314891228,https://api.github.com/repos/simonw/datasette/issues/1890,1314891228,IC_kwDOBm6k_c5OX6Xc,9599,simonw,2022-11-15T07:23:01Z,2022-11-15T07:23:01Z,OWNER,"Annoying: Mobile Safari doesn't seem to support separate labels and values. I should probably disable this feature on that browser, at least for foreign key facets (for the moment).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314856513,https://api.github.com/repos/simonw/datasette/issues/1890,1314856513,IC_kwDOBm6k_c5OXx5B,9599,simonw,2022-11-15T06:56:29Z,2022-11-15T06:56:29Z,OWNER,"Looks like I can fix that like so: ```html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314850524,https://api.github.com/repos/simonw/datasette/issues/1890,1314850524,IC_kwDOBm6k_c5OXwbc,9599,simonw,2022-11-15T06:48:37Z,2022-11-15T06:48:37Z,OWNER,"Spotted a bug with this on https://latest.datasette.io/fixtures/facetable?_facet=_city_id - the `_city_id` column is a foreign key, so you need to type `1` or `2` - but the autocomplete list shows the full text names for the cities.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314849867,https://api.github.com/repos/simonw/datasette/issues/1890,1314849867,IC_kwDOBm6k_c5OXwRL,9599,simonw,2022-11-15T06:47:51Z,2022-11-15T06:47:51Z,OWNER,Demo now live here: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=party - select `party` and start typing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314848432,https://api.github.com/repos/simonw/datasette/issues/1890,1314848432,IC_kwDOBm6k_c5OXv6w,9599,simonw,2022-11-15T06:46:08Z,2022-11-15T06:46:08Z,OWNER,Wrote a TIL about ``: https://til.simonwillison.net/html/datalist,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314835740,https://api.github.com/repos/simonw/datasette/issues/1890,1314835740,IC_kwDOBm6k_c5OXs0c,9599,simonw,2022-11-15T06:30:26Z,2022-11-15T06:30:26Z,OWNER,That prototype actually works really well! I'm going to add that to `table.js`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314833881,https://api.github.com/repos/simonw/datasette/issues/1890,1314833881,IC_kwDOBm6k_c5OXsXZ,9599,simonw,2022-11-15T06:27:21Z,2022-11-15T06:27:21Z,OWNER,"Here's a prototype: ```javascript function createDataLists() { var facetResults = document.querySelectorAll("".facet-results [data-column]""); Array.from(facetResults).forEach(function (facetResult) { // Use link text from all links in the facet result var linkTexts = Array.from( facetResult.querySelectorAll(""li:not(.facet-truncated) a"") ).map(function (link) { return link.textContent; }); // Create a datalist element var datalist = document.createElement(""datalist""); datalist.id = ""datalist-"" + facetResult.dataset.column; // Create an option element for each link text linkTexts.forEach(function (linkText) { var option = document.createElement(""option""); option.value = linkText; datalist.appendChild(option); }); // Add the datalist to the facet result facetResult.appendChild(datalist); }); } createDataLists(); // When any select with name=_filter_column changes, update the datalist document.body.addEventListener(""change"", function (event) { if (event.target.name === ""_filter_column"") { event.target .closest("".filter-row"") .querySelector("".filter-value"") .setAttribute(""list"", ""datalist-"" + event.target.value); } }); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314829751,https://api.github.com/repos/simonw/datasette/issues/1890,1314829751,IC_kwDOBm6k_c5OXrW3,9599,simonw,2022-11-15T06:20:50Z,2022-11-15T06:20:50Z,OWNER,"This finds the right links on the page: document.querySelectorAll('.facet-results [data-column] li:not(.facet-truncated) a')","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314825019,https://api.github.com/repos/simonw/datasette/issues/1890,1314825019,IC_kwDOBm6k_c5OXqM7,9599,simonw,2022-11-15T06:13:36Z,2022-11-15T06:13:36Z,OWNER,"This could start out as a purely JavaScript enhancement for pages that already figured out the available values through faceting, like you suggested.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314823752,https://api.github.com/repos/simonw/datasette/issues/1890,1314823752,IC_kwDOBm6k_c5OXp5I,9599,simonw,2022-11-15T06:11:49Z,2022-11-15T06:11:49Z,OWNER,"I tried this out on https://congress-legislators.datasettes.com/legislators/legislator_terms for the `party` column - here's the demo: ![datalist](https://user-images.githubusercontent.com/9599/201839812-db887ce0-c4b9-432c-8620-5ac73f222a63.gif) I made this work by dropping the following HTML into the page in the browser DevTools: ```html ``` And then adding `list=""party""` to the input element in the filter form.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314821337,https://api.github.com/repos/simonw/datasette/issues/1890,1314821337,IC_kwDOBm6k_c5OXpTZ,9599,simonw,2022-11-15T06:08:19Z,2022-11-15T06:08:19Z,OWNER,"Oh interesting... this doesn't even need to be attached to the visible faceting feature, necessarily: Datasette could try to detect when a column has a limited number of options (which the faceting code handles already) and could turn those into an auto-complete interface. There's actually a native HTML element for this these days: the `` https://developer.mozilla.org/en-US/docs/Web/HTML/Element/datalist","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1888#issuecomment-1313139657,https://api.github.com/repos/simonw/datasette/issues/1888,1313139657,IC_kwDOBm6k_c5OROvJ,9599,simonw,2022-11-14T06:04:48Z,2022-11-14T06:04:48Z,OWNER,Demo: https://latest-1-0-dev.datasette.io/-/api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447439985,API explorer should take immutability into account, https://github.com/simonw/datasette/issues/1887#issuecomment-1313113642,https://api.github.com/repos/simonw/datasette/issues/1887,1313113642,IC_kwDOBm6k_c5ORIYq,9599,simonw,2022-11-14T05:18:51Z,2022-11-14T05:18:51Z,OWNER,Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447388809,Add a confirm step to the drop table API, https://github.com/simonw/datasette/issues/1887#issuecomment-1313097713,https://api.github.com/repos/simonw/datasette/issues/1887,1313097713,IC_kwDOBm6k_c5OREfx,9599,simonw,2022-11-14T05:00:54Z,2022-11-14T05:00:54Z,OWNER,"I'm going to add a `""confirm"": true` option to the API. Without that, it returns a note about how many rows will be deleted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447388809,Add a confirm step to the drop table API, https://github.com/simonw/datasette/issues/1886#issuecomment-1356842576,https://api.github.com/repos/simonw/datasette/issues/1886,1356842576,IC_kwDOBm6k_c5Q38ZQ,18738650,stevecrawshaw,2022-12-18T17:34:20Z,2022-12-18T17:34:20Z,NONE,"A bit late to this, but I have made an app to publish air quality data in Bristol, UK. [air quality data in Bristol, UK.](https://brisaq-wfzqhmj43q-ew.a.run.app/) Next step to see if I can make a streamlit app based on this to produce some nice charts.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1321241426,https://api.github.com/repos/simonw/datasette/issues/1886,1321241426,IC_kwDOBm6k_c5OwItS,536941,fgregg,2022-11-20T20:58:54Z,2022-11-20T20:58:54Z,CONTRIBUTOR,i wrote up a blog post of how i'm using it! https://bunkum.us/2022/11/20/mgdo-stack.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1321003094,https://api.github.com/repos/simonw/datasette/issues/1886,1321003094,IC_kwDOBm6k_c5OvOhW,9020979,hydrosquall,2022-11-20T00:52:05Z,2022-11-20T00:52:05Z,NONE,"Happy birthday to datasette and thank you Simon for your continued effort on this project! I use datasette (python) as a fast layer on top of search for github projects using https://github.com/dogsheep/github-to-sqlite , and use the JSON API it provides to serve sample data to make Vega-Lite graphing workshop examples that don't require authentication/API keys. It's awesome to have a full SQL API support working without needing to develop any custom API middleware for both filtering and grouping. I've also enjoyed using it as a teaching tool for working with public dataset in [civic data workshops](https://2022.open-data.nyc/event/low-code-visual-data-exploration-with-nyc-public-data/) and as a platform for making visualization [plugins](https://github.com/hydrosquall/datasette-nteract-data-explorer) . I I'm especially excited about datasette-lite, as it will let people participate in future editions of this workshop without having to install anything to make use of their own tables :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1316289392,https://api.github.com/repos/simonw/datasette/issues/1886,1316289392,IC_kwDOBm6k_c5OdPtw,45195,rtanglao,2022-11-16T03:54:17Z,2022-11-16T03:58:56Z,NONE,"Happy Birthday Datasette! Thanks Simon!! I use datasette on everything most notably [my flickr metadata SQLite DB](https://www.dropbox.com/s/6j10e2vohp2j5kf/roland2019-2020.db?dl=0) to make art. Datasette lite on my 2019 flickr metadata is super helpful too: https://lite.datasette.io/?csv=https%3A%2F%2Fraw.githubusercontent.com%2Frtanglao%2Frt-flickr-sqlite-csv%2Fmain%2F2019-roland-flickr-metadata.csv Even better datasette lite on all firefox support questions from 2021: https://lite.datasette.io/?url=https%3A%2F%2Fraw.githubusercontent.com%2Frtanglao%2Frt-kits-api3%2Fmain%2FYEARLY_CSV_FILES%2F2021-firefox-sumo-questions.db Thanks again Simon! So great! What a gift to the world!!!!!! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1314627077,https://api.github.com/repos/simonw/datasette/issues/1886,1314627077,IC_kwDOBm6k_c5OW54F,11788561,jrdmb,2022-11-15T01:19:54Z,2022-11-15T01:19:54Z,NONE,"Datasette usage comments for its 5th anniversary celebration: I use Datasette and related tools for a Cosmology Researcher Talks database app project, which is [described in the github Readme](https://github.com/jrdmb/cosmotalks-datasette#readme) The app hosted on the Google Cloud Run service also uses other Datasette-related tools developed by Simon - datasette-render-markdown, csvs-to-sqlite, datasette-template-sql, and datasette-block-robots. This is one of two apps used for querying the talks database, each has it pros/cons as described in the github Readme. At present, over 170 different sites that host cosmology talks are scraped to collect new talks for import into the sqlite database. The shot-scraper and sqlite-utils tools are a major help for this. I also use the Mastodon API to get my favorites, toots, and boosts into a local database so I can do searches on the data. This was done on Twitter and was then extended to the Mastodon data. Again, sqlite-utils is an important tool for this. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1314455003,https://api.github.com/repos/simonw/datasette/issues/1886,1314455003,IC_kwDOBm6k_c5OWP3b,17053189,sachaj,2022-11-14T21:51:11Z,2022-11-14T21:51:11Z,NONE,"Happy Birthday Datasette! I am a librarian at the Université du Québec à Montréal (UQAM) and I've been using Datasette to publish excerpts of our library data. There are several use cases I'm working with as a proof of concept : 1. New titles list : based on reports of recent acquisitions by subject, discipline, etc. 2. List of all UQAM theses and dissertations : based on an extract of bibliographic records 3. List of all publications by UQAM Authors : based on an extract of bibliographic records See our prototype under construction here : https://datasette-bib.uqam.ca/ (some bits and pieces have been translated into French) Datasette is amazing, there is so much potential here for libraries. Thanks to Simon and all the contributors for this outstanding effort. Also sqlite-utils deserves special mention as incredibly handy and useful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1314241058,https://api.github.com/repos/simonw/datasette/issues/1886,1314241058,IC_kwDOBm6k_c5OVboi,25778,eyeseast,2022-11-14T19:06:35Z,2022-11-14T19:06:35Z,CONTRIBUTOR,"This probably counts as a case study: https://github.com/eyeseast/spatial-data-cooking-show. Even has video. Seriously, though, this workflow has become integral to my work with reporters and editors across USA TODAY Network. Very often, I get sent a folder of data in mixed formats, with a vague ask of how we should communicate some part of it to users. Datasette and its constellation of tools makes it easy to get a quick look at that data, run exploratory queries, map it and ask questions to figure out what's important to show. And then I export a version of the data that's exactly what I need for display. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1314223118,https://api.github.com/repos/simonw/datasette/issues/1886,1314223118,IC_kwDOBm6k_c5OVXQO,639730,virtadpt,2022-11-14T18:51:20Z,2022-11-14T18:51:20Z,NONE,I use Datasette to analyze blocklists by using csv-to-sqlite to pull their contents into a database and Datasette to look around through them. I also use its REST API to query said database as part of filtering out garbage from domains found in those blocklists.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1313271719,https://api.github.com/repos/simonw/datasette/issues/1886,1313271719,IC_kwDOBm6k_c5ORu-n,124274,lucapette,2022-11-14T08:25:12Z,2022-11-14T08:25:12Z,NONE,"Nothing spectacular yet but I think this falls under ""cool/cute application of datasette"": [improving fakedata performance for fun](https://lucapette.me/writing/improving-fakedata-performance-for-fun/). tl;dr I used datasette to visualize benchmarking data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1313252879,https://api.github.com/repos/simonw/datasette/issues/1886,1313252879,IC_kwDOBm6k_c5ORqYP,883348,adipasquale,2022-11-14T08:10:23Z,2022-11-14T08:10:23Z,CONTRIBUTOR,"Hi @simonw and thanks for the great tools you're publishing, your dedication is inspiring! I work for the French Ministry of Culture on a surveying tool for objects protected for their historical value. It is part of a program building modern public services called [beta.gouv.fr](https://beta.gouv.fr/). In that context I'm using data published by the Ministry that I have ingested into datasette and published on a free Fly instance : https://collectif-objets-datasette.fly.dev . I have also ingested another data set with infos about french cities on this instance so that I can perform joined queries. The surveying tool synchronizes its data regularly from this datasette instance, and I also use it to perform queries when asked generic questions about the distribution of objects. (The data is not very accessible as it's undocumented and for internal usage mostly)","{""total_count"": 3, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 3, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1313052863,https://api.github.com/repos/simonw/datasette/issues/1886,1313052863,IC_kwDOBm6k_c5OQ5i_,9599,simonw,2022-11-14T03:40:50Z,2022-11-14T03:40:50Z,OWNER,"Tim Sherratt on Twitter: https://twitter.com/wragge/status/1591930345469153282 > Where do I start? The [#GLAMWorkbench](https://twitter.com/hashtag/GLAMWorkbench?src=hashtag_click) now includes a number of examples where GLAM data is harvested, processed, and then made available for exploration via Datasette. > > https://glam-workbench.net/ > > For example the GLAM Name Index Search brings together 10+ million entries from 240 indexes and provides an aggregated search using the Datasette search-all plugin: > > https://glam-workbench.net/name-search/ > > Most recently I converted PDFs of the Tasmanian Postal Directories to a big Datasette instance: https://updates.timsherratt.org/2022/09/15/from-pdfs-to.html the process is documented and reusable.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1312898318,https://api.github.com/repos/simonw/datasette/issues/1886,1312898318,IC_kwDOBm6k_c5OQT0O,19851673,eigenfoo,2022-11-14T00:52:16Z,2022-11-14T00:52:16Z,NONE,"I'm a cryptic crossword enthusiast and have spent a lot of time scraping and parsing cryptic crossword clues from various blogs, forums and publications. The result is over **half a million clues from cryptic crosswords over the past twelve years**, including the clue, answer, puzzle date, puzzle name and a link to the original source. This is all hosted using Datasette, which has been a delight to use: https://cryptics.georgeho.org/ This dataset is a significant work of crossword archivism and scholarship, as acquiring historical crosswords and structuring their contents require focused effort and tedious cleaning that few are willing to do for such trivial data - for example, according to [this 2004 selection guide](https://cryptics.georgeho.org/static/documents/Selection_AppendixE_v2.pdf), the Library of Congress explicitly does not collect crossword puzzles. Anecdotally, I know that many constructors/setters of cryptic crosswords use this dataset as a resource, and some even simply call it ""the database"" - this is probably one of the most impactful data projects I've worked on!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1312814245,https://api.github.com/repos/simonw/datasette/issues/1886,1312814245,IC_kwDOBm6k_c5OP_Sl,2090382,noslouch,2022-11-13T20:28:26Z,2022-11-13T20:28:26Z,NONE,"I work at The Wall Street Journal as a computational journalist and serve as our self-appointed Datasette evangelist. They say that to a hammer everything looks like a nail, but the reality is newsrooms find themselves in a sea of nails! I've only got a couple public projects that I can share, but happy to offer you a look at some of the internal projects. More often than not the internal projects stay internal because the reporting doesn't lead anywhere or I can't convince an editor to greenlight it. But imho that's the beauty of datasette: a (relatively) painless mechanism to see if there's any there there. - [WSJ Inflation Tracker](wsj.com/inflationtracker) - I scraped the oscars website and turned it into a datasette instance and ran the numbers on [best actress/best picture overlap ](https://www.wsj.com/livecoverage/oscars-academy-awards-2022/card/the-best-actress-nominees-aren-t-in-any-best-pictures-contenders-when-is-the-last-time-that-happened--mDxvbLug3rq84pxLE8gY) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1884#issuecomment-1321460293,https://api.github.com/repos/simonw/datasette/issues/1884,1321460293,IC_kwDOBm6k_c5Ow-JF,15178711,asg017,2022-11-21T04:40:55Z,2022-11-21T04:40:55Z,CONTRIBUTOR,"Counting any virtual tables can be pretty tricky. On one hand, counting a [CSV virtual table](https://www.sqlite.org/csv.html) would return the number of rows in the CSV, which is helpful (but can be I/O intensive). Counting a [FTS5 virtual table](https://www.sqlite.org/fts5.html) would return the number of entries in the FTS index, which is kindof helpful, but can be misleading in some cases. On the other hand, arbitrarily running `COUNT(*)` on some virtual tables can be incredibly expensive. SQLite offers new shortcuts/pushdowns on `COUNT(*)` queries for virtual tables, and instead calls the underlying vtab implementation and iterates through all rows in the table without discretion. For example, a virtual table that's backed by a Postgres table would call `select * from pg_table`, which would use up a lot of network and CPU calls. Or a virtual table backed by a [google sheet](https://github.com/0x6b/libgsqlite) would make network/API requests to get all the rows from the sheet just to make a count. The [`pragma_table_list`](https://www.sqlite.org/pragma.html#pragma_table_list) pragma tells you when a table is a regular table or virtual (in the `type` column), but was only added in version 3.37.0 (2021-11-27). Personally, I wouldnt try to `COUNT(*)` virtual tables - it depends on how the virtual table is implemented, it requires that the connection has the proper extensions loaded, and it may accientally cause perf issues for new-age extensions. A few extensions that I'm writing have virtual tables that wouldn't benefit much from `COUNT(*)`, and the fact that SQLite iterates through all rows in a table to count just makes things worse. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/datasette/issues/1884#issuecomment-1314066229,https://api.github.com/repos/simonw/datasette/issues/1884,1314066229,IC_kwDOBm6k_c5OUw81,25778,eyeseast,2022-11-14T16:48:35Z,2022-11-14T16:48:35Z,CONTRIBUTOR,"I'm realizing I don't know if a virtual table will ever return a count. Maybe it depends on the implementation. For these three, just checking now, it'll always return zero. That said, I'm not sure there's any downside to having them return zero and caching that. (They're hidden, too.) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/datasette/issues/1884#issuecomment-1314054300,https://api.github.com/repos/simonw/datasette/issues/1884,1314054300,IC_kwDOBm6k_c5OUuCc,9599,simonw,2022-11-14T16:40:06Z,2022-11-14T16:40:06Z,OWNER,"I wonder if there are any reasons that inspect SHOULD try to count virtual tables? Like are there any likely uses for a cirial table where the count is both interesting and likely to be accessed often enough that it's worth caching? I have an issue open to add a setting to disable table counts entirely: - #1818 Maybe that should be expanded to automatically disable row counts for virtual tables entirely? Which would mean no count would be shown for them in the UI. If you desperately wanted a count you would then have to run a count(*) query against them explicitly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/datasette/issues/1884#issuecomment-1313962183,https://api.github.com/repos/simonw/datasette/issues/1884,1313962183,IC_kwDOBm6k_c5OUXjH,25778,eyeseast,2022-11-14T15:46:32Z,2022-11-14T15:46:32Z,CONTRIBUTOR,"It does work, though I think it's probably still worth excluding virtual tables that will always be zero. Here's the same inspection as before, now with `--load-extension spatialite`: ```json { ""alltheplaces"": { ""hash"": ""0843cfe414439ab903c22d1121b7ddbc643418c35c7f0edbcec82ef1452411df"", ""size"": 963375104, ""file"": ""alltheplaces.db"", ""tables"": { ""spatial_ref_sys"": { ""count"": 6215 }, ""spatialite_history"": { ""count"": 18 }, ""sqlite_sequence"": { ""count"": 2 }, ""geometry_columns"": { ""count"": 3 }, ""spatial_ref_sys_aux"": { ""count"": 6164 }, ""views_geometry_columns"": { ""count"": 0 }, ""virts_geometry_columns"": { ""count"": 0 }, ""geometry_columns_statistics"": { ""count"": 3 }, ""views_geometry_columns_statistics"": { ""count"": 0 }, ""virts_geometry_columns_statistics"": { ""count"": 0 }, ""geometry_columns_field_infos"": { ""count"": 0 }, ""views_geometry_columns_field_infos"": { ""count"": 0 }, ""virts_geometry_columns_field_infos"": { ""count"": 0 }, ""geometry_columns_time"": { ""count"": 3 }, ""geometry_columns_auth"": { ""count"": 3 }, ""views_geometry_columns_auth"": { ""count"": 0 }, ""virts_geometry_columns_auth"": { ""count"": 0 }, ""data_licenses"": { ""count"": 10 }, ""sql_statements_log"": { ""count"": 0 }, ""states"": { ""count"": 56 }, ""counties"": { ""count"": 3234 }, ""idx_states_geometry_rowid"": { ""count"": 56 }, ""idx_states_geometry_node"": { ""count"": 3 }, ""idx_states_geometry_parent"": { ""count"": 2 }, ""idx_counties_geometry_rowid"": { ""count"": 3234 }, ""idx_counties_geometry_node"": { ""count"": 98 }, ""idx_counties_geometry_parent"": { ""count"": 97 }, ""idx_places_geometry_rowid"": { ""count"": 1236796 }, ""idx_places_geometry_node"": { ""count"": 38163 }, ""idx_places_geometry_parent"": { ""count"": 38162 }, ""places"": { ""count"": 1332609 }, ""SpatialIndex"": { ""count"": 0 }, ""ElementaryGeometries"": { ""count"": 0 }, ""KNN"": { ""count"": 0 }, ""idx_states_geometry"": { ""count"": 56 }, ""idx_counties_geometry"": { ""count"": 3234 }, ""idx_places_geometry"": { ""count"": 1236796 } } } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/datasette/issues/1884#issuecomment-1311269045,https://api.github.com/repos/simonw/datasette/issues/1884,1311269045,IC_kwDOBm6k_c5OKGC1,9599,simonw,2022-11-11T06:08:28Z,2022-11-11T06:08:28Z,OWNER,Does that work if you add `--load-extension spatialite`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/datasette/issues/1884#issuecomment-1309735529,https://api.github.com/repos/simonw/datasette/issues/1884,1309735529,IC_kwDOBm6k_c5OEPpp,25778,eyeseast,2022-11-10T03:57:23Z,2022-11-10T03:57:23Z,CONTRIBUTOR,Here's how to get a list of virtual tables: https://stackoverflow.com/questions/46617118/how-to-fetch-names-of-virtual-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304320521,https://api.github.com/repos/simonw/sqlite-utils/issues/511,1304320521,IC_kwDOCGYnMM5NvloJ,7908073,chapmanjacobd,2022-11-04T22:54:09Z,2022-11-04T22:59:54Z,CONTRIBUTOR,I ran `PRAGMA integrity_check` and it returned `ok`. but then I tried restoring from a backup and I didn't get this `IntegrityError: constraint failed` error. So I think it was just something wrong with my database. If it happens again I will first try to reindex and see if that fixes the issue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1436539554,"[insert_all, upsert_all] IntegrityError: constraint failed", https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304078945,https://api.github.com/repos/simonw/sqlite-utils/issues/511,1304078945,IC_kwDOCGYnMM5Nuqph,7908073,chapmanjacobd,2022-11-04T19:38:36Z,2022-11-04T20:13:17Z,CONTRIBUTOR,"Even more bizarre, the source db only has one record and the target table has no conflicting record: ``` 875 0.3s lb:/ (main|✚2) [0|0]🌺 sqlite-utils tube_71.db 'select * from media where path = ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz""' | jq [ { ""size"": null, ""time_created"": null, ""play_count"": 1, ""language"": null, ""view_count"": null, ""width"": null, ""height"": null, ""fps"": null, ""average_rating"": null, ""live_status"": null, ""age_limit"": null, ""uploader"": null, ""time_played"": 0, ""path"": ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz"", ""id"": ""088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz/074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv"", ""ie_key"": ""ArchiveOrg"", ""playlist_path"": ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz"", ""duration"": 1424.05, ""tags"": null, ""title"": ""074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv"" } ] 875 0.3s lb:/ (main|✚2) [0|0]🥧 sqlite-utils video.db 'select * from media where path = ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz""' | jq [] ``` I've been able to use this code successfully several times before so not sure what's causing the issue. I guess the way that I'm handling multiple databases is an issue, though it hasn't ever inserted into the source db, not sure what's different. The only reasonable explanation is that it is trying to insert into the source db from the source db for some reason? Or maybe sqlite3 is checking the source db for primary key violation because the table name is the same","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1436539554,"[insert_all, upsert_all] IntegrityError: constraint failed", https://github.com/simonw/datasette/issues/1883#issuecomment-1311437901,https://api.github.com/repos/simonw/datasette/issues/1883,1311437901,IC_kwDOBm6k_c5OKvRN,31312775,mattmalcher,2022-11-11T09:20:21Z,2022-11-11T09:20:21Z,NONE,Amazing - thank you for fixing and releasing that so quickly and for showing your process! <3 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311314981,https://api.github.com/repos/simonw/datasette/issues/1883,1311314981,IC_kwDOBm6k_c5OKRQl,9599,simonw,2022-11-11T07:15:48Z,2022-11-11T07:15:48Z,OWNER,I released that fix in Datasette 0.63.1: https://docs.datasette.io/en/stable/changelog.html#v0-63-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311299535,https://api.github.com/repos/simonw/datasette/issues/1883,1311299535,IC_kwDOBm6k_c5OKNfP,9599,simonw,2022-11-11T06:54:58Z,2022-11-11T06:54:58Z,OWNER,"This time deployed with: ``` cd demos/apache-proxy fly deploy --build-arg DATASETTE_REF=8d9a957c6329d26cc1e417b5d6911640d74765eb ``` To ensure the exact commit with the fix. And that fixed it! ``` % curl -i 'https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid' HTTP/2 302 date: Fri, 11 Nov 2022 06:54:45 GMT server: Fly/b1863e2e7 (2022-11-09) location: /prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1 link: ; rel=preload content-type: text/plain x-proxied-by: Apache2 Debian via: 2 fly.io fly-request-id: 01GHJQGBSXBR7E53TY0EKMQ9PA-sjc ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311292463,https://api.github.com/repos/simonw/datasette/issues/1883,1311292463,IC_kwDOBm6k_c5OKLwv,9599,simonw,2022-11-11T06:44:24Z,2022-11-11T06:44:24Z,OWNER,"Modifying that test to the following does indeed cause a failure: ```python def test_base_url_affects_filter_redirects(app_client_base_url_prefix): response = app_client_base_url_prefix.get( ""/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid"" ) assert response.status == 302 assert ( response.headers[""location""] == ""/prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1"" ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311291632,https://api.github.com/repos/simonw/datasette/issues/1883,1311291632,IC_kwDOBm6k_c5OKLjw,9599,simonw,2022-11-11T06:43:00Z,2022-11-11T06:43:00Z,OWNER,"https://datasette-apache-proxy-demo.datasette.io/prefix/-/asgi-scope is useful: It confirms that `/prefix/` is nowhere to be seen in the incoming request data: ``` 'path': '/-/asgi-scope', 'query_string': b'', 'raw_path': b'/-/asgi-scope', ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311290115,https://api.github.com/repos/simonw/datasette/issues/1883,1311290115,IC_kwDOBm6k_c5OKLMD,9599,simonw,2022-11-11T06:40:14Z,2022-11-11T06:41:56Z,OWNER,"I modified that config file to have this line instead: ``` ProxyPass /prefix/ http://127.0.0.1:8001/ nocanon ``` And then deployed it by running: flyctl deploy --build-arg DATASETTE_REF=main This does NOT seem to have fixed the bug: ``` ~ % curl -i 'https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid' HTTP/2 302 date: Fri, 11 Nov 2022 06:40:01 GMT server: Fly/b1863e2e7 (2022-11-09) location: /fixtures/binary_data?_sort=rowid&rowid__exact=1 link: ; rel=preload content-type: text/plain x-proxied-by: Apache2 Debian via: 2 fly.io fly-request-id: 01GHJPNCF51CJ626EWZEHK2CH9-sjc ``` https://datasette-apache-proxy-demo.datasette.io/prefix/-/versions seems to confirm that this is the latest deployed version (0.63), so it looks like the deploy worked.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311286593,https://api.github.com/repos/simonw/datasette/issues/1883,1311286593,IC_kwDOBm6k_c5OKKVB,9599,simonw,2022-11-11T06:34:09Z,2022-11-11T06:34:09Z,OWNER,"https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass includes this note: > Normally, mod_proxy will canonicalise ProxyPassed URLs. But this may be incompatible with some backends, particularly those that make use of *PATH_INFO*. The optional *nocanon* keyword suppresses this and passes the URL path ""raw"" to the backend.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311284537,https://api.github.com/repos/simonw/datasette/issues/1883,1311284537,IC_kwDOBm6k_c5OKJ05,9599,simonw,2022-11-11T06:30:38Z,2022-11-11T06:30:38Z,OWNER,"Is there a chance that it's Apache that's messing with that `location:` header here, not Datasette? https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/demos/apache-proxy/000-default.conf#L7-L13","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311283301,https://api.github.com/repos/simonw/datasette/issues/1883,1311283301,IC_kwDOBm6k_c5OKJhl,9599,simonw,2022-11-11T06:28:38Z,2022-11-11T06:29:33Z,OWNER,"`path_with_added_args(request, redirect_params)` should be preserving the current path from the request. https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/datasette/utils/__init__.py#L273-L286","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311282970,https://api.github.com/repos/simonw/datasette/issues/1883,1311282970,IC_kwDOBm6k_c5OKJca,9599,simonw,2022-11-11T06:28:05Z,2022-11-11T06:28:05Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/datasette/views/table.py#L227-L249,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311280709,https://api.github.com/repos/simonw/datasette/issues/1883,1311280709,IC_kwDOBm6k_c5OKI5F,9599,simonw,2022-11-11T06:25:27Z,2022-11-11T06:25:27Z,OWNER,"I tried adding this test but it passed! I expected it to fail: ```python def test_base_url_affects_filter_redirects(app_client_base_url_prefix): response = app_client_base_url_prefix.get( ""/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid"" ) assert response.status == 302 assert ( response.headers[""location""] == ""/prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1"" ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311278678,https://api.github.com/repos/simonw/datasette/issues/1883,1311278678,IC_kwDOBm6k_c5OKIZW,9599,simonw,2022-11-11T06:22:37Z,2022-11-11T06:22:37Z,OWNER,"If you view source on that page the HTML looks correct: ```html

``` (I just added a test that confirms this too.) But... it looks like the bug is in the redirection code. https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid returns the following: location: /fixtures/binary_data?_sort=rowid&rowid__exact=1 Which is incorrect.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311273461,https://api.github.com/repos/simonw/datasette/issues/1883,1311273461,IC_kwDOBm6k_c5OKHH1,9599,simonw,2022-11-11T06:16:08Z,2022-11-11T06:16:08Z,OWNER,"Great catch, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1882#issuecomment-1314813205,https://api.github.com/repos/simonw/datasette/issues/1882,1314813205,IC_kwDOBm6k_c5OXnUV,9599,simonw,2022-11-15T06:00:41Z,2022-11-15T06:00:41Z,OWNER,"Documentation: - https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table - https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table-from-example-data Wrote a TIL about how I wrote some of those tests with Copilot: https://til.simonwillison.net/gpt3/writing-test-with-copilot","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312582512,https://api.github.com/repos/simonw/datasette/issues/1882,1312582512,IC_kwDOBm6k_c5OPGtw,9599,simonw,2022-11-12T22:11:18Z,2022-11-12T22:11:18Z,OWNER,"I like this: ```json { ""ok"": true, ""database"": ""data"", ""table"": ""agai2n"", ""table_url"": ""http://127.0.0.1:8001/data/agai2n"", ""schema"": ""CREATE TABLE [agai2n] (\n [hello] INTEGER\n)"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312581121,https://api.github.com/repos/simonw/datasette/issues/1882,1312581121,IC_kwDOBm6k_c5OPGYB,9599,simonw,2022-11-12T22:01:32Z,2022-11-12T22:01:32Z,OWNER,I'm going to change it to `table` in the output AND the input.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312581008,https://api.github.com/repos/simonw/datasette/issues/1882,1312581008,IC_kwDOBm6k_c5OPGWQ,9599,simonw,2022-11-12T22:00:52Z,2022-11-12T22:00:52Z,OWNER,"Tried out my prototype in the API explorer: The `""name""` on the output is bothering me a bit - should it be `table_name` or `table` instead? Problem is I really like `name` for the input, so should it be consistent to have the same name on the output here, or should I aim for consistency with other endpoints that might return `table` rather than the ambiguous `name` elsewhere?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312580348,https://api.github.com/repos/simonw/datasette/issues/1882,1312580348,IC_kwDOBm6k_c5OPGL8,9599,simonw,2022-11-12T21:55:54Z,2022-11-12T21:56:45Z,OWNER,"What should this API return? I think the name of the table (`name`), the URL to that table (`table_url` - for consistency with how faceting API works already) and the schema of the table (`schema`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312575048,https://api.github.com/repos/simonw/datasette/issues/1882,1312575048,IC_kwDOBm6k_c5OPE5I,9599,simonw,2022-11-12T21:22:58Z,2022-11-12T21:22:58Z,OWNER,Need to validate the table name. SQLite supports almost any table name - but they can't contain a newline character and cannot start with `sqlite_` - according to https://stackoverflow.com/questions/3694276/what-are-valid-table-names-in-sqlite/43049720#43049720,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312556044,https://api.github.com/repos/simonw/datasette/issues/1882,1312556044,IC_kwDOBm6k_c5OPAQM,9599,simonw,2022-11-12T19:29:11Z,2022-11-12T19:29:11Z,OWNER,"Thought of an edge-case: with `sqlite-utils` one feature I really like is that I can pipe data into it without caring if the table already exists or not: cat data.json | sqlite-utils insert my.db mytable - How could this new API support that? I thought about adding a `""create"": true` option to `/db/table/-/insert` which creates the table if it doesn't already exist, but if I do that I'll need to start adding other options to that endpoint - to set the primary key, add foreign keys and suchlike - which would be ignored except for the cases where the table was being created from scratch. This doesn't feel right to me - I want to keep those options here, on `/db/-/create`. One idea I had was to implement it such that you can call `/db/-/create` multiple times for the same table, but only if you are using the `""rows""` option. If so, and if the rows can be safely inserted, it would let you do that. But instead, I'm going to outsource this to the CLI tool I plan to write that feeds data into this API. I'm already planning to use that tool for CSV inserts (so the API doesn't need to accept CSV directly). I think it's a good place for other usability enhancements like ""insert this, creating the table if it does not exist"" as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1302818784,https://api.github.com/repos/simonw/datasette/issues/1882,1302818784,IC_kwDOBm6k_c5Np2_g,9599,simonw,2022-11-04T00:25:18Z,2022-11-04T16:12:39Z,OWNER,"On that basis I think the core API design should change to this: ``` POST /db/-/create Authorization: Bearer xxx Content-Type: application/json { ""name"": ""my new table"", ""columns"": [ { ""name"": ""id"", ""type"": ""integer"" }, { ""name"": ""title"", ""type"": ""text"" } ] ""pk"": ""id"" } ``` This leaves room for a `""rows"": []` key at the root too. Having that as a child of `""table""` felt unintuitive to me, and I didn't like the way this looked either: ```json { ""table"": { ""name"": ""my_new_table"" }, ""rows"": [ {""id"": 1, ""title"": ""Title""} ] } ``` Weird to have the table `name` nested inside `table` when `rows` wasn't.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1302818153,https://api.github.com/repos/simonw/datasette/issues/1882,1302818153,IC_kwDOBm6k_c5Np21p,9599,simonw,2022-11-04T00:23:58Z,2022-11-04T00:23:58Z,OWNER,"I made a decision here that this endpoint should also accept an optional `""rows"": [...]` list which is used to automatically create the table using a schema derived from those example rows (which then get inserted): - https://github.com/simonw/datasette/issues/1862#issuecomment-1302817807","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1302721916,https://api.github.com/repos/simonw/datasette/issues/1882,1302721916,IC_kwDOBm6k_c5NpfV8,9599,simonw,2022-11-03T21:58:50Z,2022-11-03T21:59:17Z,OWNER,"Mocked up a quick HTML+JavaScript form for creating that JSON structure using some iteration against Copilot prompts: ```html

/* JSON format:
{
  ""table"": {
      ""name"": ""my new table"",
      ""columns"": [
          {
              ""name"": ""id"",
              ""type"": ""integer""
          },
          {
              ""name"": ""title"",
              ""type"": ""text""
          }
      ]
     ""pk"": ""id""
  }
}

HTML form with Javascript for creating this JSON:
*/

Table name
Primary key
Column name Column type

Current columns:

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350,https://api.github.com/repos/simonw/datasette/issues/1882,1302716350,IC_kwDOBm6k_c5Npd--,9599,simonw,2022-11-03T21:51:14Z,2022-11-03T22:35:54Z,OWNER,"Validating this JSON object is getting a tiny bit complex. I'm tempted to adopt https://pydantic-docs.helpmanual.io/ at this point. The `create_model` example on https://stackoverflow.com/questions/66168517/generate-dynamic-model-using-pydantic/66168682#66168682 is particularly relevant, especially when I work on this issue: - #1863 ```python from pydantic import create_model d = {""strategy"": {""name"": ""test_strat2"", ""periods"": 10}} Strategy = create_model(""Strategy"", **d[""strategy""]) print(Strategy.schema_json(indent=2)) ``` `create_model()`: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1302715662,https://api.github.com/repos/simonw/datasette/issues/1882,1302715662,IC_kwDOBm6k_c5Npd0O,9599,simonw,2022-11-03T21:50:27Z,2022-11-03T21:50:27Z,OWNER,"API design for this: ``` POST /db/-/create Authorization: Bearer xxx Content-Type: application/json { ""table"": { ""name"": ""my new table"", ""columns"": [ { ""name"": ""id"", ""type"": ""integer"" }, { ""name"": ""title"", ""type"": ""text"" } ] ""pk"": ""id"" } } ``` Supported column types are: - `integer` - `text` - `float` (even though SQLite calls it a ""real"") - `blob` This matches my design for `sqlite-utils`: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/sqlite-utils/issues/510#issuecomment-1320394127,https://api.github.com/repos/simonw/sqlite-utils/issues/510,1320394127,IC_kwDOCGYnMM5Os52P,1176293,ar-jan,2022-11-18T18:37:51Z,2022-11-18T18:37:51Z,NONE,"I guess it is not incorrect when it says the version is `4`, though it is confusing. Maybe it doesn't even refer to FTS4/FTS5 versions, but something else? In any case, it's not related to sqlite-utils, but SQLite itself.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434911255,Cannot enable FTS5 despite it being available, https://github.com/simonw/sqlite-utils/issues/510#issuecomment-1318777114,https://api.github.com/repos/simonw/sqlite-utils/issues/510,1318777114,IC_kwDOCGYnMM5OmvEa,7908073,chapmanjacobd,2022-11-17T15:09:47Z,2022-11-17T15:09:47Z,CONTRIBUTOR,"why close? is the only problem that the _config table that incorrectly says 4 for fts5? if so, that's still something that should be fixed","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434911255,Cannot enable FTS5 despite it being available, https://github.com/simonw/sqlite-utils/issues/510#issuecomment-1318431389,https://api.github.com/repos/simonw/sqlite-utils/issues/510,1318431389,IC_kwDOCGYnMM5Olaqd,1176293,ar-jan,2022-11-17T10:36:28Z,2022-11-17T10:36:28Z,NONE,The virtual table's _config `version: 4` seems to indicate FTS5.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434911255,Cannot enable FTS5 despite it being available, https://github.com/simonw/datasette/issues/1881#issuecomment-1302813449,https://api.github.com/repos/simonw/datasette/issues/1881,1302813449,IC_kwDOBm6k_c5Np1sJ,9599,simonw,2022-11-04T00:14:07Z,2022-11-04T00:14:07Z,OWNER,"Tool is now live here: https://latest-1-0-dev.datasette.io/-/permissions Needs root perms, so access this first: https://latest-1-0-dev.datasette.io/login-as-root","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1302812918,https://api.github.com/repos/simonw/datasette/issues/1881,1302812918,IC_kwDOBm6k_c5Np1j2,9599,simonw,2022-11-04T00:13:05Z,2022-11-04T00:13:05Z,OWNER,Has tests now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301645921,https://api.github.com/repos/simonw/datasette/issues/1881,1301645921,IC_kwDOBm6k_c5NlYph,9599,simonw,2022-11-03T05:10:05Z,2022-12-09T01:38:21Z,OWNER,"I'd love to come up with a good short name for the second part of the resource two-tuple, the thing which is usually the name of a table but could also be the name of a SQL view or the name of a canned query. Idea 8th December: why not call it resource? A resource could be a thing that lives inside a database.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301639741,https://api.github.com/repos/simonw/datasette/issues/1881,1301639741,IC_kwDOBm6k_c5NlXI9,9599,simonw,2022-11-03T04:58:21Z,2022-11-03T04:58:21Z,OWNER,"The whole `database_name` or `(database_name, table_name)` tuple for resource is a bit of a code smell. Maybe this is a chance to tidy that up too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301639370,https://api.github.com/repos/simonw/datasette/issues/1881,1301639370,IC_kwDOBm6k_c5NlXDK,9599,simonw,2022-11-03T04:57:21Z,2022-11-03T04:57:21Z,OWNER,"The plugin hook would be called `register_permissions()`, for consistency with `register_routes()` and `register_commands()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301638918,https://api.github.com/repos/simonw/datasette/issues/1881,1301638918,IC_kwDOBm6k_c5NlW8G,9599,simonw,2022-11-03T04:56:06Z,2022-11-03T04:56:06Z,OWNER,"I've also introduced a new concept of a permission abbreviation, which like the permission name needs to be globally unique. That's a problem for plugins - they might just be able to guarantee that their permission long-form name is unique among other plugins (through sensible naming conventions) but the thing where they declare a initial-letters-only abbreviation is far more risky. I think abbreviations are optional - they are provided for core permissions but plugins are advised not to use them. Also Datasette could check that the installed plugins do not provide conflicting permissions on startup and refuse to start if they do.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301638156,https://api.github.com/repos/simonw/datasette/issues/1881,1301638156,IC_kwDOBm6k_c5NlWwM,9599,simonw,2022-11-03T04:54:00Z,2022-11-03T04:54:00Z,OWNER,"If I have the permissions defined like this: ```python PERMISSIONS = ( Permission(""view-instance"", ""vi"", False, False, True), Permission(""view-database"", ""vd"", True, False, True), Permission(""view-database-download"", ""vdd"", True, False, True), Permission(""view-table"", ""vt"", True, True, True), Permission(""view-query"", ""vq"", True, True, True), Permission(""insert-row"", ""ir"", True, True, False), Permission(""delete-row"", ""dr"", True, True, False), Permission(""drop-table"", ""dt"", True, True, False), Permission(""execute-sql"", ""es"", True, False, True), Permission(""permissions-debug"", ""pd"", False, False, False), Permission(""debug-menu"", ""dm"", False, False, False), ) ``` Instead of just calling them by their undeclared names in places like this: ```python await self.ds.permission_allowed( request.actor, ""execute-sql"", database, default=True ) ``` On the one hand I can ditch that confusing `default=True` option - whether a permission is on by default becomes a characteristic of that `Permission()` itself, which feels much neater. On the other hand though, plugins that introduce their own permissions - like https://datasette.io/plugins/datasette-edit-schema - will need a way to register those permissions with Datasette core. Probably another plugin hook.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301635906,https://api.github.com/repos/simonw/datasette/issues/1881,1301635906,IC_kwDOBm6k_c5NlWNC,9599,simonw,2022-11-03T04:48:09Z,2022-11-03T04:48:09Z,OWNER,"I built this prototype on the http://127.0.0.1:8001/-/allow-debug page, which is open to anyone to visit. But... I just realized that using this tool can leak information - you can use it to guess the names of invisible databases and tables and run theoretical permission checks against them. Using the tool also pollutes the list of permission checks that show up on the root-anlo `/-/permissions` page. So.... I'm going to restrict the usage of this tool to users with access to `/-/permissions` and put it on that page instead. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301635340,https://api.github.com/repos/simonw/datasette/issues/1881,1301635340,IC_kwDOBm6k_c5NlWEM,9599,simonw,2022-11-03T04:46:41Z,2022-11-03T04:46:41Z,OWNER,"Built this prototype: ![prototype](https://user-images.githubusercontent.com/9599/199649219-f146e43b-bfb5-45e6-9777-956f21a79887.gif) In building it I realized I needed to know which permissions took a table, a database, both or neither. So I had to bake that into the code. Here's the prototype so far (which includes a prototype of the logic for the `_r` field on actor, see #1855): ```diff diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 32b0c758..f68aa38f 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -6,8 +6,8 @@ import json import time -@hookimpl(tryfirst=True) -def permission_allowed(datasette, actor, action, resource): +@hookimpl(tryfirst=True, specname=""permission_allowed"") +def permission_allowed_default(datasette, actor, action, resource): async def inner(): if action in ( ""permissions-debug"", @@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource): return inner +@hookimpl(specname=""permission_allowed"") +def permission_allowed_actor_restrictions(actor, action, resource): + if actor is None: + return None + _r = actor.get(""_r"") + if not _r: + # No restrictions, so we have no opinion + return None + action_initials = """".join([word[0] for word in action.split(""-"")]) + # If _r is defined then we use those to further restrict the actor + # Crucially, we only use this to say NO (return False) - we never + # use it to return YES (True) because that might over-ride other + # restrictions placed on this actor + all_allowed = _r.get(""a"") + if all_allowed is not None: + assert isinstance(all_allowed, list) + if action_initials in all_allowed: + return None + # How about for the current database? + if action in (""view-database"", ""view-database-download"", ""execute-sql""): + database_allowed = _r.get(""d"", {}).get(resource) + if database_allowed is not None: + assert isinstance(database_allowed, list) + if action_initials in database_allowed: + return None + # Or the current table? That's any time the resource is (database, table) + if not isinstance(resource, str) and len(resource) == 2: + database, table = resource + table_allowed = _r.get(""t"", {}).get(database, {}).get(table) + # TODO: What should this do for canned queries? + if table_allowed is not None: + assert isinstance(table_allowed, list) + if action_initials in table_allowed: + return None + # This action is not specifically allowed, so reject it + return False + + @hookimpl def actor_from_request(datasette, request): prefix = ""dstok_"" diff --git a/datasette/templates/allow_debug.html b/datasette/templates/allow_debug.html index 0f1b30f0..ae43f0f5 100644 --- a/datasette/templates/allow_debug.html +++ b/datasette/templates/allow_debug.html @@ -35,7 +35,7 @@ p.message-warning {

Use this tool to try out different actor and allow combinations. See Defining permissions with ""allow"" blocks for documentation.

Allow block

@@ -55,4 +55,82 @@ p.message-warning { {% if result == ""False"" %}

Result: deny

{% endif %} +

Test permission check

+ +

This tool lets you simulate an actor and a permission check for that actor.

+ + + +

Actor

+ +

Permission check

Permission + +

Database name

Table or query name

+ +

+ + + + + {% endblock %} diff --git a/datasette/views/special.py b/datasette/views/special.py index 9922a621..d46fc280 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -1,6 +1,8 @@ import json +from datasette.permissions import PERMISSIONS from datasette.utils.asgi import Response, Forbidden from datasette.utils import actor_matches_allow, add_cors_headers +from datasette.permissions import PERMISSIONS from .base import BaseView import secrets import time @@ -138,9 +140,34 @@ class AllowDebugView(BaseView): ""error"": ""\n\n"".join(errors) if errors else """", ""actor_input"": actor_input, ""allow_input"": allow_input, + ""permissions"": PERMISSIONS, }, ) + async def post(self, request): + vars = await request.post_vars() + actor = json.loads(vars[""actor""]) + permission = vars[""permission""] + resource_1 = vars[""resource_1""] + resource_2 = vars[""resource_2""] + resource = [] + if resource_1: + resource.append(resource_1) + if resource_2: + resource.append(resource_2) + resource = tuple(resource) + result = await self.ds.permission_allowed( + actor, permission, resource, default=""USE_DEFAULT"" + ) + return Response.json( + { + ""actor"": actor, + ""permission"": permission, + ""resource"": resource, + ""result"": result, + } + ) + class MessagesDebugView(BaseView): name = ""messages_debug"" ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1880#issuecomment-1317420812,https://api.github.com/repos/simonw/datasette/issues/1880,1317420812,IC_kwDOBm6k_c5Ohj8M,525934,amitkoth,2022-11-16T17:50:29Z,2022-11-16T17:50:29Z,NONE,"I appreciate your response @simonw - thanks! I'll clarify what we need further - let's imagine we have 2000 SQLLite databases (for 2000 tenants), but we only want to run _one_ datasette instance for each of those tenants to query/use datasette against their _own_ database only. This means the ""connection"" between datasette and the SQLLite database would be dynamic, based on the tenantID that's required on an incoming request. Is there any specific config or other considerations in this use case, to minimize memory use on a single, efficient VM and serve queries to all these tenants? cc @muadham","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1433576351,Datasette with many and large databases > Memory use, https://github.com/simonw/datasette/issues/1880#issuecomment-1311273063,https://api.github.com/repos/simonw/datasette/issues/1880,1311273063,IC_kwDOBm6k_c5OKHBn,9599,simonw,2022-11-11T06:15:28Z,2022-11-11T06:15:28Z,OWNER,"The `_internal` database is intended to help Datasette handle much larger attached databases. Right now Datasette attempts to show every database on the https://latest.datasette.io/ index page and every table on the https://latest.datasette.io/fixtures database index page - but these are not paginated. If you had a database containing 1,000 tables the database index page would get pretty slow. So I want to be able to paginate (and search) those. But to paginate them it's useful to have them in a database table itself, since then I can paginate using SQL. My plan for `_internal` is to use it to implement those advanced browsing features. I've not completed this work yet though. See this issue for more details on that: - #417","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1433576351,Datasette with many and large databases > Memory use, https://github.com/simonw/datasette/issues/1880#issuecomment-1311271298,https://api.github.com/repos/simonw/datasette/issues/1880,1311271298,IC_kwDOBm6k_c5OKGmC,9599,simonw,2022-11-11T06:12:29Z,2022-11-11T06:12:29Z,OWNER,"I think you may have misunderstood this feature. This is talking about the `_internal` in-memory database, which maintains a set of tables that list the databases and tables that are attached to Datasette. They're not a copy of the data itself - just a list of table names, column names and database names. You can see what that database looks like by signing in as root - running `datasette --root` and clicking the link. Or you can see an example here: - Click the button on https://latest.datasette.io/login-as-root - Now visit https://latest.datasette.io/_internal For the example instance that looks like this: The two most interesting tables in there are these ones: As you can see, it's just the table schema itself and the columns that make up the tables. Even if you have hundreds of databases connected each with hundreds of tables this should still only add up to a few MB of RAM.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1433576351,Datasette with many and large databases > Memory use, https://github.com/simonw/datasette/issues/1880#issuecomment-1301043042,https://api.github.com/repos/simonw/datasette/issues/1880,1301043042,IC_kwDOBm6k_c5NjFdi,525934,amitkoth,2022-11-02T18:20:14Z,2022-11-02T18:20:14Z,NONE,"Follow on question - is all memory use @simonw - for both datasette and SQLlite confined to the ""query time"" itself i.e. the memory use is relevant only to a particular transaction or query - and then subsequently released?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1433576351,Datasette with many and large databases > Memory use, https://github.com/simonw/datasette/issues/1879#issuecomment-1299102755,https://api.github.com/repos/simonw/datasette/issues/1879,1299102755,IC_kwDOBm6k_c5Nbrwj,9599,simonw,2022-11-01T20:31:37Z,2022-11-01T20:31:37Z,OWNER,And some JavaScript that can spot if Datasette thinks it is being served over HTTP when it's actually being served over HTTPS.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1879#issuecomment-1299102108,https://api.github.com/repos/simonw/datasette/issues/1879,1299102108,IC_kwDOBm6k_c5Nbrmc,9599,simonw,2022-11-01T20:30:54Z,2022-11-01T20:33:06Z,OWNER,One idea: add a `/-/debug` page (or `/-/tips` or `/-/checks`) which shows the incoming requests headers and could even detect if there's an `x-forwarded-host` header that isn't being repeated and show a tip on how to fix that.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1879#issuecomment-1299098458,https://api.github.com/repos/simonw/datasette/issues/1879,1299098458,IC_kwDOBm6k_c5Nbqta,9599,simonw,2022-11-01T20:27:40Z,2022-11-01T20:33:52Z,OWNER,"https://github.com/simonw/datasette-x-forwarded-host/blob/main/datasette_x_forwarded_host/__init__.py could happen in core controlled by: `--setting trust_forwarded_host 1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1879#issuecomment-1299096850,https://api.github.com/repos/simonw/datasette/issues/1879,1299096850,IC_kwDOBm6k_c5NbqUS,9599,simonw,2022-11-01T20:26:12Z,2022-11-01T20:26:12Z,OWNER,"The other relevant plugin here is https://datasette.io/plugins/datasette-x-forwarded-host Maybe that should be rolled into core too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1879#issuecomment-1299090678,https://api.github.com/repos/simonw/datasette/issues/1879,1299090678,IC_kwDOBm6k_c5Nboz2,9599,simonw,2022-11-01T20:20:28Z,2022-11-01T20:20:28Z,OWNER,My first step in debugging these is to install https://datasette.io/plugins/datasette-debug-asgi - but now I'm thinking maybe something like that should be part of core.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1878#issuecomment-1336100218,https://api.github.com/repos/simonw/datasette/issues/1878,1336100218,IC_kwDOBm6k_c5Po0V6,9599,simonw,2022-12-03T07:02:15Z,2022-12-03T07:02:15Z,OWNER,"Moved this work to a PR: - #1931","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1878#issuecomment-1336094562,https://api.github.com/repos/simonw/datasette/issues/1878,1336094562,IC_kwDOBm6k_c5Poy9i,9599,simonw,2022-12-03T06:27:50Z,2022-12-03T06:29:06Z,OWNER,"This adds it to the API explorer: ```diff diff --git a/datasette/views/special.py b/datasette/views/special.py index 1f84b094..1b4a9d3c 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -316,21 +316,37 @@ class ApiExplorerView(BaseView): request.actor, ""insert-row"", (name, table) ): pks = await db.primary_keys(table) - table_links.append( - { - ""path"": self.ds.urls.table(name, table) + ""/-/insert"", - ""method"": ""POST"", - ""label"": ""Insert rows into {}"".format(table), - ""json"": { - ""rows"": [ - { - column: None - for column in await db.table_columns(table) - if column not in pks - } - ] + table_links.extend( + [ + { + ""path"": self.ds.urls.table(name, table) + ""/-/insert"", + ""method"": ""POST"", + ""label"": ""Insert rows into {}"".format(table), + ""json"": { + ""rows"": [ + { + column: None + for column in await db.table_columns(table) + if column not in pks + } + ] + }, }, - } + { + ""path"": self.ds.urls.table(name, table) + ""/-/upsert"", + ""method"": ""POST"", + ""label"": ""Upsert rows into {}"".format(table), + ""json"": { + ""rows"": [ + { + column: None + for column in await db.table_columns(table) + if column not in pks + } + ] + }, + }, + ] ) if await self.ds.permission_allowed( request.actor, ""drop-table"", (name, table) ``` Except it doesn't quite, because the example JSON this generates is invalid as it does not include the primary key column. (Made me notice that the way example columns are created for `/-/insert` will fail for tables that don't have an auto-incrementing primary key)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1878#issuecomment-1336094470,https://api.github.com/repos/simonw/datasette/issues/1878,1336094470,IC_kwDOBm6k_c5Poy8G,9599,simonw,2022-12-03T06:27:13Z,2022-12-03T06:27:13Z,OWNER,"Tests are going to need to cover both rowid-only and compound primary key tables, including all of the error states.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1878#issuecomment-1336094381,https://api.github.com/repos/simonw/datasette/issues/1878,1336094381,IC_kwDOBm6k_c5Poy6t,9599,simonw,2022-12-03T06:26:25Z,2022-12-03T06:26:25Z,OWNER,"Initial prototype: ```diff diff --git a/datasette/app.py b/datasette/app.py index 125b4969..282c0984 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -40,7 +40,7 @@ from .views.special import ( PermissionsDebugView, MessagesDebugView, ) -from .views.table import TableView, TableInsertView, TableDropView +from .views.table import TableView, TableInsertView, TableUpsertView, TableDropView from .views.row import RowView, RowDeleteView, RowUpdateView from .renderer import json_renderer from .url_builder import Urls @@ -1292,6 +1292,10 @@ class Datasette: TableInsertView.as_view(self), r""/(?P[^\/\.]+)/(?P[^\/\.]+)/-/insert$"", ) + add_route( + TableUpsertView.as_view(self), + r""/(?P[^\/\.]+)/(?P

[^\/\.]+)/-/upsert$"", + ) add_route( TableDropView.as_view(self), r""/(?P[^\/\.]+)/(?P

[^\/\.]+)/-/drop$"", diff --git a/datasette/views/table.py b/datasette/views/table.py index 7ba78c11..ae0d6366 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1074,9 +1074,15 @@ class TableInsertView(BaseView): def __init__(self, datasette): self.ds = datasette - async def _validate_data(self, request, db, table_name): + async def _validate_data(self, request, db, table_name, pks, upsert): errors = [] + pks_list = [] + if isinstance(pks, str): + pks_list = [pks] + else: + pks_list = list(pks) + def _errors(errors): return None, errors, {} @@ -1135,6 +1141,15 @@ class TableInsertView(BaseView): # Validate columns of each row columns = set(await db.table_columns(table_name)) for i, row in enumerate(rows): + if upsert: + # It MUST have the primary key + missing_pks = [pk for pk in pks_list if pk not in row] + if missing_pks: + errors.append( + 'Row {} is missing primary key column(s): ""{}""'.format( + i, '"", ""'.join(missing_pks) + ) + ) invalid_columns = set(row.keys()) - columns if invalid_columns: errors.append( @@ -1146,7 +1161,7 @@ class TableInsertView(BaseView): return _errors(errors) return rows, errors, extras - async def post(self, request): + async def post(self, request, upsert=False): try: resolved = await self.ds.resolve_table(request) except NotFound as e: @@ -1164,7 +1179,12 @@ class TableInsertView(BaseView): request.actor, ""insert-row"", resource=(database_name, table_name) ): return _error([""Permission denied""], 403) - rows, errors, extras = await self._validate_data(request, db, table_name) + + pks = await db.primary_keys(table_name) + + rows, errors, extras = await self._validate_data( + request, db, table_name, pks, upsert + ) if errors: return _error(errors, 400) @@ -1172,15 +1192,19 @@ class TableInsertView(BaseView): replace = extras.get(""replace"") should_return = bool(extras.get(""return"", False)) - # Insert rows - def insert_rows(conn): + + def insert_or_upsert_rows(conn): table = sqlite_utils.Database(conn)[table_name] + kwargs = {} + if upsert: + kwargs[""pk""] = pks[0] if len(pks) == 1 else pks + else: + kwargs = {""ignore"": ignore, ""replace"": replace} if should_return: rowids = [] + method = table.upsert if upsert else table.insert for row in rows: - rowids.append( - table.insert(row, ignore=ignore, replace=replace).last_rowid - ) + rowids.append(method(row, **kwargs).last_rowid) return list( table.rows_where( ""rowid in ({})"".format("","".join(""?"" for _ in rowids)), @@ -1188,10 +1212,11 @@ class TableInsertView(BaseView): ) ) else: - table.insert_all(rows, ignore=ignore, replace=replace) + method_all = table.upsert_all if upsert else table.insert_all + method_all(rows, **kwargs) try: - rows = await db.execute_write_fn(insert_rows) + rows = await db.execute_write_fn(insert_or_upsert_rows) except Exception as e: return _error([str(e)]) result = {""ok"": True} @@ -1200,6 +1225,13 @@ class TableInsertView(BaseView): return Response.json(result, status=201) +class TableUpsertView(TableInsertView): + name = ""table-upsert"" + + async def post(self, request): + return await super().post(request, upsert=True) + + class TableDropView(BaseView): name = ""table-drop"" ``` Manual testing reveals that this mostly works... but it's not doing the right thing for `""return"": true` - it always returns an empty list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1878#issuecomment-1336073212,https://api.github.com/repos/simonw/datasette/issues/1878,1336073212,IC_kwDOBm6k_c5Potv8,9599,simonw,2022-12-03T05:38:49Z,2022-12-03T05:38:49Z,OWNER,And on Discord today: https://discord.com/channels/823971286308356157/823971286941302908/1048426072066236536,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1878#issuecomment-1336070843,https://api.github.com/repos/simonw/datasette/issues/1878,1336070843,IC_kwDOBm6k_c5PotK7,9599,simonw,2022-12-03T05:37:53Z,2022-12-03T05:37:53Z,OWNER,Also requested here: https://news.ycombinator.com/item?id=33839894,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1878#issuecomment-1312534826,https://api.github.com/repos/simonw/datasette/issues/1878,1312534826,IC_kwDOBm6k_c5OO7Eq,18738650,stevecrawshaw,2022-11-12T17:34:58Z,2022-11-12T17:34:58Z,NONE,"Hi Simon. I have just started experimenting with datasette in earnest, looking at it's suitability for air quality open data. A bulk upsert \ upsert_all would be very useful for me in enabling real time data to be pushed from a sql server database with FME server to a datasette db. An hourly process queries the last 2 hours of data and pushes that to my database, inserting new data and updating existing combinations of pk siteid and date_time. This is already implemented on our current [open data portal](https://opendata.bristol.gov.uk/explore/dataset/air-quality-data-continuous/table/?disjunctive.location&sort=date_time). Excited to see your progress with this! Thank you for this amazing software.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1878#issuecomment-1299071456,https://api.github.com/repos/simonw/datasette/issues/1878,1299071456,IC_kwDOBm6k_c5NbkHg,9599,simonw,2022-11-01T20:02:43Z,2022-11-01T20:02:43Z,OWNER,"Note that ""update"" is partially covered by the `replace` option to `/-/insert`, added here: - https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1877#issuecomment-1331201207,https://api.github.com/repos/simonw/datasette/issues/1877,1331201207,IC_kwDOBm6k_c5PWIS3,9599,simonw,2022-11-29T19:44:07Z,2022-11-29T19:44:07Z,OWNER,"I fixed the duplicate logic issue here: https://github.com/simonw/datasette/commit/ee64130fa8a5ff4a24791916c696e10cf2375102 - #1896 Decided not to address `views/table.py`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432012302,Refactor and tidy up final write API code, https://github.com/simonw/datasette/issues/1876#issuecomment-1298856054,https://api.github.com/repos/simonw/datasette/issues/1876,1298856054,IC_kwDOBm6k_c5Navh2,9599,simonw,2022-11-01T17:16:01Z,2022-11-01T17:16:01Z,OWNER,`ta.style.height = ta.scrollHeight + 'px'` is an easy way to do that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1431786951,SQL query should wrap on SQL interrupted screen, https://github.com/simonw/datasette/issues/1876#issuecomment-1298854321,https://api.github.com/repos/simonw/datasette/issues/1876,1298854321,IC_kwDOBm6k_c5NavGx,9599,simonw,2022-11-01T17:14:33Z,2022-11-01T17:14:33Z,OWNER,"I could use a `textarea` here (would need to figure out a neat pattern to expand it to fit the query): ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1431786951,SQL query should wrap on SQL interrupted screen, https://github.com/simonw/datasette/issues/1875#issuecomment-1314620086,https://api.github.com/repos/simonw/datasette/issues/1875,1314620086,IC_kwDOBm6k_c5OW4K2,9599,simonw,2022-11-15T01:09:56Z,2022-11-15T01:09:56Z,OWNER,"Rough initial prototype: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 8b987221..518ac578 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1103,19 +1103,30 @@ class TableInsertView(BaseView): except json.JSONDecodeError as e: return _errors([""Invalid JSON: {}"".format(e)]) if not isinstance(data, dict): - return _errors([""JSON must be a dictionary""]) + return _errors([{""detail"": ""JSON must be a dictionary"", ""pointer"": ""#/""}]) keys = data.keys() # keys must contain ""row"" or ""rows"" if ""row"" not in keys and ""rows"" not in keys: return _errors(['JSON must have one or other of ""row"" or ""rows""']) rows = [] + was_single_row = False if ""row"" in keys: if ""rows"" in keys: - return _errors(['Cannot use ""row"" and ""rows"" at the same time']) + return _errors( + [ + { + ""detail"": 'Cannot use ""row"" and ""rows"" at the same time', + ""pointer"": ""#/row"", + } + ] + ) + was_single_row = True row = data[""row""] if not isinstance(row, dict): - return _errors(['""row"" must be a dictionary']) + return _errors( + [{""detail"": '""row"" must be a dictionary', ""pointer"": ""#/row""}] + ) rows = [row] data[""return""] = True else: @@ -1152,9 +1163,12 @@ class TableInsertView(BaseView): invalid_columns = set(row.keys()) - columns if invalid_columns: errors.append( - ""Row {} has invalid columns: {}"".format( - i, "", "".join(sorted(invalid_columns)) - ) + { + ""detail"": ""Invalid columns: {}"".format( + "", "".join(sorted(invalid_columns)) + ), + ""pointer"": ""#/blah/"", + } ) if errors: return _errors(errors) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314615592,https://api.github.com/repos/simonw/datasette/issues/1875,1314615592,IC_kwDOBm6k_c5OW3Eo,9599,simonw,2022-11-15T01:04:28Z,2022-11-15T01:04:28Z,OWNER,"Worth noting this bit in RFC 7807: > The fictional problem type here defines the ""errors"" extension, an > array that describes the details of each validation error. Each > member is an object containing ""detail"" to describe the issue, and > ""pointer"" to locate the problem within the request's content using a > JSON Pointer [JSON-POINTER]. So the list of `""errors""` with JSON Pointer isn't technically part of the spec, it's an imaginary extension. It fits what I need to do though, so I'm inclined to stick with it anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314545407,https://api.github.com/repos/simonw/datasette/issues/1875,1314545407,IC_kwDOBm6k_c5OWl7_,9599,simonw,2022-11-14T23:30:34Z,2022-11-14T23:30:34Z,OWNER,TIL: https://til.simonwillison.net/json/json-pointer,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314491884,https://api.github.com/repos/simonw/datasette/issues/1875,1314491884,IC_kwDOBm6k_c5OWY3s,9599,simonw,2022-11-14T22:26:11Z,2022-11-14T22:26:54Z,OWNER,"Spec looks pretty simple: > A JSON Pointer is a Unicode string (see [RFC4627], Section 3) > containing a sequence of zero or more reference tokens, each prefixed > by a `/` (%x2F) character. > > Because the characters `~` (%x7E) and `/` (%x2F) have special > meanings in JSON Pointer, `~` needs to be encoded as `~0` and `/` > needs to be encoded as `~1` when these characters appear in a > reference token.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314491150,https://api.github.com/repos/simonw/datasette/issues/1875,1314491150,IC_kwDOBm6k_c5OWYsO,9599,simonw,2022-11-14T22:25:20Z,2022-11-14T22:25:20Z,OWNER,"That's using JSON Pointer: https://www.rfc-editor.org/rfc/rfc6901 There's a Python library for that here https://github.com/stefankoegl/python-json-pointer/blob/master/jsonpointer.py - which looks simple and clean and well maintained and documented, but it only handles the ""what is at this pointer within this JSON object"" case - I need to generate the correct JSON pointer to explain where my error is. So I think I'll end up hand-rolling this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314488010,https://api.github.com/repos/simonw/datasette/issues/1875,1314488010,IC_kwDOBm6k_c5OWX7K,9599,simonw,2022-11-14T22:21:43Z,2022-11-14T22:21:43Z,OWNER,"Here's the most relevant example from the RFC spec: ``` POST /details HTTP/1.1 Host: account.example.com Accept: application/json ``` ```json { ""age"": 42.3, ""profile"": { ""color"": ""yellow"" } } ``` ``` HTTP/1.1 400 Bad Request Content-Type: application/problem+json Content-Language: en ``` ```json { ""type"": ""https://example.net/validation-error"", ""title"": ""Your request is not valid."", ""errors"": [ { ""detail"": ""must be a positive integer"", ""pointer"": ""#/age"" }, { ""detail"": ""must be 'green', 'red' or 'blue'"", ""pointer"": ""#/profile/color"" } ] } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1313114283,https://api.github.com/repos/simonw/datasette/issues/1875,1313114283,IC_kwDOBm6k_c5ORIir,9599,simonw,2022-11-14T05:20:00Z,2022-11-14T05:20:00Z,OWNER,"I started a conversation about JSON error standards on Mastodon here: https://fedi.simonwillison.net/web/@simon/109338725610487457 Quite a few people pointed to this RFC independently.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/sqlite-utils/pull/508#issuecomment-1297788531,https://api.github.com/repos/simonw/sqlite-utils/issues/508,1297788531,IC_kwDOCGYnMM5NWq5z,7908073,chapmanjacobd,2022-10-31T22:54:33Z,2022-11-17T15:11:16Z,CONTRIBUTOR,"Maybe this is actually a problem in the python sqlite bindings. Given [SQLITE's stance on this](https://www.sqlite.org/invalidutf.html) they should probably use `encode('utf-8', 'surrogatepass')`. As far as I understand the error here won't actually be resolved by this PR as-is. We would need to modify the data with `surrogateescape`... :/ or modify the sqlite3 module to use `surrogatepass`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430563092,Allow surrogates in parameters, https://github.com/simonw/sqlite-utils/pull/508#issuecomment-1297754631,https://api.github.com/repos/simonw/sqlite-utils/issues/508,1297754631,IC_kwDOCGYnMM5NWioH,22429695,codecov[bot],2022-10-31T22:14:48Z,2022-10-31T22:53:59Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **96.25**% // Head: **96.09**% // Decreases project coverage by **`-0.15%`** :warning: > Coverage data is based on head [(`2d6a149`)](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`529110e`)](https://codecov.io/gh/simonw/sqlite-utils/commit/529110e7d8c4a6b1bbf5fb61f2e29d72aa95a611?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 63.63% of modified lines in pull request are covered. > :exclamation: Current head 2d6a149 differs from pull request most recent head 43a8c4c. Consider uploading reports for the commit 43a8c4c to get more accurate results

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #508 +/- ## ========================================== - Coverage 96.25% 96.09% -0.16% ========================================== Files 4 4 Lines 2401 2407 +6 ========================================== + Hits 2311 2313 +2 - Misses 90 94 +4 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/508/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `96.79% <63.63%> (-0.30%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430563092,Allow surrogates in parameters, https://github.com/simonw/sqlite-utils/issues/507#issuecomment-1297859539,https://api.github.com/repos/simonw/sqlite-utils/issues/507,1297859539,IC_kwDOCGYnMM5NW8PT,7908073,chapmanjacobd,2022-11-01T00:40:16Z,2022-11-01T00:40:16Z,CONTRIBUTOR,"Ideally people could fix their data if they run into this issue. If you are using filenames try [convmv](https://linux.die.net/man/1/convmv) ``` convmv --preserve-mtimes -f utf8 -t utf8 --notest -i -r . ``` maybe this script will also help: ```py import argparse, shutil from pathlib import Path import ftfy from xklb import utils from xklb.utils import log def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument(""paths"", nargs='*') parser.add_argument(""--verbose"", ""-v"", action=""count"", default=0) args = parser.parse_args() log.info(utils.dict_filter_bool(args.__dict__)) return args def rename_invalid_paths() -> None: args = parse_args() for path in args.paths: log.info(path) for p in sorted([str(p) for p in Path(path).rglob(""*"")], key=len): fixed = ftfy.fix_text(p, uncurl_quotes=False).replace(""\r\n"", ""\n"").replace(""\r"", ""\n"").replace(""\n"", """") if p != fixed: try: shutil.move(p, fixed) except FileNotFoundError: log.warning(""FileNotFound. %s"", p) else: log.info(fixed) if __name__ == ""__main__"": rename_invalid_paths() ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430325103,conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character, https://github.com/simonw/datasette/issues/1874#issuecomment-1313127054,https://api.github.com/repos/simonw/datasette/issues/1874,1313127054,IC_kwDOBm6k_c5ORLqO,9599,simonw,2022-11-14T05:45:00Z,2022-11-14T05:45:00Z,OWNER,"Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=&method=POST ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,API to drop a table, https://github.com/simonw/datasette/issues/1874#issuecomment-1313125123,https://api.github.com/repos/simonw/datasette/issues/1874,1313125123,IC_kwDOBm6k_c5ORLMD,9599,simonw,2022-11-14T05:41:20Z,2022-11-14T05:42:23Z,OWNER,"I also changed the confirmation JSON returned by this endpoint to add the `database` and `table` like so: ```json { ""ok"": true, ""database"": ""data"", ""table"": ""docs"", ""row_count"": 1, ""message"": ""Pass \""confirm\"": true to confirm"" } ``` Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,API to drop a table, https://github.com/simonw/datasette/issues/1874#issuecomment-1313119558,https://api.github.com/repos/simonw/datasette/issues/1874,1313119558,IC_kwDOBm6k_c5ORJ1G,9599,simonw,2022-11-14T05:30:27Z,2022-11-14T05:30:27Z,OWNER,Found a bug: you get a 500 error if you try this against an immutable database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,API to drop a table, https://github.com/simonw/datasette/issues/1874#issuecomment-1296363981,https://api.github.com/repos/simonw/datasette/issues/1874,1296363981,IC_kwDOBm6k_c5NRPHN,9599,simonw,2022-10-30T22:19:47Z,2022-10-30T22:19:47Z,OWNER,Documentation: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,API to drop a table, https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298879701,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1298879701,IC_kwDOCGYnMM5Na1TV,9599,simonw,2022-11-01T17:37:13Z,2022-11-01T17:37:13Z,OWNER,"The question I was originally trying to answer here was this: how many rows were actually inserted by that call to `.insert_all()`? I don't know that `.rowcount` would ever be useful here, since the ""correct"" answer depends on other factors - had I determined to ignore or replace records with a primary key that matches an existing record for example? So I think if people need `rowcount` they can get it by using a `cursor` directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298877872,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1298877872,IC_kwDOCGYnMM5Na02w,9599,simonw,2022-11-01T17:35:30Z,2022-11-01T17:35:30Z,OWNER,"This may not make sense. First, `.last_rowid` is a property on table - but that doesn't make sense for `rowcount` since it should clearly be a property on the database itself (you can run a query directly using `db.execute()` without going through a `Table` object). So I tried this prototype: ```diff diff --git a/docs/python-api.rst b/docs/python-api.rst index 206e5e6..78d3a8d 100644 --- a/docs/python-api.rst +++ b/docs/python-api.rst @@ -186,6 +186,15 @@ The ``db.query(sql)`` function executes a SQL query and returns an iterator over # {'name': 'Cleo'} # {'name': 'Pancakes'} +After executing a query the ``db.rowcount`` property on that database instance will reflect the number of rows affected by any insert, update or delete operations performed by that query: + +.. code-block:: python + + db = Database(memory=True) + db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}]) + print(db.rowcount) + # Outputs: 2 + .. _python_api_execute: db.execute(sql, params) diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a06f4b7..c19c2dd 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -294,6 +294,8 @@ class Database: _counts_table_name = ""_counts"" use_counts_table = False + # Number of rows inserted, updated or deleted + rowcount: Optional[int] = None def __init__( self, @@ -480,9 +482,11 @@ class Database: if self._tracer: self._tracer(sql, parameters) if parameters is not None: - return self.conn.execute(sql, parameters) + cursor = self.conn.execute(sql, parameters) else: - return self.conn.execute(sql) + cursor = self.conn.execute(sql) + self.rowcount = cursor.rowcount + return cursor def executescript(self, sql: str) -> sqlite3.Cursor: """""" ``` But this happens: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}])

>>> db.rowcount -1 ``` Turning on query tracing demonstrates why: ```pycon >>> db = Database(memory=True, tracer=print) PRAGMA recursive_triggers=on; None >>> db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}]) select name from sqlite_master where type = 'view' None select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'view' None CREATE TABLE [dogs] ( [name] TEXT ); None select name from sqlite_master where type = 'view' None INSERT INTO [dogs] ([name]) VALUES (?), (?); ['Cleo', 'Pancakes'] select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'table' None PRAGMA table_info([dogs]) None

>>> ``` The `.insert_all()` function does a bunch of other queries too, so `.rowcount` is quickly over-ridden by the same result from extra queries that it executed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1296358636,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1296358636,IC_kwDOCGYnMM5NRNzs,9599,simonw,2022-10-30T21:52:11Z,2022-10-30T21:52:11Z,OWNER,This could work in a similar way to `db.insert(...).last_rowid`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/datasette/issues/1873#issuecomment-1298919552,https://api.github.com/repos/simonw/datasette/issues/1873,1298919552,IC_kwDOBm6k_c5Na_CA,9599,simonw,2022-11-01T18:11:27Z,2022-11-01T18:11:27Z,OWNER,"I forgot to document `ignore` and `replace`. Also I need to add tests that cover: - Forgetting to include a primary key on a non-autoincrement table - Compound primary keys - Rowid only tables with and without rowid specified I think my validation logic here will get caught out by the fact that `rowid` does not show up as a valid column name: https://github.com/simonw/datasette/blob/9bec7c38eb93cde5afb16df9bdd96aea2a5b0459/datasette/views/table.py#L1151-L1160 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1298905135,https://api.github.com/repos/simonw/datasette/issues/1873,1298905135,IC_kwDOBm6k_c5Na7gv,9599,simonw,2022-11-01T17:59:59Z,2022-11-01T17:59:59Z,OWNER,"It's a bit surprising that you can send `""ignore"": true, ""return_rows"": true` and the returned `""inserted""` key will list rows that were NOT inserted (since they were ignored). Three options: 1. Ignore that and document it 2. Fix it so `""inserted""` only returns rows that were actually inserted (bit tricky) 3. Change the name of `""inserted""` to something else I'm picking 3 - I'm going to change it to be called `""rows""` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451,https://api.github.com/repos/simonw/datasette/issues/1873,1298885451,IC_kwDOBm6k_c5Na2tL,9599,simonw,2022-11-01T17:42:20Z,2022-11-01T17:42:20Z,OWNER,"Design decision: ```json { ""rows"": [{""id"": 1, ""title"": ""The title""}], ""ignore"": true } ``` Or `""replace"": true`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343716,https://api.github.com/repos/simonw/datasette/issues/1873,1296343716,IC_kwDOBm6k_c5NRKKk,9599,simonw,2022-10-30T20:24:55Z,2022-10-30T20:24:55Z,OWNER,"I think the key feature I need here is going to be the equivalent of `ignore=True` and `replace=True` for dealing with primary key collisions, see https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite_utils.db.Table.insert","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343317,https://api.github.com/repos/simonw/datasette/issues/1873,1296343317,IC_kwDOBm6k_c5NRKEV,9599,simonw,2022-10-30T20:22:40Z,2022-10-30T20:22:40Z,OWNER,"So maybe they're not actually worth worrying about separately, because they are guaranteed to have a primary key set.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343173,https://api.github.com/repos/simonw/datasette/issues/1873,1296343173,IC_kwDOBm6k_c5NRKCF,9599,simonw,2022-10-30T20:21:54Z,2022-10-30T20:22:20Z,OWNER,"One last case to consider: `WITHOUT ROWID` tables. https://www.sqlite.org/withoutrowid.html > By default, every row in SQLite has a special column, usually called the ""[rowid](https://www.sqlite.org/lang_createtable.html#rowid)"", that uniquely identifies that row within the table. However if the phrase ""WITHOUT ROWID"" is added to the end of a [CREATE TABLE](https://www.sqlite.org/lang_createtable.html) statement, then the special ""rowid"" column is omitted. There are sometimes space and performance advantages to omitting the rowid. > > ... > > Every WITHOUT ROWID table must have a [PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#primkeyconst). An error is raised if a CREATE TABLE statement with the WITHOUT ROWID clause lacks a PRIMARY KEY.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343014,https://api.github.com/repos/simonw/datasette/issues/1873,1296343014,IC_kwDOBm6k_c5NRJ_m,9599,simonw,2022-10-30T20:21:01Z,2022-10-30T20:21:01Z,OWNER,"Actually, for simplicity I'm going to say that you can always set the primary key, even for auto-incrementing primary key columns... but you cannot set it on pure `rowid` columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296342814,https://api.github.com/repos/simonw/datasette/issues/1873,1296342814,IC_kwDOBm6k_c5NRJ8e,9599,simonw,2022-10-30T20:20:05Z,2022-10-30T20:20:05Z,OWNER,"Some notes on what Datasette does already https://latest.datasette.io/fixtures/tags.json?_shape=array returns: ```json [ { ""tag"": ""canine"" }, { ""tag"": ""feline"" } ] ``` That table is defined [like this](https://latest.datasette.io/fixtures/tags): ```sql CREATE TABLE tags ( tag TEXT PRIMARY KEY ); ``` Here's a `rowid` table with no explicit primary key: https://latest.datasette.io/fixtures/binary_data https://latest.datasette.io/fixtures/binary_data.json?_shape=array ```json [ { ""rowid"": 1, ""data"": { ""$base64"": true, ""encoded"": ""FRwCx60F/g=="" } }, { ""rowid"": 2, ""data"": { ""$base64"": true, ""encoded"": ""FRwDx60F/g=="" } }, { ""rowid"": 3, ""data"": null } ] ``` ```sql CREATE TABLE binary_data ( data BLOB ); ``` https://latest.datasette.io/fixtures/simple_primary_key has a text primary key: https://latest.datasette.io/fixtures/simple_primary_key.json?_shape=array ```json [ { ""id"": ""1"", ""content"": ""hello"" }, { ""id"": ""2"", ""content"": ""world"" }, { ""id"": ""3"", ""content"": """" }, { ""id"": ""4"", ""content"": ""RENDER_CELL_DEMO"" }, { ""id"": ""5"", ""content"": ""RENDER_CELL_ASYNC"" } ] ``` ```sql CREATE TABLE simple_primary_key ( id varchar(30) primary key, content text ); ``` https://latest.datasette.io/fixtures/compound_primary_key is a compound primary key. https://latest.datasette.io/fixtures/compound_primary_key.json?_shape=array ```json [ { ""pk1"": ""a"", ""pk2"": ""b"", ""content"": ""c"" }, { ""pk1"": ""a/b"", ""pk2"": "".c-d"", ""content"": ""c"" } ] ``` ```sql CREATE TABLE compound_primary_key ( pk1 varchar(30), pk2 varchar(30), content text, PRIMARY KEY (pk1, pk2) ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296341469,https://api.github.com/repos/simonw/datasette/issues/1873,1296341469,IC_kwDOBm6k_c5NRJnd,9599,simonw,2022-10-30T20:13:50Z,2022-10-30T20:13:50Z,OWNER,"I checked and SQLite itself does allow you to set the `rowid` on that kind of table - it then increments from whatever you inserted: ``` % sqlite3 /tmp/t.db SQLite version 3.39.4 2022-09-07 20:51:41 Enter "".help"" for usage hints. sqlite> create table docs (title text); sqlite> insert into docs (title) values ('one'); sqlite> select rowid, title from docs; 1|one sqlite> insert into docs (rowid, title) values (3, 'three'); sqlite> select rowid, title from docs; 1|one 3|three sqlite> insert into docs (title) values ('another'); sqlite> select rowid, title from docs; 1|one 3|three 4|another ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296341055,https://api.github.com/repos/simonw/datasette/issues/1873,1296341055,IC_kwDOBm6k_c5NRJg_,9599,simonw,2022-10-30T20:11:47Z,2022-10-30T20:12:30Z,OWNER,"If a table has an auto-incrementing primary key, should you be allowed to insert records with an explicit key into it? I'm torn on this one. It's something you can do with direct database access, but it's something I very rarely want to do. I'm inclined to disallow it and say that if you want that you can get it using a writable canned query instead. Likewise, I'm not going to provide a way to set the `rowid` explicitly on a freshly inserted row.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296131681,https://api.github.com/repos/simonw/datasette/issues/1873,1296131681,IC_kwDOBm6k_c5NQWZh,9599,simonw,2022-10-30T06:27:12Z,2022-10-30T06:27:12Z,OWNER,Relevant TODO: https://github.com/simonw/datasette/blob/c35859ae3df163406f1a1895ccf9803e933b2d8e/datasette/views/table.py#L1131-L1135,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1872#issuecomment-1296131343,https://api.github.com/repos/simonw/datasette/issues/1872,1296131343,IC_kwDOBm6k_c5NQWUP,9599,simonw,2022-10-30T06:26:01Z,2022-10-30T06:26:01Z,OWNER,"Good spot fixing that! Sorry about this - it was a change in Datasette 0.63 which should have been better called out. My goal for Datasette 1.0 (which I aim to have out by the end of the year) is to introduce a formal process for avoiding problems like this, with very clear documentation when something like this might happen.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428560020,"SITE-BUSTING ERROR: ""render_template() called before await ds.invoke_startup()""", https://github.com/simonw/datasette/issues/1872#issuecomment-1296080804,https://api.github.com/repos/simonw/datasette/issues/1872,1296080804,IC_kwDOBm6k_c5NQJ-k,192568,mroswell,2022-10-30T03:06:32Z,2022-10-30T03:06:32Z,CONTRIBUTOR,"I updated datasette-publish-vercel to 0.14.2 in requirements.txt And the site is back up! Is there a way that we can get some sort of notice when something like this will have critical impact on website function?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428560020,"SITE-BUSTING ERROR: ""render_template() called before await ds.invoke_startup()""", https://github.com/simonw/datasette/issues/1872#issuecomment-1296076803,https://api.github.com/repos/simonw/datasette/issues/1872,1296076803,IC_kwDOBm6k_c5NQJAD,192568,mroswell,2022-10-30T02:50:34Z,2022-10-30T02:50:34Z,CONTRIBUTOR,"should this issue be under https://github.com/simonw/datasette-publish-vercel/issues ? Perhaps I just need to update: datasette-publish-vercel==0.11 in requirements.txt? I'll try that and see what happens... ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428560020,"SITE-BUSTING ERROR: ""render_template() called before await ds.invoke_startup()""", https://github.com/simonw/datasette/issues/1871#issuecomment-1313125870,https://api.github.com/repos/simonw/datasette/issues/1871,1313125870,IC_kwDOBm6k_c5ORLXu,9599,simonw,2022-11-14T05:42:50Z,2022-11-14T05:42:50Z,OWNER,Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=%7B%22confirm%22%3A+true%7D&method=POST,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1313097057,https://api.github.com/repos/simonw/datasette/issues/1871,1313097057,IC_kwDOBm6k_c5OREVh,9599,simonw,2022-11-14T04:59:28Z,2022-11-14T04:59:28Z,OWNER,In playing with the API explorer just now I realized it's way too easy to accidentally drop a table using it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1313072900,https://api.github.com/repos/simonw/datasette/issues/1871,1313072900,IC_kwDOBm6k_c5OQ-cE,9599,simonw,2022-11-14T04:15:50Z,2022-11-14T04:15:50Z,OWNER,"For the example links - I'm going to have these at the bottom of the page so you don't have to scroll past them. Ideally these would take the user's permissions into account. This could make the page expensive to load, but I'm going to risk it for the moment. Something like this then: > - data > - /data/-/create - create table > - /data/table1/-/insert - insert into table1 > - /data/table1/-/drop - drop table1 I won't bother with per-row demo links (for update and delete) because there could be thousands of them for each table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1313062699,https://api.github.com/repos/simonw/datasette/issues/1871,1313062699,IC_kwDOBm6k_c5OQ78r,9599,simonw,2022-11-14T04:03:29Z,2022-11-14T04:12:41Z,OWNER,"Two things left before I close this issue: - [x] I want to preserve the state of the forms in the URL - probably after a `#` - [ ] Instead of hard-coding the current examples, I want to provide a list of links which populate the forms","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1312822353,https://api.github.com/repos/simonw/datasette/issues/1871,1312822353,IC_kwDOBm6k_c5OQBRR,9599,simonw,2022-11-13T21:07:40Z,2022-11-13T21:07:40Z,OWNER,I'm going to need extra code to toggle POST closed when GET opens and vice-versa.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1312821031,https://api.github.com/repos/simonw/datasette/issues/1871,1312821031,IC_kwDOBm6k_c5OQA8n,9599,simonw,2022-11-13T21:02:06Z,2022-11-13T21:03:11Z,OWNER,"Actually no, I'm going to add a class of `details-menu` to the other details elements that SHOULD be closed. That way custom templates using `

` won't close in a surprising way.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1312816451,https://api.github.com/repos/simonw/datasette/issues/1871,1312816451,IC_kwDOBm6k_c5OP_1D,9599,simonw,2022-11-13T20:39:26Z,2022-11-13T20:39:34Z,OWNER,I'm going to add a special `no-auto-close` class to these and teach that code not to close them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1312816292,https://api.github.com/repos/simonw/datasette/issues/1871,1312816292,IC_kwDOBm6k_c5OP_yk,9599,simonw,2022-11-13T20:38:42Z,2022-11-13T20:38:42Z,OWNER,"The current API explorer uses details/summary elements for the GET and POST dialogs. I only want one of these to be open at a time, to reflect that you can make either a GET or a POST. I just noticed that clicking anywhere else on the page closes both elements, which isn't what I want to happen. Turns out that's because of this code I added as part of Datasette's menu implementation! https://github.com/simonw/datasette/blob/9f54f00a50a4d950cfd69a0ff3526ae82c858826/datasette/templates/_close_open_menus.html#L2-L15","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1309650806,https://api.github.com/repos/simonw/datasette/issues/1871,1309650806,IC_kwDOBm6k_c5OD692,3556,davidbgk,2022-11-10T01:38:58Z,2022-11-10T01:38:58Z,CONTRIBUTOR,"> Realized the API explorer doesn't need the API key piece at all - it can work with standard cookie-based auth. > > This also reflects how most plugins are likely to use this API, where they'll be adding JavaScript that uses `fetch()` to call the write API directly. I agree (that's what I did with the previous insert plugin), maybe a complete example using `fetch()` in the documentation would be valuable as a “Getting started with the API” or similar?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1302815105,https://api.github.com/repos/simonw/datasette/issues/1871,1302815105,IC_kwDOBm6k_c5Np2GB,9599,simonw,2022-11-04T00:17:23Z,2022-11-04T00:17:23Z,OWNER,"I'll probably enhance it a bit more though, I want to provide a UI that lists all the tables you can explore and lets you click to pre-fill the forms with them. Though at that point what should I do about the other endpoints? Probably list those too. Gets a bit complex, especially with the row-level update and delete endpoints.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1302814693,https://api.github.com/repos/simonw/datasette/issues/1871,1302814693,IC_kwDOBm6k_c5Np1_l,9599,simonw,2022-11-04T00:16:36Z,2022-11-04T00:16:36Z,OWNER,"I can close this issue once I fix it so it no longer hard-codes a potentially invalid example endpoint: https://github.com/simonw/datasette/blob/bcc781f4c50a8870e3389c4e60acb625c34b0317/datasette/templates/api_explorer.html#L24-L26 https://github.com/simonw/datasette/blob/bcc781f4c50a8870e3389c4e60acb625c34b0317/datasette/templates/api_explorer.html#L34-L35","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299607082,https://api.github.com/repos/simonw/datasette/issues/1871,1299607082,IC_kwDOBm6k_c5Ndm4q,9599,simonw,2022-11-02T05:45:31Z,2022-11-02T05:45:31Z,OWNER,"I'm going to add a link to the Datasette API docs for the current running version of Datasette, e.g. to https://docs.datasette.io/en/0.63/json_api.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299600257,https://api.github.com/repos/simonw/datasette/issues/1871,1299600257,IC_kwDOBm6k_c5NdlOB,9599,simonw,2022-11-02T05:36:40Z,2022-11-02T05:36:40Z,OWNER,"The API Explorer should definitely link to the `/-/create-token` page for users who have permission though. And it should probably go in the Datasette application menu?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299599461,https://api.github.com/repos/simonw/datasette/issues/1871,1299599461,IC_kwDOBm6k_c5NdlBl,9599,simonw,2022-11-02T05:35:36Z,2022-11-02T05:36:15Z,OWNER,"Here's a slightly wild idea: what if there was a button on `/-/api` that you could click to turn on ""API explorer mode"" for the rest of the Datasette interface - which sets a cookie, and that cookie means you then see ""API explorer"" links in all sorts of other relevant places in the Datasette UI (maybe tucked away in cog menus). Only reason I don't want to show these to everyone is that I don't think this is a very user-friendly feature: if you don't know what an API is I don't want to expose you to it unnecessarily.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299598570,https://api.github.com/repos/simonw/datasette/issues/1871,1299598570,IC_kwDOBm6k_c5Ndkzq,9599,simonw,2022-11-02T05:34:28Z,2022-11-02T05:34:28Z,OWNER,"This is pretty useful now. Two features I still want to add: - The ability to link to the API explorer such that the form is pre-filled with material from the URL. Need to guard against clickjacking first though, so no-one can link to it in an invisible iframe and trick the user into hitting POST. - Some kind of list of endpoints so people can click links to start using the API explorer. A list of every table the user can write to with each of their `/db/table/-/insert` endpoints for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299597066,https://api.github.com/repos/simonw/datasette/issues/1871,1299597066,IC_kwDOBm6k_c5NdkcK,9599,simonw,2022-11-02T05:32:22Z,2022-11-02T05:32:22Z,OWNER,"Demo of the latest API explorer: ![explorer](https://user-images.githubusercontent.com/9599/199406184-1292df42-25ea-4daf-8b54-ca26170ec1ea.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299388341,https://api.github.com/repos/simonw/datasette/issues/1871,1299388341,IC_kwDOBm6k_c5Ncxe1,9599,simonw,2022-11-02T00:24:28Z,2022-11-02T00:25:00Z,OWNER,"I want JSON syntax highlighting. https://github.com/luyilin/json-format-highlight is an MIT licensed tiny highlighter that looks decent for this. https://unpkg.com/json-format-highlight@1.0.1/dist/json-format-highlight.js","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299349741,https://api.github.com/repos/simonw/datasette/issues/1871,1299349741,IC_kwDOBm6k_c5NcoDt,9599,simonw,2022-11-01T23:22:55Z,2022-11-01T23:22:55Z,OWNER,"It's weird that the API explorer only lets you explore POST APIs. It should probably also let you explore GET APIs, or be renamed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296339386,https://api.github.com/repos/simonw/datasette/issues/1871,1296339386,IC_kwDOBm6k_c5NRJG6,9599,simonw,2022-10-30T20:03:04Z,2022-10-30T20:03:04Z,OWNER,"I do need to skip CSRF for these API calls. I'm going to start out by doing that using the `skip_csrf()` hook to skip CSRF checks on anything with a `content-type: application/json` request header. ```python @hookimpl def skip_csrf(scope): if scope[""type""] == ""http"": headers = scope.get(""headers"") if dict(headers).get(b'content-type') == b'application/json': return True ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296339205,https://api.github.com/repos/simonw/datasette/issues/1871,1296339205,IC_kwDOBm6k_c5NRJEF,9599,simonw,2022-10-30T20:02:05Z,2022-10-30T20:02:05Z,OWNER,"Realized the API explorer doesn't need the API key piece at all - it can work with standard cookie-based auth. This also reflects how most plugins are likely to use this API, where they'll be adding JavaScript that uses `fetch()` to call the write API directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296131872,https://api.github.com/repos/simonw/datasette/issues/1871,1296131872,IC_kwDOBm6k_c5NQWcg,9599,simonw,2022-10-30T06:27:56Z,2022-10-30T06:27:56Z,OWNER,Initial prototype API explorer is now live at https://latest-1-0-dev.datasette.io/-/api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296130073,https://api.github.com/repos/simonw/datasette/issues/1871,1296130073,IC_kwDOBm6k_c5NQWAZ,9599,simonw,2022-10-30T06:20:56Z,2022-10-30T06:20:56Z,OWNER,"That initial prototype looks like this: It currently shows the returned JSON from the API in an `alert()`. Next I should make that part of the page instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296126389,https://api.github.com/repos/simonw/datasette/issues/1871,1296126389,IC_kwDOBm6k_c5NQVG1,9599,simonw,2022-10-30T06:04:48Z,2022-10-30T06:04:48Z,OWNER,"This is even more important now I have pushed: - #1866","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296114136,https://api.github.com/repos/simonw/datasette/issues/1871,1296114136,IC_kwDOBm6k_c5NQSHY,9599,simonw,2022-10-30T05:15:40Z,2022-10-30T05:15:40Z,OWNER,"Host it at `/-/api` It's an input box with a path in and a textarea you can put JSON in, plus a submit button to post the request. It lists the API endpoints you can use - click on a link to populate the form field plus a example. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/pull/1870#issuecomment-1295667649,https://api.github.com/repos/simonw/datasette/issues/1870,1295667649,IC_kwDOBm6k_c5NOlHB,536941,fgregg,2022-10-29T00:52:43Z,2022-10-29T00:53:43Z,CONTRIBUTOR,"> Are you saying that I can build a container, but then when I run it and it does `datasette serve -i data.db ...` it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that `serve` command? Somehow, `datasette serve -i data.db` will lead to the `data.db` being modified, which will trigger a [copy-on-write](https://docs.docker.com/storage/storagedriver/#the-copy-on-write-cow-strategy) of `data.db` into the read-write layer of the container. I don't understand **how** that happens. it kind of feels like a bug in sqlite, but i can't quite follow the sqlite code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/pull/1870#issuecomment-1295660092,https://api.github.com/repos/simonw/datasette/issues/1870,1295660092,IC_kwDOBm6k_c5NOjQ8,9599,simonw,2022-10-29T00:25:26Z,2022-10-29T00:25:26Z,OWNER,"Saw your comment here too: https://github.com/simonw/datasette/issues/1480#issuecomment-1271101072 > switching from `immutable=1` to `mode=ro` completely addressed this. see https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 for details. So maybe we need a special case for containers that are intended to be run using Docker - the ones produced by `datasette package` and `datasette publish cloudrun`? Those are cases where the `-i` option should actually be opened in read-only mode, not immutable mode. Maybe a `datasette serve --irw data.db` option for opening a file in immutable-but-actually-read-only mode? Bit ugly though. I should run some benchmarks to figure out if `immutable` really does offer significant performance benefits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/pull/1870#issuecomment-1295657771,https://api.github.com/repos/simonw/datasette/issues/1870,1295657771,IC_kwDOBm6k_c5NOisr,9599,simonw,2022-10-29T00:19:03Z,2022-10-29T00:19:03Z,OWNER,"Just saw your comment here: https://github.com/simonw/datasette/issues/1836#issuecomment-1272357976 > when you are running from docker, you **always** will want to run as `mode=ro` because the same thing that is causing duplication in the inspect layer will cause duplication in the final container read/write layer when `datasette serve` runs. I don't understand this. My mental model of how Docker works is that the image itself is created using `docker build`... but then when the image runs later on (`docker run`) the image itself isn't touched at all. Are you saying that I can build a container, but then when I run it and it does `datasette serve -i data.db ...` it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that `serve` command?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/pull/1870#issuecomment-1294285471,https://api.github.com/repos/simonw/datasette/issues/1870,1294285471,IC_kwDOBm6k_c5NJTqf,536941,fgregg,2022-10-28T01:06:03Z,2022-10-28T01:06:03Z,CONTRIBUTOR,"as far as i can tell, [this is where the ""immutable"" argument is used](https://github.com/sqlite/sqlite/blob/c97bb14fab566f6fa8d967c8fd1e90f3702d5b73/src/pager.c#L4926-L4931) in sqlite: ```c pPager->noLock = sqlite3_uri_boolean(pPager->zFilename, ""nolock"", 0); if( (iDc & SQLITE_IOCAP_IMMUTABLE)!=0 || sqlite3_uri_boolean(pPager->zFilename, ""immutable"", 0) ){ vfsFlags |= SQLITE_OPEN_READONLY; goto act_like_temp_file; } ``` so it does set the read only flag, but then has a goto.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/pull/1870#issuecomment-1294238862,https://api.github.com/repos/simonw/datasette/issues/1870,1294238862,IC_kwDOBm6k_c5NJISO,22429695,codecov[bot],2022-10-27T23:44:25Z,2022-10-27T23:44:25Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.55**% // Head: **92.55**% // No change to project coverage :thumbsup: > Coverage data is based on head [(`4faa4fd`)](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`bf00b0b`)](https://codecov.io/gh/simonw/datasette/commit/bf00b0b59b6692bdec597ac9db4e0b497c5a47b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch has no changes to coverable lines.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1870 +/- ## ======================================= Coverage 92.55% 92.55% ======================================= Files 35 35 Lines 4432 4432 ======================================= Hits 4102 4102 Misses 330 330 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1870/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.30% <ø> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/pull/1870#issuecomment-1294237783,https://api.github.com/repos/simonw/datasette/issues/1870,1294237783,IC_kwDOBm6k_c5NJIBX,536941,fgregg,2022-10-27T23:42:18Z,2022-10-27T23:42:18Z,CONTRIBUTOR,Relevant sqlite forum thread: https://www.sqlite.org/forum/forumpost/02f7bda329f41e30451472421cf9ce7f715b768ce3db02797db1768e47950d48,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/issues/1869#issuecomment-1294181485,https://api.github.com/repos/simonw/datasette/issues/1869,1294181485,IC_kwDOBm6k_c5NI6Rt,9599,simonw,2022-10-27T22:24:37Z,2022-10-27T22:24:37Z,OWNER,"https://docs.datasette.io/en/stable/changelog.html#v0-63 Annotated release notes: https://simonwillison.net/2022/Oct/27/datasette-0-63/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426253476,Release 0.63, https://github.com/simonw/datasette/issues/1869#issuecomment-1294105558,https://api.github.com/repos/simonw/datasette/issues/1869,1294105558,IC_kwDOBm6k_c5NInvW,9599,simonw,2022-10-27T21:44:13Z,2022-10-27T21:44:13Z,OWNER,I'm going to do annotated release notes for this one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426253476,Release 0.63,

{{ row[""title""] }}

Create an API token

All databases and tables

All tables in database: {{ database }}

Specific tables

{{ dbt.database }}: {{ table }}

Test permission check