{"html_url": "https://github.com/simonw/datasette/issues/1863#issuecomment-1302790013", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1863", "id": 1302790013, "node_id": "IC_kwDOBm6k_c5Npv99", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T23:32:30Z", "updated_at": "2022-11-03T23:32:30Z", "author_association": "OWNER", "body": "I'm not going to allow updates to primary keys. If you need to do that, you can instead delete the record and then insert a new one with the new primary keys you wanted - or maybe use a custom SQL query.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425029242, "label": "Update a single record in an existing table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1863#issuecomment-1302785086", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1863", "id": 1302785086, "node_id": "IC_kwDOBm6k_c5Npuw-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T23:24:33Z", "updated_at": "2022-11-03T23:24:56Z", "author_association": "OWNER", "body": "Thinking more about validation: I'm considering if this should validate that columns which are defined as SQLite foreign keys are being updated to values that exist in those other tables.\r\n\r\nI like the sound of this. It seems like a sensible default behaviour for Datasette. And it fits with the fact that Datasette treats foreign keys specially elsewhere in the interface.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425029242, "label": "Update a single record in an existing table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1863#issuecomment-1302760549", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1863", "id": 1302760549, "node_id": "IC_kwDOBm6k_c5Npoxl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T22:43:04Z", "updated_at": "2022-11-03T23:21:31Z", "author_association": "OWNER", "body": "The `id=(int, ...)` thing is weird, but is apparently Pydantic syntax for a required field?\r\n\r\nhttps://cs.github.com/starlite-api/starlite/blob/28ddc847c4cb072f0d5d21a9ecd5259711f12ec9/docs/usage/11-data-transfer-objects.md#L161 confirms:\r\n\r\n> 1. For required fields use a tuple of type + ellipsis, for example `(str, ...)`.\r\n> 2. For optional fields use a tuple of type + `None`, for example `(str, None)`\r\n> 3. To set a default value use a tuple of type + default value, for example `(str, \"Hello World\")`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425029242, "label": "Update a single record in an existing table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1863#issuecomment-1302760382", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1863", "id": 1302760382, "node_id": "IC_kwDOBm6k_c5Npou-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T22:42:47Z", "updated_at": "2022-11-03T22:42:47Z", "author_association": "OWNER", "body": "```python\r\nprint(create_model('document', id=(int, ...), title=(str, None)).schema_json(indent=2))\r\n```\r\n```json\r\n{\r\n  \"title\": \"document\",\r\n  \"type\": \"object\",\r\n  \"properties\": {\r\n    \"id\": {\r\n      \"title\": \"Id\",\r\n      \"type\": \"integer\"\r\n    },\r\n    \"title\": {\r\n      \"title\": \"Title\",\r\n      \"type\": \"string\"\r\n    }\r\n  },\r\n  \"required\": [\r\n    \"id\"\r\n  ]\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425029242, "label": "Update a single record in an existing table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1863#issuecomment-1302759174", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1863", "id": 1302759174, "node_id": "IC_kwDOBm6k_c5NpocG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T22:40:47Z", "updated_at": "2022-11-03T22:40:47Z", "author_association": "OWNER", "body": "I'm considering Pydantic for this, see:\r\n- https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350\r\n\r\nIn particular the `create_model()` method: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation\r\n\r\nThis would give me good validation. It would also, weirdly, give me the ability to output JSON schema. Maybe I could have this as the JSON schema for a row?\r\n\r\n`/db/table/-/json-schema`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425029242, "label": "Update a single record in an existing table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1302721916", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1302721916, "node_id": "IC_kwDOBm6k_c5NpfV8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T21:58:50Z", "updated_at": "2022-11-03T21:59:17Z", "author_association": "OWNER", "body": "Mocked up a quick HTML+JavaScript form for creating that JSON structure using some iteration against Copilot prompts:\r\n```html\r\n<pre>\r\n/* JSON format:\r\n{\r\n  \"table\": {\r\n      \"name\": \"my new table\",\r\n      \"columns\": [\r\n          {\r\n              \"name\": \"id\",\r\n              \"type\": \"integer\"\r\n          },\r\n          {\r\n              \"name\": \"title\",\r\n              \"type\": \"text\"\r\n          }\r\n      ]\r\n     \"pk\": \"id\"\r\n  }\r\n}\r\n\r\nHTML form with Javascript for creating this JSON:\r\n*/</pre>\r\n<form id=\"create-table-form\">\r\n  <label for=\"table-name\">Table name</label>\r\n  <input type=\"text\" id=\"table-name\" name=\"table-name\" required><br>\r\n  <label for=\"table-pk\">Primary key</label>\r\n  <input type=\"text\" id=\"table-pk\" name=\"table-pk\" required><br>\r\n  <label for=\"column-name\">Column name</label>\r\n  <input type=\"text\" id=\"column-name\" name=\"column-name\" required>\r\n  <label for=\"column-type\">Column type</label>\r\n  <input type=\"text\" id=\"column-type\" name=\"column-type\" required>\r\n  <button type=\"button\" id=\"add-column\">Add column</button>\r\n  <p>Current columns:</p>\r\n  <ul id=\"columns\"></ul>\r\n  <button type=\"button\" id=\"create-table\">Create table</button>\r\n</form>\r\n<script>\r\nvar form = document.getElementById('create-table-form');\r\nvar tableName = document.getElementById('table-name');\r\nvar tablePk = document.getElementById('table-pk');\r\nvar columnName = document.getElementById('column-name');\r\nvar columnType = document.getElementById('column-type');\r\nvar addColumn = document.getElementById('add-column');\r\nvar createTable = document.getElementById('create-table');\r\nvar columns = [];\r\naddColumn.addEventListener('click', () => {\r\n  columns.push({\r\n    name: columnName.value,\r\n    type: columnType.value\r\n  });\r\n  var li = document.createElement('li');\r\n  li.textContent = columnName.value + ' (' + columnType.value + ')';\r\n  // Add a delete button to each column\r\n  var deleteButton = document.createElement('button');\r\n  deleteButton.textContent = 'Delete';\r\n  deleteButton.addEventListener('click', () => {\r\n    columns.splice(columns.indexOf(li), 1);\r\n    li.remove();\r\n  });\r\n  li.appendChild(deleteButton);\r\n  document.getElementById('columns').appendChild(li);\r\n});\r\ncreateTable.addEventListener('click', () => {\r\n  var table = {\r\n    name: tableName.value,\r\n    pk: tablePk.value,\r\n    columns: columns\r\n  };\r\n  console.log(JSON.stringify(table, null, 2));\r\n});\r\n</script>\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1302716350, "node_id": "IC_kwDOBm6k_c5Npd--", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T21:51:14Z", "updated_at": "2022-11-03T22:35:54Z", "author_association": "OWNER", "body": "Validating this JSON object is getting a tiny bit complex. I'm tempted to adopt https://pydantic-docs.helpmanual.io/ at this point.\r\n\r\nThe `create_model` example on https://stackoverflow.com/questions/66168517/generate-dynamic-model-using-pydantic/66168682#66168682 is particularly relevant, especially when I work on this issue:\r\n\r\n- #1863\r\n\r\n```python\r\nfrom pydantic import create_model\r\n\r\nd = {\"strategy\": {\"name\": \"test_strat2\", \"periods\": 10}}\r\n\r\nStrategy = create_model(\"Strategy\", **d[\"strategy\"])\r\n\r\nprint(Strategy.schema_json(indent=2))\r\n```\r\n`create_model()`: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1302715662", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1302715662, "node_id": "IC_kwDOBm6k_c5Npd0O", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T21:50:27Z", "updated_at": "2022-11-03T21:50:27Z", "author_association": "OWNER", "body": "API design for this:\r\n```\r\nPOST /db/-/create\r\nAuthorization: Bearer xxx\r\nContent-Type: application/json\r\n{\r\n    \"table\": {\r\n        \"name\": \"my new table\",\r\n        \"columns\": [\r\n            {\r\n                \"name\": \"id\",\r\n                \"type\": \"integer\"\r\n            },\r\n            {\r\n                \"name\": \"title\",\r\n                \"type\": \"text\"\r\n            }\r\n        ]\r\n       \"pk\": \"id\"\r\n    }\r\n}\r\n```\r\nSupported column types are:\r\n\r\n- `integer`\r\n- `text`\r\n- `float` (even though SQLite calls it a \"real\")\r\n- `blob`\r\n\r\nThis matches my design for `sqlite-utils`: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1843#issuecomment-1302679026", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1843", "id": 1302679026, "node_id": "IC_kwDOBm6k_c5NpU3y", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T21:22:42Z", "updated_at": "2022-11-03T21:22:42Z", "author_association": "OWNER", "body": "Docs for the new `db.close()` method: https://docs.datasette.io/en/latest/internals.html#db-close", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1408757705, "label": "Intermittent \"Too many open files\" error running tests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1843#issuecomment-1302678384", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1843", "id": 1302678384, "node_id": "IC_kwDOBm6k_c5NpUtw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T21:21:59Z", "updated_at": "2022-11-03T21:21:59Z", "author_association": "OWNER", "body": "I added extra debug info to `/-/threads` to see this for myself:\r\n\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 02bd38f1..16579e28 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -969,6 +969,13 @@ class Datasette:\r\n             \"threads\": [\r\n                 {\"name\": t.name, \"ident\": t.ident, \"daemon\": t.daemon} for t in threads\r\n             ],\r\n+            \"file_connections\": {\r\n+                db.name: [\r\n+                    [dict(r) for r in conn.execute(\"pragma database_list\").fetchall()]\r\n+                    for conn in db._all_file_connections\r\n+                ]\r\n+                for db in self.databases.values()\r\n+            },\r\n         }\r\n         # Only available in Python 3.7+\r\n         if hasattr(asyncio, \"all_tasks\"):\r\n```\r\nOutput after hitting refresh on a few `/fixtures` tables to ensure more threads started:\r\n\r\n```\r\n    \"file_connections\": {\r\n        \"_internal\": [],\r\n        \"fixtures\": [\r\n            [\r\n                {\r\n                    \"seq\": 0,\r\n                    \"name\": \"main\",\r\n                    \"file\": \"/Users/simon/Dropbox/Development/datasette/fixtures.db\"\r\n                }\r\n            ],\r\n            [\r\n                {\r\n                    \"seq\": 0,\r\n                    \"name\": \"main\",\r\n                    \"file\": \"/Users/simon/Dropbox/Development/datasette/fixtures.db\"\r\n                }\r\n            ],\r\n            [\r\n                {\r\n                    \"seq\": 0,\r\n                    \"name\": \"main\",\r\n                    \"file\": \"/Users/simon/Dropbox/Development/datasette/fixtures.db\"\r\n                }\r\n            ]\r\n        ]\r\n    },\r\n```\r\nI decided not to ship this feature though as it leaks the names of internal database files.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1408757705, "label": "Intermittent \"Too many open files\" error running tests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1843#issuecomment-1302634332", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1843", "id": 1302634332, "node_id": "IC_kwDOBm6k_c5NpJ9c", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T20:34:56Z", "updated_at": "2022-11-03T20:34:56Z", "author_association": "OWNER", "body": "Confirmed that calling `conn.close()` on each SQLite file-based connection is the way to fix this problem.\r\n\r\nI'm adding a `db.close()` method (sync, not async - I tried async first but it was really hard to cause every thread in the pool to close its threadlocal database connection) which loops through all known open file-based connections and closes them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1408757705, "label": "Intermittent \"Too many open files\" error running tests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1843#issuecomment-1302574330", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1843", "id": 1302574330, "node_id": "IC_kwDOBm6k_c5No7T6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T19:30:22Z", "updated_at": "2022-11-03T19:30:22Z", "author_association": "OWNER", "body": "This is affecting me a lot at the moment, on my laptop (runs fine in CI).\r\n\r\nHere's a change to `conftest.py` which highlights the problem - it cause a failure the moment there are more than 5 open files according to `psutil`:\r\n\r\n```diff\r\ndiff --git a/tests/conftest.py b/tests/conftest.py\r\nindex f4638a14..21d433c1 100644\r\n--- a/tests/conftest.py\r\n+++ b/tests/conftest.py\r\n@@ -1,6 +1,7 @@\r\n import httpx\r\n import os\r\n import pathlib\r\n+import psutil\r\n import pytest\r\n import re\r\n import subprocess\r\n@@ -192,3 +193,8 @@ def ds_unix_domain_socket_server(tmp_path_factory):\r\n     yield ds_proc, uds\r\n     # Shut it down at the end of the pytest session\r\n     ds_proc.terminate()\r\n+\r\n+\r\n+def pytest_runtest_teardown(item: pytest.Item) -> None:\r\n+    open_files = psutil.Process().open_files()\r\n+    assert len(open_files) < 5\r\n```\r\nThe first error I get from this with `pytest --pdb -x` is here:\r\n\r\n```\r\ntests/test_api.py ............E\r\n>>>>> traceback >>>>>\r\n\r\nitem = <Function test_sql_time_limit>\r\n\r\n    def pytest_runtest_teardown(item: pytest.Item) -> None:\r\n        open_files = psutil.Process().open_files()\r\n>       assert len(open_files) < 5\r\nE       AssertionError: assert 5 < 5\r\nE        +  where 5 = len([popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfglrt4p2/fixtures.db', fd=14), popenfile(... fd=19), popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmphdi5b250/fixtures.dot.db', fd=20)])\r\n\r\n/Users/simon/Dropbox/Development/datasette/tests/conftest.py:200: AssertionError\r\n>>>>> entering PDB >>>>>\r\n\r\n>>>>> PDB post_mortem (IO-capturing turned off) >>>>>\r\n> /Users/simon/Dropbox/Development/datasette/tests/conftest.py(200)pytest_runtest_teardown()\r\n-> assert len(open_files) < 5\r\n```\r\nThat's this test:\r\n\r\nhttps://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/test_api.py#L656-L673\r\n\r\nWhich uses this fixture:\r\n\r\nhttps://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L228-L231\r\n\r\nWhich calls this function:\r\n\r\nhttps://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L105-L122\r\n\r\nSo now I'm suspicious that, even though the fixture is meant to be session scoped, the way I'm using `with tempfile.TemporaryDirectory() as tmpdir:` is causing a whole load of files to be created and held open which are not later closed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1408757705, "label": "Intermittent \"Too many open files\" error running tests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1855#issuecomment-1301646670", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1855", "id": 1301646670, "node_id": "IC_kwDOBm6k_c5NlY1O", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T05:11:26Z", "updated_at": "2022-11-03T05:11:26Z", "author_association": "OWNER", "body": "That still needs comprehensive tests before I land it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1423336089, "label": "`datasette create-token` ability to create tokens with a reduced set of permissions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1855#issuecomment-1301646493", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1855", "id": 1301646493, "node_id": "IC_kwDOBm6k_c5NlYyd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T05:11:06Z", "updated_at": "2022-11-03T05:11:06Z", "author_association": "OWNER", "body": "Built a prototype of the above:\r\n\r\n```diff\r\ndiff --git a/datasette/default_permissions.py b/datasette/default_permissions.py\r\nindex 32b0c758..f68aa38f 100644\r\n--- a/datasette/default_permissions.py\r\n+++ b/datasette/default_permissions.py\r\n@@ -6,8 +6,8 @@ import json\r\n import time\r\n \r\n \r\n-@hookimpl(tryfirst=True)\r\n-def permission_allowed(datasette, actor, action, resource):\r\n+@hookimpl(tryfirst=True, specname=\"permission_allowed\")\r\n+def permission_allowed_default(datasette, actor, action, resource):\r\n     async def inner():\r\n         if action in (\r\n             \"permissions-debug\",\r\n@@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource):\r\n     return inner\r\n \r\n \r\n+@hookimpl(specname=\"permission_allowed\")\r\n+def permission_allowed_actor_restrictions(actor, action, resource):\r\n+    if actor is None:\r\n+        return None\r\n+    _r = actor.get(\"_r\")\r\n+    if not _r:\r\n+        # No restrictions, so we have no opinion\r\n+        return None\r\n+    action_initials = \"\".join([word[0] for word in action.split(\"-\")])\r\n+    # If _r is defined then we use those to further restrict the actor\r\n+    # Crucially, we only use this to say NO (return False) - we never\r\n+    # use it to return YES (True) because that might over-ride other\r\n+    # restrictions placed on this actor\r\n+    all_allowed = _r.get(\"a\")\r\n+    if all_allowed is not None:\r\n+        assert isinstance(all_allowed, list)\r\n+        if action_initials in all_allowed:\r\n+            return None\r\n+    # How about for the current database?\r\n+    if action in (\"view-database\", \"view-database-download\", \"execute-sql\"):\r\n+        database_allowed = _r.get(\"d\", {}).get(resource)\r\n+        if database_allowed is not None:\r\n+            assert isinstance(database_allowed, list)\r\n+            if action_initials in database_allowed:\r\n+                return None\r\n+    # Or the current table? That's any time the resource is (database, table)\r\n+    if not isinstance(resource, str) and len(resource) == 2:\r\n+        database, table = resource\r\n+        table_allowed = _r.get(\"t\", {}).get(database, {}).get(table)\r\n+        # TODO: What should this do for canned queries?\r\n+        if table_allowed is not None:\r\n+            assert isinstance(table_allowed, list)\r\n+            if action_initials in table_allowed:\r\n+                return None\r\n+    # This action is not specifically allowed, so reject it\r\n+    return False\r\n+\r\n+\r\n @hookimpl\r\n def actor_from_request(datasette, request):\r\n     prefix = \"dstok_\"\r\n\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1423336089, "label": "`datasette create-token` ability to create tokens with a reduced set of permissions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301645921", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301645921, "node_id": "IC_kwDOBm6k_c5NlYph", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T05:10:05Z", "updated_at": "2022-12-09T01:38:21Z", "author_association": "OWNER", "body": "I'd love to come up with a good short name for the second part of the resource two-tuple, the thing which is usually the name of a table but could also be the name of a SQL view or the name of a canned query.\r\n\r\nIdea 8th December: why not call it resource? A resource could be a thing that lives inside a database.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301639741", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301639741, "node_id": "IC_kwDOBm6k_c5NlXI9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T04:58:21Z", "updated_at": "2022-11-03T04:58:21Z", "author_association": "OWNER", "body": "The whole `database_name` or `(database_name, table_name)` tuple for resource is a bit of a code smell. Maybe this is a chance to tidy that up too?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301639370", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301639370, "node_id": "IC_kwDOBm6k_c5NlXDK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T04:57:21Z", "updated_at": "2022-11-03T04:57:21Z", "author_association": "OWNER", "body": "The plugin hook would be called `register_permissions()`, for consistency with `register_routes()` and `register_commands()`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301638918", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301638918, "node_id": "IC_kwDOBm6k_c5NlW8G", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T04:56:06Z", "updated_at": "2022-11-03T04:56:06Z", "author_association": "OWNER", "body": "I've also introduced a new concept of a permission abbreviation, which like the permission name needs to be globally unique.\r\n\r\nThat's a problem for plugins - they might just be able to guarantee that their permission long-form name is unique among other plugins (through sensible naming conventions) but the thing where they declare a initial-letters-only abbreviation is far more risky.\r\n\r\nI think abbreviations are optional - they are provided for core permissions but plugins are advised not to use them.\r\n\r\nAlso Datasette could check that the installed plugins do not provide conflicting permissions on startup and refuse to start if they do.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301638156", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301638156, "node_id": "IC_kwDOBm6k_c5NlWwM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T04:54:00Z", "updated_at": "2022-11-03T04:54:00Z", "author_association": "OWNER", "body": "If I have the permissions defined like this:\r\n```python\r\nPERMISSIONS = (\r\n    Permission(\"view-instance\", \"vi\", False, False, True),\r\n    Permission(\"view-database\", \"vd\", True, False, True),\r\n    Permission(\"view-database-download\", \"vdd\", True, False, True),\r\n    Permission(\"view-table\", \"vt\", True, True, True),\r\n    Permission(\"view-query\", \"vq\", True, True, True),\r\n    Permission(\"insert-row\", \"ir\", True, True, False),\r\n    Permission(\"delete-row\", \"dr\", True, True, False),\r\n    Permission(\"drop-table\", \"dt\", True, True, False),\r\n    Permission(\"execute-sql\", \"es\", True, False, True),\r\n    Permission(\"permissions-debug\", \"pd\", False, False, False),\r\n    Permission(\"debug-menu\", \"dm\", False, False, False),\r\n)\r\n```\r\nInstead of just calling them by their undeclared names in places like this:\r\n```python\r\nawait self.ds.permission_allowed(\r\n    request.actor, \"execute-sql\", database, default=True\r\n)\r\n```\r\nOn the one hand I can ditch that confusing `default=True` option - whether a permission is on by default becomes a characteristic of that `Permission()` itself, which feels much neater.\r\n\r\nOn the other hand though, plugins that introduce their own permissions - like https://datasette.io/plugins/datasette-edit-schema - will need a way to register those permissions with Datasette core. Probably another plugin hook.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301635906", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301635906, "node_id": "IC_kwDOBm6k_c5NlWNC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T04:48:09Z", "updated_at": "2022-11-03T04:48:09Z", "author_association": "OWNER", "body": "I built this prototype on the http://127.0.0.1:8001/-/allow-debug page, which is open to anyone to visit.\r\n\r\nBut... I just realized that using this tool can leak information - you can use it to guess the names of invisible databases and tables and run theoretical permission checks against them.\r\n\r\nUsing the tool also pollutes the list of permission checks that show up on the root-anlo `/-/permissions` page.\r\n\r\nSo.... I'm going to restrict the usage of this tool to users with access to `/-/permissions` and put it on that page instead.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301635340", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301635340, "node_id": "IC_kwDOBm6k_c5NlWEM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T04:46:41Z", "updated_at": "2022-11-03T04:46:41Z", "author_association": "OWNER", "body": "Built this prototype:\r\n\r\n![prototype](https://user-images.githubusercontent.com/9599/199649219-f146e43b-bfb5-45e6-9777-956f21a79887.gif)\r\n\r\nIn building it I realized I needed to know which permissions took a table, a database, both or neither. So I had to bake that into the code.\r\n\r\nHere's the prototype so far (which includes a prototype of the logic for the `_r` field on actor, see #1855):\r\n\r\n```diff\r\ndiff --git a/datasette/default_permissions.py b/datasette/default_permissions.py\r\nindex 32b0c758..f68aa38f 100644\r\n--- a/datasette/default_permissions.py\r\n+++ b/datasette/default_permissions.py\r\n@@ -6,8 +6,8 @@ import json\r\n import time\r\n \r\n \r\n-@hookimpl(tryfirst=True)\r\n-def permission_allowed(datasette, actor, action, resource):\r\n+@hookimpl(tryfirst=True, specname=\"permission_allowed\")\r\n+def permission_allowed_default(datasette, actor, action, resource):\r\n     async def inner():\r\n         if action in (\r\n             \"permissions-debug\",\r\n@@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource):\r\n     return inner\r\n \r\n \r\n+@hookimpl(specname=\"permission_allowed\")\r\n+def permission_allowed_actor_restrictions(actor, action, resource):\r\n+    if actor is None:\r\n+        return None\r\n+    _r = actor.get(\"_r\")\r\n+    if not _r:\r\n+        # No restrictions, so we have no opinion\r\n+        return None\r\n+    action_initials = \"\".join([word[0] for word in action.split(\"-\")])\r\n+    # If _r is defined then we use those to further restrict the actor\r\n+    # Crucially, we only use this to say NO (return False) - we never\r\n+    # use it to return YES (True) because that might over-ride other\r\n+    # restrictions placed on this actor\r\n+    all_allowed = _r.get(\"a\")\r\n+    if all_allowed is not None:\r\n+        assert isinstance(all_allowed, list)\r\n+        if action_initials in all_allowed:\r\n+            return None\r\n+    # How about for the current database?\r\n+    if action in (\"view-database\", \"view-database-download\", \"execute-sql\"):\r\n+        database_allowed = _r.get(\"d\", {}).get(resource)\r\n+        if database_allowed is not None:\r\n+            assert isinstance(database_allowed, list)\r\n+            if action_initials in database_allowed:\r\n+                return None\r\n+    # Or the current table? That's any time the resource is (database, table)\r\n+    if not isinstance(resource, str) and len(resource) == 2:\r\n+        database, table = resource\r\n+        table_allowed = _r.get(\"t\", {}).get(database, {}).get(table)\r\n+        # TODO: What should this do for canned queries?\r\n+        if table_allowed is not None:\r\n+            assert isinstance(table_allowed, list)\r\n+            if action_initials in table_allowed:\r\n+                return None\r\n+    # This action is not specifically allowed, so reject it\r\n+    return False\r\n+\r\n+\r\n @hookimpl\r\n def actor_from_request(datasette, request):\r\n     prefix = \"dstok_\"\r\ndiff --git a/datasette/templates/allow_debug.html b/datasette/templates/allow_debug.html\r\nindex 0f1b30f0..ae43f0f5 100644\r\n--- a/datasette/templates/allow_debug.html\r\n+++ b/datasette/templates/allow_debug.html\r\n@@ -35,7 +35,7 @@ p.message-warning {\r\n \r\n <p>Use this tool to try out different actor and allow combinations. See <a href=\"https://docs.datasette.io/en/stable/authentication.html#defining-permissions-with-allow-blocks\">Defining permissions with \"allow\" blocks</a> for documentation.</p>\r\n \r\n-<form action=\"{{ urls.path('-/allow-debug') }}\" method=\"get\">\r\n+<form action=\"{{ urls.path('-/allow-debug') }}\" method=\"get\" style=\"margin-bottom: 1em\">\r\n     <div class=\"two-col\">\r\n         <p><label>Allow block</label></p>\r\n         <textarea name=\"allow\">{{ allow_input }}</textarea>\r\n@@ -55,4 +55,82 @@ p.message-warning {\r\n \r\n {% if result == \"False\" %}<p class=\"message-error\">Result: deny</p>{% endif %}\r\n \r\n+<h2>Test permission check</h2>\r\n+\r\n+<p>This tool lets you simulate an actor and a permission check for that actor.</p>\r\n+\r\n+<form action=\"{{ urls.path('-/allow-debug') }}\" id=\"debug-post\" method=\"post\" style=\"margin-bottom: 1em\">\r\n+    <input type=\"hidden\" name=\"csrftoken\" value=\"{{ csrftoken() }}\">\r\n+    <div class=\"two-col\">\r\n+        <p><label>Actor</label></p>\r\n+        <textarea name=\"actor\">{% if actor_input %}{{ actor_input }}{% else %}{\"id\": \"root\"}{% endif %}</textarea>\r\n+    </div>\r\n+    <div class=\"two-col\" style=\"vertical-align: top\">\r\n+        <p><label>Permission check</label></p>\r\n+        <p><label for=\"permission\" style=\"display:block\">Permission</label>\r\n+        <select name=\"permission\" id=\"permission\">\r\n+            {% for permission in [\r\n+                \"view-instance\",\r\n+                \"view-database\",\r\n+                \"view-database-download\",\r\n+                \"view-table\",\r\n+                \"view-query\",\r\n+                \"insert-row\",\r\n+                \"delete-row\",\r\n+                \"drop-table\",\r\n+                \"execute-sql\",\r\n+                \"permissions-debug\",\r\n+                \"debug-menu\"] %}\r\n+            <option value=\"{{ permission }}\">{{ permission }}</option>\r\n+            {% endfor %}\r\n+        </select>\r\n+        <p><label for=\"resource_1\">Database name</label><input type=\"text\" id=\"resource_1\" name=\"resource_1\"></p>\r\n+        <p><label for=\"resource_2\">Table or query name</label><input type=\"text\" id=\"resource_2\" name=\"resource_2\"></p>\r\n+    </div>\r\n+    <div style=\"margin-top: 1em;\">\r\n+        <input type=\"submit\" value=\"Simulate permission check\">\r\n+    </div>\r\n+</form>\r\n+\r\n+<script>\r\n+var rawPerms = {{ permissions|tojson }};\r\n+var permissions = Object.fromEntries(rawPerms.map(([label, abbr, needs_resource_1, needs_resource_2, def]) => [label, {needs_resource_1, needs_resource_2, def}]))\r\n+var permissionSelect = document.getElementById('permission');\r\n+var resource1 = document.getElementById('resource_1');\r\n+var resource2 = document.getElementById('resource_2');\r\n+function updateResourceVisibility() {\r\n+    var permission = permissionSelect.value;\r\n+    var {needs_resource_1, needs_resource_2} = permissions[permission];\r\n+    if (needs_resource_1) {\r\n+        resource1.closest('p').style.display = 'block';\r\n+    } else {\r\n+        resource1.closest('p').style.display = 'none';\r\n+    }\r\n+    if (needs_resource_2) {\r\n+        resource2.closest('p').style.display = 'block';\r\n+    } else {\r\n+        resource2.closest('p').style.display = 'none';\r\n+    }\r\n+}\r\n+permissionSelect.addEventListener('change', updateResourceVisibility);\r\n+updateResourceVisibility();\r\n+\r\n+// When #debug-post form is submitted, use fetch() to POST data\r\n+var debugPost = document.getElementById('debug-post');\r\n+debugPost.addEventListener('submit', function(ev) {\r\n+    ev.preventDefault();\r\n+    var formData = new FormData(debugPost);\r\n+    console.log(formData);\r\n+    fetch(debugPost.action, {\r\n+        method: 'POST',\r\n+        body: new URLSearchParams(formData),\r\n+    }).then(function(response) {\r\n+        return response.json();\r\n+    }).then(function(data) {\r\n+        alert(JSON.stringify(data, null, 4));\r\n+    });\r\n+});\r\n+</script>\r\n+    \r\n+\r\n {% endblock %}\r\ndiff --git a/datasette/views/special.py b/datasette/views/special.py\r\nindex 9922a621..d46fc280 100644\r\n--- a/datasette/views/special.py\r\n+++ b/datasette/views/special.py\r\n@@ -1,6 +1,8 @@\r\n import json\r\n+from datasette.permissions import PERMISSIONS\r\n from datasette.utils.asgi import Response, Forbidden\r\n from datasette.utils import actor_matches_allow, add_cors_headers\r\n+from datasette.permissions import PERMISSIONS\r\n from .base import BaseView\r\n import secrets\r\n import time\r\n@@ -138,9 +140,34 @@ class AllowDebugView(BaseView):\r\n                 \"error\": \"\\n\\n\".join(errors) if errors else \"\",\r\n                 \"actor_input\": actor_input,\r\n                 \"allow_input\": allow_input,\r\n+                \"permissions\": PERMISSIONS,\r\n             },\r\n         )\r\n \r\n+    async def post(self, request):\r\n+        vars = await request.post_vars()\r\n+        actor = json.loads(vars[\"actor\"])\r\n+        permission = vars[\"permission\"]\r\n+        resource_1 = vars[\"resource_1\"]\r\n+        resource_2 = vars[\"resource_2\"]\r\n+        resource = []\r\n+        if resource_1:\r\n+            resource.append(resource_1)\r\n+        if resource_2:\r\n+            resource.append(resource_2)\r\n+        resource = tuple(resource)\r\n+        result = await self.ds.permission_allowed(\r\n+            actor, permission, resource, default=\"USE_DEFAULT\"\r\n+        )\r\n+        return Response.json(\r\n+            {\r\n+                \"actor\": actor,\r\n+                \"permission\": permission,\r\n+                \"resource\": resource,\r\n+                \"result\": result,\r\n+            }\r\n+        )\r\n+\r\n \r\n class MessagesDebugView(BaseView):\r\n     name = \"messages_debug\"\r\n```\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1855#issuecomment-1301594495", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1855", "id": 1301594495, "node_id": "IC_kwDOBm6k_c5NlMF_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T03:11:17Z", "updated_at": "2022-11-03T03:11:17Z", "author_association": "OWNER", "body": "Maybe the way to do this is through a new standard mechanism on the actor: a set of additional restrictions, e.g.:\r\n\r\n```\r\n{\r\n  \"id\": \"root\",\r\n  \"_r\": {\r\n    \"a\": [\"ir\", \"ur\", \"dr\"],\r\n    \"d\": {\r\n        \"fixtures\": [\"ir\", \"ur\", \"dr\"]\r\n    },\r\n    \"t\": {\r\n        \"fixtures\": {\r\n            \"searchable\": [\"ir\"]\r\n        }\r\n    }\r\n}\r\n```\r\n`\"a\"` is \"all permissions\" - these apply to everything.\r\n`\"d\"` permissions only apply to the specified database\r\n`\"t\"` permissions only apply to the specified table\r\n\r\nThe way this works is there's a default [permission_allowed(datasette, actor, action, resource)](https://docs.datasette.io/en/stable/plugin_hooks.html#id25) hook which only consults these, and crucially just says NO if those rules do not match.\r\n\r\nIn this way it would apply as an extra layer of permission rules over the defaults (which for this `root` instance would all return yes).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1423336089, "label": "`datasette create-token` ability to create tokens with a reduced set of permissions"}, "performed_via_github_app": null}