{"html_url": "https://github.com/simonw/datasette/issues/1439#issuecomment-1060870237", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1439", "id": 1060870237, "node_id": "IC_kwDOBm6k_c4_O5hd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T16:19:22Z", "updated_at": "2022-03-07T16:19:22Z", "author_association": "OWNER", "body": "I didn't need to do any of the fancy regular expression routing stuff after all, since the new dash encoding format avoids using `/` so a simple `[^/]+` can capture the correct segments from the URL.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 973139047, "label": "Rethink how .ext formats (v.s. ?_format=) works before 1.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1650#issuecomment-1060836262", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1650", "id": 1060836262, "node_id": "IC_kwDOBm6k_c4_OxOm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T15:52:09Z", "updated_at": "2022-03-07T15:52:09Z", "author_association": "OWNER", "body": "This is a bit tricky.\r\n\r\nI tried this, sending a redirect only if a 404 happens:\r\n\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 8c5480c..420664c 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -1211,6 +1211,10 @@ class DatasetteRouter:\r\n return await self.handle_404(request, send)\r\n \r\n async def handle_404(self, request, send, exception=None):\r\n+ # If path contains % encoding, redirect to dash encoding\r\n+ if '%' in request.scope[\"path\"]:\r\n+ await asgi_send_redirect(send, request.scope[\"path\"].replace(\"%\", \"-\"))\r\n+ return\r\n # If URL has a trailing slash, redirect to URL without it\r\n path = request.scope.get(\r\n \"raw_path\", request.scope[\"path\"].encode(\"utf8\")\r\n```\r\nBut this URL didn't work:\r\n\r\n- http://127.0.0.1:8001/fivethirtyeight/twitter-ratio%2Fsenators\r\n\r\nI was expecting that to redirect to this page:\r\n\r\n- http://127.0.0.1:8001/fivethirtyeight/twitter-2Dratio-2Fsenators\r\n\r\nBut instead it took me to another 404:\r\n\r\n- http://127.0.0.1:8001/fivethirtyeight/twitter-ratio%2Fsenators\r\n\r\nThis is because that URL contains both a %-escaped `/` AND a plain `-` - which was not escaped in the old system but is escaped in the new system.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1160750713, "label": "Implement redirects from old % encoding to new dash encoding"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1650#issuecomment-1060863311", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1650", "id": 1060863311, "node_id": "IC_kwDOBm6k_c4_O31P", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T16:13:17Z", "updated_at": "2022-03-07T16:13:17Z", "author_association": "OWNER", "body": "This doesn't seem to work.\r\n\r\nhttps://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv should be redirecting now that this is deployed - which it is, because https://latest.datasette.io/-/versions shows 644d25d1de78a36b105cca479e7b3e4375a6eadc - but I'm not getting that redirect.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1160750713, "label": "Implement redirects from old % encoding to new dash encoding"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1650#issuecomment-1060864823", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1650", "id": 1060864823, "node_id": "IC_kwDOBm6k_c4_O4M3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T16:14:33Z", "updated_at": "2022-03-07T16:14:33Z", "author_association": "OWNER", "body": "Same problem here: https://fivethirtyeight.datasettes.com/fivethirtyeight/ahca-2Dpolls%2Fahca_polls should redirect but doesn't.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1160750713, "label": "Implement redirects from old % encoding to new dash encoding"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1650#issuecomment-1061038414", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1650", "id": 1061038414, "node_id": "IC_kwDOBm6k_c4_PilO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T19:14:04Z", "updated_at": "2022-03-07T19:14:04Z", "author_association": "OWNER", "body": "The problem seems to be that `http://127.0.0.1:8002/fixtures/table%2Fwith%2Fslashes.csv` doesn't result in a 404 at all. If it did, it would be redirected.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1160750713, "label": "Implement redirects from old % encoding to new dash encoding"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1650#issuecomment-1061041034", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1650", "id": 1061041034, "node_id": "IC_kwDOBm6k_c4_PjOK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T19:16:51Z", "updated_at": "2022-03-07T19:16:51Z", "author_association": "OWNER", "body": "Here's the problem: https://github.com/simonw/datasette/blob/020effe47bf89f35182960a9645f2383a42ebd54/datasette/utils/__init__.py#L1173-L1175\r\n\r\nWhich is called here:\r\n\r\nhttps://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L469-L473\r\n\r\nSo `table%2Fwith%2Fslashes` ends up decoded as if it was using dash encoding.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1160750713, "label": "Implement redirects from old % encoding to new dash encoding"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1651#issuecomment-1060853226", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1651", "id": 1060853226, "node_id": "IC_kwDOBm6k_c4_O1Xq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T16:04:26Z", "updated_at": "2022-03-07T16:04:26Z", "author_association": "OWNER", "body": "Here's the relevant code:\r\n\r\nhttps://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/utils/__init__.py#L753-L772\r\n\r\nhttps://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L451-L479", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1161584460, "label": "Get rid of the no-longer necessary ?_format=json hack for tables called x.json"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1651#issuecomment-1061053094", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1651", "id": 1061053094, "node_id": "IC_kwDOBm6k_c4_PmKm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T19:29:01Z", "updated_at": "2022-03-07T19:29:01Z", "author_association": "OWNER", "body": "I found an obscure bug in #1650 which I can fix with this too. The following test should pass:\r\n```python\r\n@pytest.mark.parametrize(\r\n \"path,expected\",\r\n (\r\n (\r\n \"/fivethirtyeight/twitter-ratio%2Fsenators\",\r\n \"/fivethirtyeight/twitter-2Dratio-2Fsenators\",\r\n ),\r\n (\r\n \"/fixtures/table%2Fwith%2Fslashes.csv\",\r\n \"/fixtures/table-2Fwith-2Fslashes-2Ecsv\",\r\n ),\r\n # query string should be preserved\r\n (\"/foo/bar%2Fbaz?id=5\", \"/foo/bar-2Fbaz?id=5\"),\r\n ),\r\n)\r\ndef test_redirect_percent_encoding_to_dash_encoding(app_client, path, expected):\r\n response = app_client.get(path)\r\n assert response.status == 302\r\n assert response.headers[\"location\"] == expected\r\n```\r\nIt currently fails like this:\r\n```\r\n> assert response.headers[\"location\"] == expected\r\nE AssertionError: assert '/fixtures/table-2Fwith-2Fslashes.csv?_nofacet=1&_nocount=1' == '/fixtures/table-2Fwith-2Fslashes-2Ecsv'\r\nE - /fixtures/table-2Fwith-2Fslashes-2Ecsv\r\nE + /fixtures/table-2Fwith-2Fslashes.csv?_nofacet=1&_nocount=1\r\n```\r\nBecause the logic in that `get_format()` function notices that the table exists, and then weird things happen here: https://github.com/simonw/datasette/blob/1baa030eca375f839f3471237547ab403523e643/datasette/views/base.py#L288-L303\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1161584460, "label": "Get rid of the no-longer necessary ?_format=json hack for tables called x.json"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1651#issuecomment-1061169528", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1651", "id": 1061169528, "node_id": "IC_kwDOBm6k_c4_QCl4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T21:47:01Z", "updated_at": "2022-03-07T21:47:01Z", "author_association": "OWNER", "body": "Wow, this code is difficult to follow! Look at this bit inside the `get_format()` method:\r\n\r\nhttps://github.com/simonw/datasette/blob/bb499942c15c4e2cfa4b6afab8f8debe5948c009/datasette/views/base.py#L469-L478\r\n\r\nThat's modifying the arguments that were extracted from the path by the routing regular expressions to have `table` as ` dash-decoded value! So calling `.get_format()` has the side effect of decoding the table names for you. Nasty.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1161584460, "label": "Get rid of the no-longer necessary ?_format=json hack for tables called x.json"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1651#issuecomment-1061170897", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1651", "id": 1061170897, "node_id": "IC_kwDOBm6k_c4_QC7R", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T21:48:35Z", "updated_at": "2022-03-07T21:48:35Z", "author_association": "OWNER", "body": "My attempts to simplify `get_format()` keep resulting in errors like this one:\r\n```\r\n File \"/Users/simon/Dropbox/Development/datasette/datasette/views/base.py\", line 474, in view_get\r\n response_or_template_contexts = await self.data(\r\nTypeError: TableView.data() missing 1 required positional argument: 'table'\r\n```\r\nI really need to clean this up.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1161584460, "label": "Get rid of the no-longer necessary ?_format=json hack for tables called x.json"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1651#issuecomment-1061223822", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1651", "id": 1061223822, "node_id": "IC_kwDOBm6k_c4_QP2O", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T22:54:54Z", "updated_at": "2022-03-07T22:54:54Z", "author_association": "OWNER", "body": "I'm going to do a review of how URL routing works at the moment for the various views.\r\n\r\nI edited down [the full list](https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/app.py#L997-L1107) a bit - these are the most relevant:\r\n```python\r\nadd_route(IndexView.as_view(self), r\"/(?P(\\.jsono?)?$)\")\r\nadd_route(\r\n DatabaseView.as_view(self),\r\n r\"/(?P[^/]+?)(?P\"\r\n + renderer_regex\r\n + r\"|.jsono|\\.csv)?$\",\r\n)\r\nadd_route(\r\n TableView.as_view(self),\r\n r\"/(?P[^/]+)/(?P[^/]+?$)\",\r\n)\r\nadd_route(\r\n RowView.as_view(self),\r\n r\"/(?P[^/]+)/(?P[^/]+?)/(?P[^/]+?)(?P\"\r\n + renderer_regex\r\n + r\")?$\",\r\n)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1161584460, "label": "Get rid of the no-longer necessary ?_format=json hack for tables called x.json"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1653#issuecomment-1061148807", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1653", "id": 1061148807, "node_id": "IC_kwDOBm6k_c4_P9iH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T21:21:23Z", "updated_at": "2022-03-07T21:21:23Z", "author_association": "OWNER", "body": "This is currently blocked on the fact that Datasette doesn't have a mechanism for sorting by more than one column:\r\n- #197", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1161937073, "label": "Mechanism to default a table to sorting by multiple columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1653#issuecomment-1061150672", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1653", "id": 1061150672, "node_id": "IC_kwDOBm6k_c4_P9_Q", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T21:23:39Z", "updated_at": "2022-03-07T21:23:39Z", "author_association": "OWNER", "body": "There may be a short-term fix for this: table view could start accepting a `?_sort_sql=SQLfragment` parameter, similar to the `?_where=` parameter described here: https://docs.datasette.io/en/stable/json_api.html#special-table-arguments\r\n\r\nThat fragment could then be pre-populated in `metadata`. Makes me think maybe that `?_where=` should be optionally settable in metadata too?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1161937073, "label": "Mechanism to default a table to sorting by multiple columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1654#issuecomment-1061181089", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1654", "id": 1061181089, "node_id": "IC_kwDOBm6k_c4_QFah", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T22:01:38Z", "updated_at": "2022-03-07T22:01:38Z", "author_association": "OWNER", "body": "I'm going to use the [widely adopted](https://www.contributor-covenant.org/adopters/) Contributor Covenant:\r\n\r\nhttps://www.contributor-covenant.org/version/1/4/code-of-conduct/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1161969891, "label": "Adopt a code of conduct"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1654#issuecomment-1061181530", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1654", "id": 1061181530, "node_id": "IC_kwDOBm6k_c4_QFha", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T22:02:06Z", "updated_at": "2022-03-07T22:02:06Z", "author_association": "OWNER", "body": "https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/adding-a-code-of-conduct-to-your-project says this should be called `CODE_OF_CONDUCT.md` in order for GitHub to pick it up.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1161969891, "label": "Adopt a code of conduct"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1654#issuecomment-1061182132", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1654", "id": 1061182132, "node_id": "IC_kwDOBm6k_c4_QFq0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T22:02:43Z", "updated_at": "2022-03-07T22:02:43Z", "author_association": "OWNER", "body": "Neat, GitHub have a template for this https://github.com/simonw/datasette/community/code-of-conduct/new?template=contributor-covenant", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1161969891, "label": "Adopt a code of conduct"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1654#issuecomment-1061184206", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1654", "id": 1061184206, "node_id": "IC_kwDOBm6k_c4_QGLO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T22:04:51Z", "updated_at": "2022-03-07T22:04:51Z", "author_association": "OWNER", "body": "I'm going to add this to the main Datasette repo (done) and the `datasette.io` website too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1161969891, "label": "Adopt a code of conduct"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1654#issuecomment-1061197133", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1654", "id": 1061197133, "node_id": "IC_kwDOBm6k_c4_QJVN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T22:19:35Z", "updated_at": "2022-03-07T22:19:35Z", "author_association": "OWNER", "body": "Also now live on https://datasette.io\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1161969891, "label": "Adopt a code of conduct"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/647#issuecomment-1061226942", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/647", "id": 1061226942, "node_id": "IC_kwDOBm6k_c4_QQm-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-07T23:00:06Z", "updated_at": "2022-03-07T23:00:06Z", "author_association": "OWNER", "body": "This needs to take into account the changes made here:\r\n- #1439\r\n\r\nIn the new encoding scheme, `-` has a special meaning in a table name: https://docs.datasette.io/en/latest/internals.html#dash-encoding\r\n\r\nI think `~` is the right character to use to separate a database name from its hash. `~` should be a URL safe character according to Python's implementation of percent-encoding, see comment here: https://github.com/simonw/datasette/blob/c5791156d92615f25696ba93dae5bb2dcc192c98/datasette/utils/__init__.py#L1146-L1152\r\n\r\nSo the plugin could check for `dbname~hash` and react based on that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 531755959, "label": "Move hashed URL mode out to a plugin"}, "performed_via_github_app": null}