{"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692832113", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692832113, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjgzMjExMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T16:34:53Z", "updated_at": "2020-09-15T16:37:43Z", "author_association": "OWNER", "body": "This is so weird. In the test I wrote for this the following passed:\r\n\r\n response = magic_parameters_client.post(\"/data/runme_post?_json=1\", {}, csrftoken_from=True)\r\n\r\nBut without the `csrftoken_from=True` parameter it failed with the bindings error:\r\n\r\n response = magic_parameters_client.post(\"/data/runme_post?_json=1\", {})\r\n\r\nHere's the test I wrote:\r\n\r\n```python\r\ndef test_magic_parameters_json_body(magic_parameters_client):\r\n magic_parameters_client.ds._metadata[\"databases\"][\"data\"][\"queries\"][\"runme_post\"][\r\n \"sql\"\r\n ] = \"insert into logs (line) values (:_header_host)\"\r\n response = magic_parameters_client.post(\"/data/runme_post?_json=1\", {}, csrftoken_from=True)\r\n assert response.status == 200\r\n assert response.json[\"ok\"], response.json\r\n post_actual = magic_parameters_client.get(\r\n \"/data/logs.json?_sort_desc=rowid&_shape=array\"\r\n ).json[0][\"line\"]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692834064", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692834064, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjgzNDA2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T16:38:21Z", "updated_at": "2020-09-15T16:38:21Z", "author_association": "OWNER", "body": "So the mystery here is why does omitting `csrftoken_from=True` break the `MagicParameters` mechanism?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692834670", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692834670, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjgzNDY3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T16:39:29Z", "updated_at": "2020-09-15T16:39:29Z", "author_association": "OWNER", "body": "Relevant code: https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L222-L236\r\n\r\nThis issue may not be about `_json=1` interacting with magic parameters after all.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692835066", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692835066, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjgzNTA2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T16:40:12Z", "updated_at": "2020-09-15T16:40:12Z", "author_association": "OWNER", "body": "Is the bug here that magic parameters are incompatible with CSRF-exempt requests (e.g. request with no cookies)?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692940375", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692940375, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk0MDM3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T19:47:09Z", "updated_at": "2020-09-15T19:47:09Z", "author_association": "OWNER", "body": "Yes! The tests all pass if I update the test function to do this:\r\n```python\r\n response = magic_parameters_client.post(\r\n \"/data/runme_post{}\".format(qs),\r\n {\"ignore_me\": \"1\"},\r\n csrftoken_from=use_csrf or None,\r\n allow_redirects=False,\r\n )\r\n```\r\nSo the bug only occurs if the POST body is completely empty.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692945504", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692945504, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk0NTUwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T19:57:10Z", "updated_at": "2020-09-15T19:57:10Z", "author_association": "OWNER", "body": "So the problem actually occurs when the `MagicParameters` class wraps an empty dictionary.\r\n\r\nRelevant code:\r\n\r\nhttps://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L228-L236\r\n\r\nAnd:\r\n\r\nhttps://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L364-L383\r\n\r\nI'm passing a special magic parameters dictionary for the Python `sqlite3` module to look up parameters in. When that dictionary is `{}` a `__len__` check is performed on that dictionary, the result comes back as 0 and as a result it assumes there are no parameters.\r\n\r\nI tracked down the relevant C code:\r\n\r\nhttps://github.com/python/cpython/blob/81715808716198471fbca0a3db42ac408468dbc5/Modules/_sqlite/statement.c#L218-L237\r\n\r\n```c\r\n Py_BEGIN_ALLOW_THREADS\r\n num_params_needed = sqlite3_bind_parameter_count(self->st);\r\n Py_END_ALLOW_THREADS\r\n\r\n if (PyTuple_CheckExact(parameters) || PyList_CheckExact(parameters) || (!PyDict_Check(parameters) && PySequence_Check(parameters))) {\r\n /* parameters passed as sequence */\r\n if (PyTuple_CheckExact(parameters)) {\r\n num_params = PyTuple_GET_SIZE(parameters);\r\n } else if (PyList_CheckExact(parameters)) {\r\n num_params = PyList_GET_SIZE(parameters);\r\n } else {\r\n num_params = PySequence_Size(parameters);\r\n }\r\n if (num_params != num_params_needed) {\r\n PyErr_Format(pysqlite_ProgrammingError,\r\n \"Incorrect number of bindings supplied. The current \"\r\n \"statement uses %d, and there are %zd supplied.\",\r\n num_params_needed, num_params);\r\n return;\r\n }\r\n```\r\n\r\nIt looks to me like this should fail if the number of keys known to be in the dictionary differs from the number of named parameters in the query. But if those numbers fail to match it still works as far as I can tell - it's only dictionary length of 0 that is causing the problems.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692946616", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692946616, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk0NjYxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T19:59:21Z", "updated_at": "2020-09-15T19:59:21Z", "author_association": "OWNER", "body": "I wish I could call https://www.sqlite.org/c3ref/bind_parameter_count.html and https://www.sqlite.org/c3ref/bind_parameter_name.html from Python.\r\n\r\nMight be possible to do that using `ctypes` - see this example code: https://mail.python.org/pipermail//pypy-commit/2013-February/071372.html\r\n\r\n```python\r\n param_count = lib.sqlite3_bind_parameter_count(self.statement)\r\n for idx in range(1, param_count + 1):\r\n param_name = lib.sqlite3_bind_parameter_name(self.statement,\r\n idx)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692951144", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692951144, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk1MTE0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T20:08:12Z", "updated_at": "2020-09-15T20:08:12Z", "author_association": "OWNER", "body": "I think the easiest fix is for me to ensure that calls to `__len__` on the `MagicParameters` class always return at least 1.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/492#issuecomment-692953174", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/492", "id": 692953174, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk1MzE3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T20:12:29Z", "updated_at": "2020-09-15T20:12:29Z", "author_association": "OWNER", "body": "I fixed this in ea340cf320a2566d24517fb4a0c9852c5059e771 for #963 (a duplicate of this issue).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 449854604, "label": "Facets not correctly persisted in hidden form fields"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/946#issuecomment-692955379", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/946", "id": 692955379, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk1NTM3OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T20:16:50Z", "updated_at": "2020-09-15T20:16:50Z", "author_association": "OWNER", "body": "Can't reproduce this bug now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 682184050, "label": "Exception in tracing code"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/956#issuecomment-692955850", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/956", "id": 692955850, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk1NTg1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T20:17:49Z", "updated_at": "2020-09-15T20:17:49Z", "author_association": "OWNER", "body": "I think I've fixed this with recent changes I made as part of #941 - but I won't know until I release the next version.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688427751, "label": "Push to Docker Hub failed - but it shouldn't run for alpha releases anyway"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/956#issuecomment-692965022", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/956", "id": 692965022, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk2NTAyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T20:36:34Z", "updated_at": "2020-09-15T20:36:34Z", "author_association": "OWNER", "body": "https://hub.docker.com/r/datasetteproject/datasette/tags - 0.49.1 was successfully pushed to Docker Hub by https://github.com/simonw/datasette/runs/1119815175?check_suite_focus=true", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688427751, "label": "Push to Docker Hub failed - but it shouldn't run for alpha releases anyway"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/849#issuecomment-692965391", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/849", "id": 692965391, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk2NTM5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T20:37:14Z", "updated_at": "2020-09-15T20:37:14Z", "author_association": "OWNER", "body": "I've been running on `main` for a while now with no issues.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 639072811, "label": "Rename master branch to main"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/634#issuecomment-692965761", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/634", "id": 692965761, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk2NTc2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T20:37:58Z", "updated_at": "2020-09-15T20:37:58Z", "author_association": "OWNER", "body": "I fixed this in 5e0b72247ecab4ce0fcec599b77a83d73a480872", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 522352520, "label": "Don't run tests twice when releasing a tag"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/888#issuecomment-692966625", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/888", "id": 692966625, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk2NjYyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T20:39:49Z", "updated_at": "2020-09-15T20:39:49Z", "author_association": "OWNER", "body": "Thanks, I've fixed that now. It only affected the GitHub release notes - the ones at https://docs.datasette.io/en/stable/changelog.html#v0-45 had the correct links.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 649702801, "label": "URLs in release notes point to 127.0.0.1"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/889#issuecomment-692967123", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/889", "id": 692967123, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk2NzEyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T20:40:52Z", "updated_at": "2020-09-15T20:40:52Z", "author_association": "OWNER", "body": "Thanks - I've fixed this in `datasette-media` and the other plugins that use that hook now I think.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 649907676, "label": "asgi_wrapper plugin hook is crashing at startup"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/877#issuecomment-692967733", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/877", "id": 692967733, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk2NzczMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T20:42:04Z", "updated_at": "2020-09-15T20:42:04Z", "author_association": "OWNER", "body": "I'm not going to drop CSRF protection - it's still needed for older browsers - but I have relaxed the circumstances under which it is applied. It only applies to requests that include cookies for example, so API clients that don't send cookies don't need to worry about it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648421105, "label": "Consider dropping explicit CSRF protection entirely?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/891#issuecomment-692968792", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/891", "id": 692968792, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk2ODc5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T20:44:15Z", "updated_at": "2020-09-15T20:44:15Z", "author_association": "OWNER", "body": "https://github.com/peter-wangxu/persist-queue/issues/74 warns that this might not work with PyPy.\r\n\r\nI could solve that with:\r\n```python\r\nif hasattr(sqlite3, \"enable_callback_tracebacks\"):\r\n sqlite3.enable_callback_tracebacks(True)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 653529088, "label": "Consider using enable_callback_tracebacks(True)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/891#issuecomment-692998061", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/891", "id": 692998061, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk5ODA2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T21:49:03Z", "updated_at": "2020-09-15T21:49:03Z", "author_association": "OWNER", "body": "I've been trying to figure out why this is an optional setting that defaults to off.\r\n\r\nI think it's because it writes directly to `stderr`, so the maintainers of `sqlite3` reasonably decided that people should be able to opt in to that rather than having weird stuff show up on `stderr` that they weren't expecting.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 653529088, "label": "Consider using enable_callback_tracebacks(True)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/891#issuecomment-692999893", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/891", "id": 692999893, "node_id": "MDEyOklzc3VlQ29tbWVudDY5Mjk5OTg5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T21:53:36Z", "updated_at": "2020-09-15T21:53:36Z", "author_association": "OWNER", "body": "Here's the commit (from 15 years ago) where `enable_callback_tracebacks` was first added: https://github.com/ghaering/pysqlite/commit/1e8bd36be93b7d7425910642b72e4152c77b0dfd\r\n\r\n> - Exceptions in callbacks lead to the query being aborted now instead of silently leading to generating values.\r\n> - Exceptions in callbacks can be echoed to stderr if you call the module level function enable_callback_tracebacks: enable_callback_tracebacks(1).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 653529088, "label": "Consider using enable_callback_tracebacks(True)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692927867", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692927867, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjkyNzg2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T19:25:23Z", "updated_at": "2020-09-15T19:25:23Z", "author_association": "OWNER", "body": "Hunch: I think the `asgi-csrf` middleware may be consuming the request body and failing to restore it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692937150", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692937150, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjkzNzE1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T19:42:57Z", "updated_at": "2020-09-15T19:42:57Z", "author_association": "OWNER", "body": "New (failing) test:\r\n```python\r\n@pytest.mark.parametrize(\"use_csrf\", [True, False])\r\n@pytest.mark.parametrize(\"return_json\", [True, False])\r\ndef test_magic_parameters_csrf_json(magic_parameters_client, use_csrf, return_json):\r\n magic_parameters_client.ds._metadata[\"databases\"][\"data\"][\"queries\"][\"runme_post\"][\r\n \"sql\"\r\n ] = \"insert into logs (line) values (:_header_host)\"\r\n qs = \"\"\r\n if return_json:\r\n qs = \"?_json=1\"\r\n response = magic_parameters_client.post(\r\n \"/data/runme_post{}\".format(qs),\r\n {},\r\n csrftoken_from=use_csrf or None,\r\n allow_redirects=False,\r\n )\r\n if return_json:\r\n assert response.status == 200\r\n assert response.json[\"ok\"], response.json\r\n else:\r\n assert response.status == 302\r\n messages = magic_parameters_client.ds.unsign(\r\n response.cookies[\"ds_messages\"], \"messages\"\r\n )\r\n assert [[\"Query executed, 1 row affected\", 1]] == messages\r\n post_actual = magic_parameters_client.get(\r\n \"/data/logs.json?_sort_desc=rowid&_shape=array\"\r\n ).json[0][\"line\"]\r\n assert post_actual == \"localhost\"\r\n```\r\nIt passes twice, fails twice - failures are for the ones where `use_csrf` is `False`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/967#issuecomment-692938935", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/967", "id": 692938935, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MjkzODkzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T19:44:21Z", "updated_at": "2020-09-15T19:44:41Z", "author_association": "OWNER", "body": "While I'm running the above test, in the rounds that work the `receive()` awaitable returns `{'type': 'http.request', 'body': b'csrftoken=IlpwUGlSMFVVa3Z3ZlVoamQi.uY2U1tF4i0M-5M6x34vnBCmJgr0'}`\r\n\r\nIn the rounds that fails it returns `{'type': 'http.request'}`\r\n\r\nSo it looks like the `csrftoken_from=True` parameter may be helping just by ensuring the `body` key is present and not missing. I wonder if it would work if a body of `b''` was present there?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702069429, "label": "Writable canned queries with magic parameters fail if POST body is empty"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/891#issuecomment-693000522", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/891", "id": 693000522, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MzAwMDUyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T21:55:11Z", "updated_at": "2020-09-15T21:55:11Z", "author_association": "OWNER", "body": "I'm going to turn this on. If people complain about it I can turn it off again (or make it a configuration setting).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 653529088, "label": "Consider using enable_callback_tracebacks(True)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/891#issuecomment-693001937", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/891", "id": 693001937, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MzAwMTkzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T21:58:56Z", "updated_at": "2020-09-15T21:58:56Z", "author_association": "OWNER", "body": "Here's what that looks like:\r\n```\r\nTraceback (most recent call last):\r\n File \"/Users/simon/Dropbox/Development/datasette/plugins/sql_error.py\", line 5, in oh_no_error\r\n return 100 / 0\r\nZeroDivisionError: division by zero\r\nERROR: conn=, sql = 'select oh_no_error()', params = {}: user-defined function raised exception\r\nINFO: 127.0.0.1:54066 - \"GET /data?sql=select+oh_no_error%28%29 HTTP/1.1\" 400 Bad Request\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 653529088, "label": "Consider using enable_callback_tracebacks(True)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-693003652", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 693003652, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MzAwMzY1Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T22:03:08Z", "updated_at": "2020-09-15T22:03:08Z", "author_association": "OWNER", "body": "I'm not going to mess around with formats - you'll get back the exact response that a web client would receive.\r\n\r\nQuestion: what should the response object look like? e.g. if you do:\r\n\r\n response = await datasette.get(\"/db/table.json\")\r\n\r\nWhat should `response` be?\r\n\r\nI could reuse the Datasette `Response` class from `datasette.utils.asgi`. This would work well for regular responses which just have a status code, some headers and a response body. It wouldn't be great for streaming responses though such as you get back from `?_stream=1` CSV exports.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-693004296", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 693004296, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MzAwNDI5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T22:04:54Z", "updated_at": "2020-09-15T22:04:54Z", "author_association": "OWNER", "body": "So what should I do about streaming responses?\r\n\r\nI could deliberately ignore them - through an exception if you attempt to run `await datasette.get(...)` against a streaming URL.\r\n\r\nI could load the entire response into memory and return it as a wrapped object.\r\n\r\nI could support some kind of asynchronous iterator mechanism. This would be pretty elegant if I could decide the right syntax for it - it would allow plugins to take advantage of other internal URLs that return streaming content without needing to load that content entirely into memory in order to process it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-693004572", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 693004572, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MzAwNDU3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T22:05:39Z", "updated_at": "2020-09-15T22:05:39Z", "author_association": "OWNER", "body": "Maybe these methods become the way most Datasette tests are written, replacing the existing `TestClient` mechanism?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-693004770", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 693004770, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MzAwNDc3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T22:06:13Z", "updated_at": "2020-09-15T22:06:13Z", "author_association": "OWNER", "body": "I'm tempted to create a `await datasette.request()` method which can take any HTTP verb - then have `datasette.get()` and `datasette.post()` as thin wrappers around it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-693005033", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 693005033, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MzAwNTAzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T22:06:58Z", "updated_at": "2020-09-15T22:10:58Z", "author_association": "OWNER", "body": "What if `datasette.get()` was an alias for `httpx.get()`, pre-configured to route to the correct application? And with some sugar that added `http://localhost/` to the beginning of the path if it was missing?\r\n\r\nThis would make `httpx` a dependency of core Datasette, which I think is OK.\r\n\r\nIt would also solve the return type problem: I would return whatever `httpx` returns.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-693007512", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 693007512, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MzAwNzUxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T22:13:30Z", "updated_at": "2020-09-15T22:13:30Z", "author_association": "OWNER", "body": "I could solve streaming using something like this:\r\n```python\r\nasync with datasette.stream(\"GET\", \"/fixtures/compound_three_primary_keys.csv?_stream=on&_size=max\") as response:\r\n async for chunk in response.aiter_bytes():\r\n print(chunk)\r\n```\r\nWhich would be a wrapper around `AsyncClient.stream(method, url, ...)` from https://www.python-httpx.org/async/#streaming-responses", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-693008540", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 693008540, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MzAwODU0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T22:16:07Z", "updated_at": "2020-09-15T22:16:07Z", "author_association": "OWNER", "body": "I think I can use `async with httpx.AsyncClient(base_url=\"http://localhost/\") as client:` to ensure I don't need to use `http://localhost/` on every call.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-693010291", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 693010291, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MzAxMDI5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-15T22:20:55Z", "updated_at": "2020-09-15T22:20:55Z", "author_association": "OWNER", "body": "Should I instantiate a single `Client` and reuse it for all internal requests, or can I instantiate a new `Client` for each request?\r\n\r\nhttps://www.python-httpx.org/advanced/#why-use-a-client says that the main benefit of a Client instance is HTTP connection pooling - which isn't an issue for these internal requests since they won't be using the HTTP protocol at all, they'll be calling the ASGI application directly.\r\n\r\nSo I'm leaning towards instantiating a fresh client for every internal request. I'll run a microbenchmark to check that this doesn't have any unpleasant performance implications.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null}