html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/967#issuecomment-692832113,https://api.github.com/repos/simonw/datasette/issues/967,692832113,MDEyOklzc3VlQ29tbWVudDY5MjgzMjExMw==,9599,simonw,2020-09-15T16:34:53Z,2020-09-15T16:37:43Z,OWNER,"This is so weird. In the test I wrote for this the following passed:
response = magic_parameters_client.post(""/data/runme_post?_json=1"", {}, csrftoken_from=True)
But without the `csrftoken_from=True` parameter it failed with the bindings error:
response = magic_parameters_client.post(""/data/runme_post?_json=1"", {})
Here's the test I wrote:
```python
def test_magic_parameters_json_body(magic_parameters_client):
magic_parameters_client.ds._metadata[""databases""][""data""][""queries""][""runme_post""][
""sql""
] = ""insert into logs (line) values (:_header_host)""
response = magic_parameters_client.post(""/data/runme_post?_json=1"", {}, csrftoken_from=True)
assert response.status == 200
assert response.json[""ok""], response.json
post_actual = magic_parameters_client.get(
""/data/logs.json?_sort_desc=rowid&_shape=array""
).json[0][""line""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692834064,https://api.github.com/repos/simonw/datasette/issues/967,692834064,MDEyOklzc3VlQ29tbWVudDY5MjgzNDA2NA==,9599,simonw,2020-09-15T16:38:21Z,2020-09-15T16:38:21Z,OWNER,So the mystery here is why does omitting `csrftoken_from=True` break the `MagicParameters` mechanism?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692834670,https://api.github.com/repos/simonw/datasette/issues/967,692834670,MDEyOklzc3VlQ29tbWVudDY5MjgzNDY3MA==,9599,simonw,2020-09-15T16:39:29Z,2020-09-15T16:39:29Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L222-L236
This issue may not be about `_json=1` interacting with magic parameters after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692835066,https://api.github.com/repos/simonw/datasette/issues/967,692835066,MDEyOklzc3VlQ29tbWVudDY5MjgzNTA2Ng==,9599,simonw,2020-09-15T16:40:12Z,2020-09-15T16:40:12Z,OWNER,Is the bug here that magic parameters are incompatible with CSRF-exempt requests (e.g. request with no cookies)?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692927867,https://api.github.com/repos/simonw/datasette/issues/967,692927867,MDEyOklzc3VlQ29tbWVudDY5MjkyNzg2Nw==,9599,simonw,2020-09-15T19:25:23Z,2020-09-15T19:25:23Z,OWNER,Hunch: I think the `asgi-csrf` middleware may be consuming the request body and failing to restore it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692937150,https://api.github.com/repos/simonw/datasette/issues/967,692937150,MDEyOklzc3VlQ29tbWVudDY5MjkzNzE1MA==,9599,simonw,2020-09-15T19:42:57Z,2020-09-15T19:42:57Z,OWNER,"New (failing) test:
```python
@pytest.mark.parametrize(""use_csrf"", [True, False])
@pytest.mark.parametrize(""return_json"", [True, False])
def test_magic_parameters_csrf_json(magic_parameters_client, use_csrf, return_json):
magic_parameters_client.ds._metadata[""databases""][""data""][""queries""][""runme_post""][
""sql""
] = ""insert into logs (line) values (:_header_host)""
qs = """"
if return_json:
qs = ""?_json=1""
response = magic_parameters_client.post(
""/data/runme_post{}"".format(qs),
{},
csrftoken_from=use_csrf or None,
allow_redirects=False,
)
if return_json:
assert response.status == 200
assert response.json[""ok""], response.json
else:
assert response.status == 302
messages = magic_parameters_client.ds.unsign(
response.cookies[""ds_messages""], ""messages""
)
assert [[""Query executed, 1 row affected"", 1]] == messages
post_actual = magic_parameters_client.get(
""/data/logs.json?_sort_desc=rowid&_shape=array""
).json[0][""line""]
assert post_actual == ""localhost""
```
It passes twice, fails twice - failures are for the ones where `use_csrf` is `False`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692938935,https://api.github.com/repos/simonw/datasette/issues/967,692938935,MDEyOklzc3VlQ29tbWVudDY5MjkzODkzNQ==,9599,simonw,2020-09-15T19:44:21Z,2020-09-15T19:44:41Z,OWNER,"While I'm running the above test, in the rounds that work the `receive()` awaitable returns `{'type': 'http.request', 'body': b'csrftoken=IlpwUGlSMFVVa3Z3ZlVoamQi.uY2U1tF4i0M-5M6x34vnBCmJgr0'}`
In the rounds that fails it returns `{'type': 'http.request'}`
So it looks like the `csrftoken_from=True` parameter may be helping just by ensuring the `body` key is present and not missing. I wonder if it would work if a body of `b''` was present there?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692940375,https://api.github.com/repos/simonw/datasette/issues/967,692940375,MDEyOklzc3VlQ29tbWVudDY5Mjk0MDM3NQ==,9599,simonw,2020-09-15T19:47:09Z,2020-09-15T19:47:09Z,OWNER,"Yes! The tests all pass if I update the test function to do this:
```python
response = magic_parameters_client.post(
""/data/runme_post{}"".format(qs),
{""ignore_me"": ""1""},
csrftoken_from=use_csrf or None,
allow_redirects=False,
)
```
So the bug only occurs if the POST body is completely empty.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692945504,https://api.github.com/repos/simonw/datasette/issues/967,692945504,MDEyOklzc3VlQ29tbWVudDY5Mjk0NTUwNA==,9599,simonw,2020-09-15T19:57:10Z,2020-09-15T19:57:10Z,OWNER,"So the problem actually occurs when the `MagicParameters` class wraps an empty dictionary.
Relevant code:
https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L228-L236
And:
https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L364-L383
I'm passing a special magic parameters dictionary for the Python `sqlite3` module to look up parameters in. When that dictionary is `{}` a `__len__` check is performed on that dictionary, the result comes back as 0 and as a result it assumes there are no parameters.
I tracked down the relevant C code:
https://github.com/python/cpython/blob/81715808716198471fbca0a3db42ac408468dbc5/Modules/_sqlite/statement.c#L218-L237
```c
Py_BEGIN_ALLOW_THREADS
num_params_needed = sqlite3_bind_parameter_count(self->st);
Py_END_ALLOW_THREADS
if (PyTuple_CheckExact(parameters) || PyList_CheckExact(parameters) || (!PyDict_Check(parameters) && PySequence_Check(parameters))) {
/* parameters passed as sequence */
if (PyTuple_CheckExact(parameters)) {
num_params = PyTuple_GET_SIZE(parameters);
} else if (PyList_CheckExact(parameters)) {
num_params = PyList_GET_SIZE(parameters);
} else {
num_params = PySequence_Size(parameters);
}
if (num_params != num_params_needed) {
PyErr_Format(pysqlite_ProgrammingError,
""Incorrect number of bindings supplied. The current ""
""statement uses %d, and there are %zd supplied."",
num_params_needed, num_params);
return;
}
```
It looks to me like this should fail if the number of keys known to be in the dictionary differs from the number of named parameters in the query. But if those numbers fail to match it still works as far as I can tell - it's only dictionary length of 0 that is causing the problems.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692946616,https://api.github.com/repos/simonw/datasette/issues/967,692946616,MDEyOklzc3VlQ29tbWVudDY5Mjk0NjYxNg==,9599,simonw,2020-09-15T19:59:21Z,2020-09-15T19:59:21Z,OWNER,"I wish I could call https://www.sqlite.org/c3ref/bind_parameter_count.html and https://www.sqlite.org/c3ref/bind_parameter_name.html from Python.
Might be possible to do that using `ctypes` - see this example code: https://mail.python.org/pipermail//pypy-commit/2013-February/071372.html
```python
param_count = lib.sqlite3_bind_parameter_count(self.statement)
for idx in range(1, param_count + 1):
param_name = lib.sqlite3_bind_parameter_name(self.statement,
idx)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692951144,https://api.github.com/repos/simonw/datasette/issues/967,692951144,MDEyOklzc3VlQ29tbWVudDY5Mjk1MTE0NA==,9599,simonw,2020-09-15T20:08:12Z,2020-09-15T20:08:12Z,OWNER,I think the easiest fix is for me to ensure that calls to `__len__` on the `MagicParameters` class always return at least 1.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/492#issuecomment-692953174,https://api.github.com/repos/simonw/datasette/issues/492,692953174,MDEyOklzc3VlQ29tbWVudDY5Mjk1MzE3NA==,9599,simonw,2020-09-15T20:12:29Z,2020-09-15T20:12:29Z,OWNER,I fixed this in ea340cf320a2566d24517fb4a0c9852c5059e771 for #963 (a duplicate of this issue).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",449854604,Facets not correctly persisted in hidden form fields,
https://github.com/simonw/datasette/issues/946#issuecomment-692955379,https://api.github.com/repos/simonw/datasette/issues/946,692955379,MDEyOklzc3VlQ29tbWVudDY5Mjk1NTM3OQ==,9599,simonw,2020-09-15T20:16:50Z,2020-09-15T20:16:50Z,OWNER,Can't reproduce this bug now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",682184050,Exception in tracing code,
https://github.com/simonw/datasette/issues/956#issuecomment-692955850,https://api.github.com/repos/simonw/datasette/issues/956,692955850,MDEyOklzc3VlQ29tbWVudDY5Mjk1NTg1MA==,9599,simonw,2020-09-15T20:17:49Z,2020-09-15T20:17:49Z,OWNER,I think I've fixed this with recent changes I made as part of #941 - but I won't know until I release the next version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/956#issuecomment-692965022,https://api.github.com/repos/simonw/datasette/issues/956,692965022,MDEyOklzc3VlQ29tbWVudDY5Mjk2NTAyMg==,9599,simonw,2020-09-15T20:36:34Z,2020-09-15T20:36:34Z,OWNER,https://hub.docker.com/r/datasetteproject/datasette/tags - 0.49.1 was successfully pushed to Docker Hub by https://github.com/simonw/datasette/runs/1119815175?check_suite_focus=true,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/849#issuecomment-692965391,https://api.github.com/repos/simonw/datasette/issues/849,692965391,MDEyOklzc3VlQ29tbWVudDY5Mjk2NTM5MQ==,9599,simonw,2020-09-15T20:37:14Z,2020-09-15T20:37:14Z,OWNER,I've been running on `main` for a while now with no issues.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/634#issuecomment-692965761,https://api.github.com/repos/simonw/datasette/issues/634,692965761,MDEyOklzc3VlQ29tbWVudDY5Mjk2NTc2MQ==,9599,simonw,2020-09-15T20:37:58Z,2020-09-15T20:37:58Z,OWNER,I fixed this in 5e0b72247ecab4ce0fcec599b77a83d73a480872,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",522352520,Don't run tests twice when releasing a tag,
https://github.com/simonw/datasette/issues/888#issuecomment-692966625,https://api.github.com/repos/simonw/datasette/issues/888,692966625,MDEyOklzc3VlQ29tbWVudDY5Mjk2NjYyNQ==,9599,simonw,2020-09-15T20:39:49Z,2020-09-15T20:39:49Z,OWNER,"Thanks, I've fixed that now. It only affected the GitHub release notes - the ones at https://docs.datasette.io/en/stable/changelog.html#v0-45 had the correct links.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649702801,URLs in release notes point to 127.0.0.1,
https://github.com/simonw/datasette/issues/889#issuecomment-692967123,https://api.github.com/repos/simonw/datasette/issues/889,692967123,MDEyOklzc3VlQ29tbWVudDY5Mjk2NzEyMw==,9599,simonw,2020-09-15T20:40:52Z,2020-09-15T20:40:52Z,OWNER,Thanks - I've fixed this in `datasette-media` and the other plugins that use that hook now I think.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649907676,asgi_wrapper plugin hook is crashing at startup,
https://github.com/simonw/datasette/issues/877#issuecomment-692967733,https://api.github.com/repos/simonw/datasette/issues/877,692967733,MDEyOklzc3VlQ29tbWVudDY5Mjk2NzczMw==,9599,simonw,2020-09-15T20:42:04Z,2020-09-15T20:42:04Z,OWNER,"I'm not going to drop CSRF protection - it's still needed for older browsers - but I have relaxed the circumstances under which it is applied. It only applies to requests that include cookies for example, so API clients that don't send cookies don't need to worry about it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/891#issuecomment-692968792,https://api.github.com/repos/simonw/datasette/issues/891,692968792,MDEyOklzc3VlQ29tbWVudDY5Mjk2ODc5Mg==,9599,simonw,2020-09-15T20:44:15Z,2020-09-15T20:44:15Z,OWNER,"https://github.com/peter-wangxu/persist-queue/issues/74 warns that this might not work with PyPy.
I could solve that with:
```python
if hasattr(sqlite3, ""enable_callback_tracebacks""):
sqlite3.enable_callback_tracebacks(True)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/891#issuecomment-692998061,https://api.github.com/repos/simonw/datasette/issues/891,692998061,MDEyOklzc3VlQ29tbWVudDY5Mjk5ODA2MQ==,9599,simonw,2020-09-15T21:49:03Z,2020-09-15T21:49:03Z,OWNER,"I've been trying to figure out why this is an optional setting that defaults to off.
I think it's because it writes directly to `stderr`, so the maintainers of `sqlite3` reasonably decided that people should be able to opt in to that rather than having weird stuff show up on `stderr` that they weren't expecting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/891#issuecomment-692999893,https://api.github.com/repos/simonw/datasette/issues/891,692999893,MDEyOklzc3VlQ29tbWVudDY5Mjk5OTg5Mw==,9599,simonw,2020-09-15T21:53:36Z,2020-09-15T21:53:36Z,OWNER,"Here's the commit (from 15 years ago) where `enable_callback_tracebacks` was first added: https://github.com/ghaering/pysqlite/commit/1e8bd36be93b7d7425910642b72e4152c77b0dfd
> - Exceptions in callbacks lead to the query being aborted now instead of silently leading to generating values.
> - Exceptions in callbacks can be echoed to stderr if you call the module level function enable_callback_tracebacks: enable_callback_tracebacks(1).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/891#issuecomment-693000522,https://api.github.com/repos/simonw/datasette/issues/891,693000522,MDEyOklzc3VlQ29tbWVudDY5MzAwMDUyMg==,9599,simonw,2020-09-15T21:55:11Z,2020-09-15T21:55:11Z,OWNER,I'm going to turn this on. If people complain about it I can turn it off again (or make it a configuration setting).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/891#issuecomment-693001937,https://api.github.com/repos/simonw/datasette/issues/891,693001937,MDEyOklzc3VlQ29tbWVudDY5MzAwMTkzNw==,9599,simonw,2020-09-15T21:58:56Z,2020-09-15T21:58:56Z,OWNER,"Here's what that looks like:
```
Traceback (most recent call last):
File ""/Users/simon/Dropbox/Development/datasette/plugins/sql_error.py"", line 5, in oh_no_error
return 100 / 0
ZeroDivisionError: division by zero
ERROR: conn=, sql = 'select oh_no_error()', params = {}: user-defined function raised exception
INFO: 127.0.0.1:54066 - ""GET /data?sql=select+oh_no_error%28%29 HTTP/1.1"" 400 Bad Request
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/943#issuecomment-693003652,https://api.github.com/repos/simonw/datasette/issues/943,693003652,MDEyOklzc3VlQ29tbWVudDY5MzAwMzY1Mg==,9599,simonw,2020-09-15T22:03:08Z,2020-09-15T22:03:08Z,OWNER,"I'm not going to mess around with formats - you'll get back the exact response that a web client would receive.
Question: what should the response object look like? e.g. if you do:
response = await datasette.get(""/db/table.json"")
What should `response` be?
I could reuse the Datasette `Response` class from `datasette.utils.asgi`. This would work well for regular responses which just have a status code, some headers and a response body. It wouldn't be great for streaming responses though such as you get back from `?_stream=1` CSV exports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693004296,https://api.github.com/repos/simonw/datasette/issues/943,693004296,MDEyOklzc3VlQ29tbWVudDY5MzAwNDI5Ng==,9599,simonw,2020-09-15T22:04:54Z,2020-09-15T22:04:54Z,OWNER,"So what should I do about streaming responses?
I could deliberately ignore them - through an exception if you attempt to run `await datasette.get(...)` against a streaming URL.
I could load the entire response into memory and return it as a wrapped object.
I could support some kind of asynchronous iterator mechanism. This would be pretty elegant if I could decide the right syntax for it - it would allow plugins to take advantage of other internal URLs that return streaming content without needing to load that content entirely into memory in order to process it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693004572,https://api.github.com/repos/simonw/datasette/issues/943,693004572,MDEyOklzc3VlQ29tbWVudDY5MzAwNDU3Mg==,9599,simonw,2020-09-15T22:05:39Z,2020-09-15T22:05:39Z,OWNER,"Maybe these methods become the way most Datasette tests are written, replacing the existing `TestClient` mechanism?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693004770,https://api.github.com/repos/simonw/datasette/issues/943,693004770,MDEyOklzc3VlQ29tbWVudDY5MzAwNDc3MA==,9599,simonw,2020-09-15T22:06:13Z,2020-09-15T22:06:13Z,OWNER,I'm tempted to create a `await datasette.request()` method which can take any HTTP verb - then have `datasette.get()` and `datasette.post()` as thin wrappers around it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693005033,https://api.github.com/repos/simonw/datasette/issues/943,693005033,MDEyOklzc3VlQ29tbWVudDY5MzAwNTAzMw==,9599,simonw,2020-09-15T22:06:58Z,2020-09-15T22:10:58Z,OWNER,"What if `datasette.get()` was an alias for `httpx.get()`, pre-configured to route to the correct application? And with some sugar that added `http://localhost/` to the beginning of the path if it was missing?
This would make `httpx` a dependency of core Datasette, which I think is OK.
It would also solve the return type problem: I would return whatever `httpx` returns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693007512,https://api.github.com/repos/simonw/datasette/issues/943,693007512,MDEyOklzc3VlQ29tbWVudDY5MzAwNzUxMg==,9599,simonw,2020-09-15T22:13:30Z,2020-09-15T22:13:30Z,OWNER,"I could solve streaming using something like this:
```python
async with datasette.stream(""GET"", ""/fixtures/compound_three_primary_keys.csv?_stream=on&_size=max"") as response:
async for chunk in response.aiter_bytes():
print(chunk)
```
Which would be a wrapper around `AsyncClient.stream(method, url, ...)` from https://www.python-httpx.org/async/#streaming-responses","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693008540,https://api.github.com/repos/simonw/datasette/issues/943,693008540,MDEyOklzc3VlQ29tbWVudDY5MzAwODU0MA==,9599,simonw,2020-09-15T22:16:07Z,2020-09-15T22:16:07Z,OWNER,"I think I can use `async with httpx.AsyncClient(base_url=""http://localhost/"") as client:` to ensure I don't need to use `http://localhost/` on every call.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693009048,https://api.github.com/repos/simonw/datasette/issues/943,693009048,MDEyOklzc3VlQ29tbWVudDY5MzAwOTA0OA==,9599,simonw,2020-09-15T22:17:30Z,2020-09-22T14:37:00Z,OWNER,"Maybe instead of implementing `datasette.get()` and `datasette.post()` and `datasette.request()` and `datasette.stream()` I could instead have a nested object called `datasette.client` which is a preconfigured `AsyncClient` instance.
```python
response = await datasette.client.get(""/"")
```
Or perhaps this should be a method in case I ever need to be able to `await` it:
```python
response = await (await datasette.client()).get(""/"")
```
This is a bit cosmetically ugly though, I'd rather avoid that if possible.
Maybe I could get this working by returning an object from `.client()` which provides a `await obj.get()` method:
```python
response = await datasette.client().get(""/"")
```
I don't think there's any benefit to that over `await datasette.client.get()` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693010291,https://api.github.com/repos/simonw/datasette/issues/943,693010291,MDEyOklzc3VlQ29tbWVudDY5MzAxMDI5MQ==,9599,simonw,2020-09-15T22:20:55Z,2020-09-15T22:20:55Z,OWNER,"Should I instantiate a single `Client` and reuse it for all internal requests, or can I instantiate a new `Client` for each request?
https://www.python-httpx.org/advanced/#why-use-a-client says that the main benefit of a Client instance is HTTP connection pooling - which isn't an issue for these internal requests since they won't be using the HTTP protocol at all, they'll be calling the ASGI application directly.
So I'm leaning towards instantiating a fresh client for every internal request. I'll run a microbenchmark to check that this doesn't have any unpleasant performance implications.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,