issue_comments

Table actions

6,420 rows where author_association = "OWNER" sorted by issue_url

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

author_association 1 ✖

  • OWNER · 6,420 ✖
id html_url issue_url ▼ node_id user created_at updated_at author_association body reactions issue performed_via_github_app
338523957 https://github.com/simonw/datasette/issues/1#issuecomment-338523957 https://api.github.com/repos/simonw/datasette/issues/1 MDEyOklzc3VlQ29tbWVudDMzODUyMzk1Nw== simonw 9599 2017-10-23T01:09:05Z 2017-10-24T02:42:12Z OWNER I also need to solve for weird primary keys. If it’s a single integer or a single char field that’s easy. But what if it is a compound key with more than one chat field? What delimiter can I use that will definitely be safe? Let’s say I use hyphen. Now I need to find a durable encoding for any hyphens that might exist in the key fields themselves. How about I use URLencoding for every non-alpha-numeric character? That will turn hyphens into (I think) %2D. It should also solve for unicode characters, but it means the vast majority of keys (integers) will display neatly, including a compound key of eg 5678-345 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Addressable pages for every row in a table 267513424  
338524454 https://github.com/simonw/datasette/issues/1#issuecomment-338524454 https://api.github.com/repos/simonw/datasette/issues/1 MDEyOklzc3VlQ29tbWVudDMzODUyNDQ1NA== simonw 9599 2017-10-23T01:15:24Z 2017-10-23T01:15:24Z OWNER Table rendering logic needs to detect the primary key field and turn it into a hyperlink. If there is a compound primary key it should add an extra column at the start of the table which displays the compound key as a link {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Addressable pages for every row in a table 267513424  
338857568 https://github.com/simonw/datasette/issues/1#issuecomment-338857568 https://api.github.com/repos/simonw/datasette/issues/1 MDEyOklzc3VlQ29tbWVudDMzODg1NzU2OA== simonw 9599 2017-10-24T02:57:12Z 2017-10-24T02:57:12Z OWNER I can find the primary keys using: PRAGMA table_info(myTable) {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Addressable pages for every row in a table 267513424  
338861511 https://github.com/simonw/datasette/issues/1#issuecomment-338861511 https://api.github.com/repos/simonw/datasette/issues/1 MDEyOklzc3VlQ29tbWVudDMzODg2MTUxMQ== simonw 9599 2017-10-24T03:24:17Z 2017-10-24T03:24:17Z OWNER Some tables won't have primary keys, in which case I won't generate pages for individual records. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Addressable pages for every row in a table 267513424  
338872286 https://github.com/simonw/datasette/issues/1#issuecomment-338872286 https://api.github.com/repos/simonw/datasette/issues/1 MDEyOklzc3VlQ29tbWVudDMzODg3MjI4Ng== simonw 9599 2017-10-24T04:46:06Z 2017-10-24T04:46:06Z OWNER I'm going to use `,` as the separator between elements of a compound primary key. If those elements themselves include a comma I will use `%2C` in its place. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Addressable pages for every row in a table 267513424  
338882207 https://github.com/simonw/datasette/issues/1#issuecomment-338882207 https://api.github.com/repos/simonw/datasette/issues/1 MDEyOklzc3VlQ29tbWVudDMzODg4MjIwNw== simonw 9599 2017-10-24T05:56:04Z 2017-10-24T05:56:04Z OWNER Next step: generate links to these. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Addressable pages for every row in a table 267513424  
341938424 https://github.com/simonw/datasette/issues/10#issuecomment-341938424 https://api.github.com/repos/simonw/datasette/issues/10 MDEyOklzc3VlQ29tbWVudDM0MTkzODQyNA== simonw 9599 2017-11-04T23:48:57Z 2017-11-04T23:48:57Z OWNER Done: https://github.com/simonw/stateless-datasets/commit/edaa10587e60946e0c1935333f6b79553db33798 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Set up Travis 267517381  
344771130 https://github.com/simonw/datasette/issues/100#issuecomment-344771130 https://api.github.com/repos/simonw/datasette/issues/100 MDEyOklzc3VlQ29tbWVudDM0NDc3MTEzMA== simonw 9599 2017-11-16T00:06:00Z 2017-11-16T00:06:00Z OWNER Aha... it looks like this is a Jinja version problem: https://github.com/ansible/ansible/issues/25381#issuecomment-306492389 Datasette depends on sanic-jinja2 - and that doesn't depend on a particular jinja2 version: https://github.com/lixxu/sanic-jinja2/blob/7e9520850d8c6bb66faf43b7f252593d7efe3452/setup.py#L22 So if you have an older version of Jinja installed, stuff breaks. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} TemplateAssertionError: no filter named 'tojson' 274160723  
705889120 https://github.com/simonw/datasette/pull/1000#issuecomment-705889120 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTg4OTEyMA== simonw 9599 2020-10-08T23:59:01Z 2020-10-08T23:59:01Z OWNER Needs tests and documentation. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705899629 https://github.com/simonw/datasette/pull/1000#issuecomment-705899629 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTg5OTYyOQ== simonw 9599 2020-10-09T00:37:02Z 2020-10-09T00:37:02Z OWNER I'm going to route the existing `TestClient` through this mechanism to exercise it during the tests. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705902902 https://github.com/simonw/datasette/pull/1000#issuecomment-705902902 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkwMjkwMg== simonw 9599 2020-10-09T00:50:49Z 2020-10-09T00:50:49Z OWNER Almost all of the tests are passing: ``` =========================== short test summary info ============================ FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200 FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200 FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert... FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5] FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyE... FAILED tests/test_plugins.py::test_hook_register_magic_parameters - Assertion... ============ 11 failed, 718 passed, 6 warnings in 225.77s (0:03:45) ============ ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705918844 https://github.com/simonw/datasette/pull/1000#issuecomment-705918844 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkxODg0NA== simonw 9599 2020-10-09T01:46:06Z 2020-10-09T01:46:06Z OWNER For this failing test I'm suspicious that the AsyncClient may be persisting cookies in between requests: ``` def test_actor_cookie(app_client): "A valid actor cookie sets request.scope['actor']" cookie = app_client.actor_cookie({"id": "test"}) response = app_client.get("/", cookies={"ds_actor": cookie}) > assert {"id": "test"} == app_client.ds._last_request.scope["actor"] E AssertionError: assert {'id': 'test'} == {'id': 'root'} E Differing items: E {'id': 'test'} != {'id': 'root'} E Use -v to get the full diff ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705920055 https://github.com/simonw/datasette/pull/1000#issuecomment-705920055 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkyMDA1NQ== simonw 9599 2020-10-09T01:51:05Z 2020-10-09T01:51:05Z OWNER The topic of disabling cookie persistence is discussed a little here: https://github.com/encode/httpx/issues/422#issuecomment-537906693 > We have a cookiejar abstraction, I think setting it to an always-empty jar like you describe is best. :) {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705920228 https://github.com/simonw/datasette/pull/1000#issuecomment-705920228 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkyMDIyOA== simonw 9599 2020-10-09T01:51:44Z 2020-10-09T01:51:44Z OWNER I'm going to switch back to having each request run through a new client. I'm worried about the impact on test performance though. I'll run a microbenchmark before and after. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705921006 https://github.com/simonw/datasette/pull/1000#issuecomment-705921006 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkyMTAwNg== simonw 9599 2020-10-09T01:55:01Z 2020-10-09T01:55:01Z OWNER With the single client that is reused for all tests: ``` % time pytest tests/test_api.py ... 6.73s user 9.91s system 81% cpu 20.365 total ``` After switching back to this class: ```python class DatasetteClient: def __init__(self, ds): self.app = ds.app() def _fix(self, path): if path.startswith("/"): path = "http://localhost{}".format(path) return path async def get(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.get(self._fix(path), **kwargs) async def options(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.options(self._fix(path), **kwargs) async def head(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.head(self._fix(path), **kwargs) async def post(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.post(self._fix(path), **kwargs) async def put(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.put(self._fix(path), **kwargs) async def patch(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.patch(self._fix(path), **kwargs) async def delete(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.delete(self._fix(path), **kwargs) async def request(self, method, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.request(method, self._fix(path), **kwargs) ``` The time taken is: ``` % time pytest tests/test_api.py ... 7.26s user 10.02s system 82% cpu 21.014 total ``` That's close enough that I don't feel I need to investigate this further. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705926035 https://github.com/simonw/datasette/pull/1000#issuecomment-705926035 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkyNjAzNQ== simonw 9599 2020-10-09T02:14:14Z 2020-10-09T02:14:14Z OWNER Still need to handle these six failing tests: ``` FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'elemen... FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', '... FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<... FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<p cla... FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages' FAILED tests/test_plugins.py::test_hook_register_magic_parameters - AssertionError: assert [{'line': '1.0', 'rowid': 1}] == [{'line': '1.1', 'rowid': 1}] =========================================================== 6 failed, 731 passed, 6 warnings in 129.17s (0:02:09) =========================================================== ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705926445 https://github.com/simonw/datasette/pull/1000#issuecomment-705926445 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkyNjQ0NQ== simonw 9599 2020-10-09T02:15:38Z 2020-10-09T02:15:38Z OWNER > FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages' That one is caused by `response.cookies` skipping cookies that were set to the empty string. Same fix as this: https://github.com/simonw/datasette/blob/a1687351fb75b01f737fda4ad07e0781029de05c/tests/test_auth.py#L90-L95 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705937696 https://github.com/simonw/datasette/pull/1000#issuecomment-705937696 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTkzNzY5Ng== simonw 9599 2020-10-09T02:52:53Z 2020-10-09T02:52:53Z OWNER These failures are giving me a severe "how did this ever work in the first place?" vibe: ``` FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] ``` I have a fix for them, no idea why they weren't already failing though. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705940507 https://github.com/simonw/datasette/pull/1000#issuecomment-705940507 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTk0MDUwNw== simonw 9599 2020-10-09T03:04:15Z 2020-10-09T03:04:15Z OWNER This is really weird: new set of test failures that I wasn't seeing before, and those tests aren't failing on my laptop: ``` =========================== short test summary info ============================ FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200 FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200 FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert... FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5] FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] ================== 5 failed, 738 passed in 194.73s (0:03:14) =================== ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705941580 https://github.com/simonw/datasette/pull/1000#issuecomment-705941580 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTk0MTU4MA== simonw 9599 2020-10-09T03:08:43Z 2020-10-09T03:08:43Z OWNER Most likely reason for those failures is that `path` and `raw_path` are not being simulated correctly. I used to do those here: https://github.com/simonw/datasette/blob/402cf870b7d65f9b5fba9e23aa99433294bd4523/datasette/utils/testing.py#L116-L125 But now I'm delegating that to `httpx` to handle. WEIRD that it passes on my laptop but fails in GitHub Actions CI though. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705945591 https://github.com/simonw/datasette/pull/1000#issuecomment-705945591 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTk0NTU5MQ== simonw 9599 2020-10-09T03:24:48Z 2020-10-09T03:24:48Z OWNER I'm testing this with a `print(scope)` and `pytest -k test_table_with_slashes_in_name -s`. Against the `main` branch: `{'type': 'http', 'http_version': '1.0', 'method': 'GET', 'path': '/fixtures/table/with/slashes.csv', 'raw_path': b'/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'headers': [[b'host', b'localhost']], 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x10e2e6040>}` Against the broken branch: `tests/test_api.py {'type': 'http', 'asgi': {'version': '3.0'}, 'http_version': '1.1', 'method': 'GET', 'headers': [(b'host', b'localhost'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'user-agent', b'python-httpx/0.15.0')], 'scheme': 'http', 'path': '/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'server': ('localhost', None), 'client': ('127.0.0.1', 123), 'root_path': '', 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x109e0eca0>}` This is on my laptop though so both of those pass the tests. Key difference: the `httpx` version doesn't set a `raw_path` at all. BUT.. it does set `path` and sets it to `'/fixtures/table%2Fwith%2Fslashes.csv'` The non-httpx version sets `raw_path` to `b'/fixtures/table%2Fwith%2Fslashes.csv'` and `path` to `'/fixtures/table/with/slashes.csv'`. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705946120 https://github.com/simonw/datasette/pull/1000#issuecomment-705946120 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTk0NjEyMA== simonw 9599 2020-10-09T03:27:05Z 2020-10-09T03:27:05Z OWNER I may need to fuss around with how the `httpx` client sends things to the ASGI app. https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L26 is relevant: Alternatively, you can setup the transport instance explicitly. This allows you to include any additional configuration arguments specific to the ASGITransport class: ``` transport = httpx.ASGITransport( app=app, root_path="/submount", client=("1.2.3.4", 123) ) client = httpx.AsyncClient(transport=transport) ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705946360 https://github.com/simonw/datasette/pull/1000#issuecomment-705946360 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNTk0NjM2MA== simonw 9599 2020-10-09T03:28:08Z 2020-10-09T03:28:08Z OWNER Here's where `httpx` sets up the ASGI scope: https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L82-L97 ```python # ASGI scope. scheme, host, port, full_path = url path, _, query = full_path.partition(b"?") scope = { "type": "http", "asgi": {"version": "3.0"}, "http_version": "1.1", "method": method.decode(), "headers": [(k.lower(), v) for (k, v) in headers], "scheme": scheme.decode("ascii"), "path": unquote(path.decode("ascii")), "query_string": query, "server": (host.decode("ascii"), port), "client": self.client, "root_path": self.root_path, } ``` Sure enough, it doesn't set the `raw_path`. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
706263157 https://github.com/simonw/datasette/pull/1000#issuecomment-706263157 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNjI2MzE1Nw== simonw 9599 2020-10-09T15:57:15Z 2020-10-09T15:57:15Z OWNER My `httpx` pull request adding `raw_path` support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet. I'm going to mark these tests as `xfail` so I can land this change - I'll remove that once an `httpx` release comes out that I can use to get the tests passing. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
706269271 https://github.com/simonw/datasette/pull/1000#issuecomment-706269271 https://api.github.com/repos/simonw/datasette/issues/1000 MDEyOklzc3VlQ29tbWVudDcwNjI2OTI3MQ== simonw 9599 2020-10-09T16:08:49Z 2020-10-09T16:08:49Z OWNER I'm going to document this in a separate issue. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette.client internal requests mechanism 717746043  
705904566 https://github.com/simonw/datasette/issues/1001#issuecomment-705904566 https://api.github.com/repos/simonw/datasette/issues/1001 MDEyOklzc3VlQ29tbWVudDcwNTkwNDU2Ng== simonw 9599 2020-10-09T00:58:08Z 2020-10-09T00:58:08Z OWNER To get a traceback: ``` datasette . -p 8009 --pdb ``` And then: ``` curl -XOPTIONS http://127.0.0.1:8009 ``` This causes the server to open a debugging prompt: ``` INFO: 127.0.0.1:59514 - "OPTIONS / HTTP/1.1" 500 Internal Server Error > /Users/simon/Dropbox/Development/datasette/datasette/views/base.py(115)dispatch_request() -> return await handler(request, *args, **kwargs) (Pdb) list 110 def database_color(self, database): 111 return "ff0000" 112 113 async def dispatch_request(self, request, *args, **kwargs): 114 handler = getattr(self, request.method.lower(), None) 115 -> return await handler(request, *args, **kwargs) 116 117 async def render(self, templates, request, context=None): 118 context = context or {} 119 template = self.ds.jinja_env.select_template(templates) 120 template_context = { (Pdb) ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} OPTIONS requests return a 500 error 717768441  
705904679 https://github.com/simonw/datasette/issues/1001#issuecomment-705904679 https://api.github.com/repos/simonw/datasette/issues/1001 MDEyOklzc3VlQ29tbWVudDcwNTkwNDY3OQ== simonw 9599 2020-10-09T00:58:32Z 2020-10-09T00:58:32Z OWNER So the bug is in this code here: https://github.com/simonw/datasette/blob/703439bdc37e724b01bc6d7a1fc1d955795132f2/datasette/views/base.py#L113-L115 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} OPTIONS requests return a 500 error 717768441  
705904759 https://github.com/simonw/datasette/issues/1001#issuecomment-705904759 https://api.github.com/repos/simonw/datasette/issues/1001 MDEyOklzc3VlQ29tbWVudDcwNTkwNDc1OQ== simonw 9599 2020-10-09T00:58:47Z 2020-10-09T00:58:47Z OWNER What should an OPTIONS request return, anyway? {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} OPTIONS requests return a 500 error 717768441  
705904917 https://github.com/simonw/datasette/issues/1001#issuecomment-705904917 https://api.github.com/repos/simonw/datasette/issues/1001 MDEyOklzc3VlQ29tbWVudDcwNTkwNDkxNw== simonw 9599 2020-10-09T00:59:25Z 2020-10-09T00:59:25Z OWNER ``` ~ % curl -XOPTIONS https://www.google.com/ <!DOCTYPE html> <html lang=en> <meta charset=utf-8> <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width"> <title>Error 405 (Method Not Allowed)!!1</title> ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} OPTIONS requests return a 500 error 717768441  
705905121 https://github.com/simonw/datasette/issues/1001#issuecomment-705905121 https://api.github.com/repos/simonw/datasette/issues/1001 MDEyOklzc3VlQ29tbWVudDcwNTkwNTEyMQ== simonw 9599 2020-10-09T01:00:07Z 2020-10-09T01:00:07Z OWNER The www.djangoproject.com site returns an empty page: ``` ~ % curl -vv -XOPTIONS https://www.djangoproject.com/ * Trying 151.101.42.217:443... * Connected to www.djangoproject.com (151.101.42.217) port 443 (#0) * ALPN, offering http/1.1 * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: osff2.map.fastly.net * Server certificate: GlobalSign CloudSSL CA - SHA256 - G3 * Server certificate: GlobalSign Root CA > OPTIONS / HTTP/1.1 > Host: www.djangoproject.com > User-Agent: curl/7.70.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Connection: keep-alive < Content-Length: 0 < Server: nginx < Content-Type: text/html; charset=utf-8 < Allow: GET, HEAD, OPTIONS < Content-Language: en < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < X-XSS-Protection: 1; mode=block < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload < Access-Control-Allow-Origin: https://code.djangoproject.com < Accept-Ranges: bytes < Date: Fri, 09 Oct 2020 00:59:42 GMT < Via: 1.1 varnish < X-Served-By: cache-sjc10047-SJC < X-Cache: MISS < X-Cache-Hits: 0 < X-Timer: S1602205182.833493,VS0,VE305 < Vary: Accept-Language, Accept-Encoding < * Connection #0 to host www.djangoproject.com left intact ~ % ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} OPTIONS requests return a 500 error 717768441  
705905418 https://github.com/simonw/datasette/issues/1001#issuecomment-705905418 https://api.github.com/repos/simonw/datasette/issues/1001 MDEyOklzc3VlQ29tbWVudDcwNTkwNTQxOA== simonw 9599 2020-10-09T01:00:54Z 2020-10-09T01:00:54Z OWNER https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/OPTIONS {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} OPTIONS requests return a 500 error 717768441  
705916614 https://github.com/simonw/datasette/issues/1001#issuecomment-705916614 https://api.github.com/repos/simonw/datasette/issues/1001 MDEyOklzc3VlQ29tbWVudDcwNTkxNjYxNA== simonw 9599 2020-10-09T01:37:06Z 2020-10-09T01:37:06Z OWNER I'm tempted to imitate Django Rest Framework here: https://github.com/encode/django-rest-framework/blob/2e721cdbc85a924d0b0f093b86fe1497b58fe287/rest_framework/views.py#L514-L521 ```python def options(self, request, *args, **kwargs): """ Handler method for HTTP 'OPTIONS' request. """ if self.metadata_class is None: return self.http_method_not_allowed(request, *args, **kwargs) data = self.metadata_class().determine_metadata(request, self) return Response(data, status=status.HTTP_200_OK) ``` That `determine_metadata()` default method does this: https://github.com/encode/django-rest-framework/blob/335054a5d36b352a58286b303b608b6bf48152f8/rest_framework/metadata.py#L29 Note the comment at the top: ``` There are not any formalized standards for `OPTIONS` responses for us to base this on. ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} OPTIONS requests return a 500 error 717768441  
705917015 https://github.com/simonw/datasette/issues/1001#issuecomment-705917015 https://api.github.com/repos/simonw/datasette/issues/1001 MDEyOklzc3VlQ29tbWVudDcwNTkxNzAxNQ== simonw 9599 2020-10-09T01:38:49Z 2020-10-09T01:38:49Z OWNER I actually have a sensible `OPTIONS` implementation here: https://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/datasette/views/base.py#L154-L165 I'm going to set the default one to return a 405 (Method Not Allowed) like Google does. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} OPTIONS requests return a 500 error 717768441  
706306214 https://github.com/simonw/datasette/issues/1002#issuecomment-706306214 https://api.github.com/repos/simonw/datasette/issues/1002 MDEyOklzc3VlQ29tbWVudDcwNjMwNjIxNA== simonw 9599 2020-10-09T17:23:51Z 2020-10-09T17:23:51Z OWNER I can start by combining the release notes for the 0.50 alphas. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Release notes for Datasette 0.50 717783692  
706272322 https://github.com/simonw/datasette/issues/1003#issuecomment-706272322 https://api.github.com/repos/simonw/datasette/issues/1003 MDEyOklzc3VlQ29tbWVudDcwNjI3MjMyMg== simonw 9599 2020-10-09T16:14:56Z 2020-10-09T16:14:56Z OWNER Yes I think that makes sense. I added `json` to the template context in Dogsheep Beta just a few days ago because I needed that: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L176 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} from_json jinja2 filter 718238967  
706273211 https://github.com/simonw/datasette/issues/1003#issuecomment-706273211 https://api.github.com/repos/simonw/datasette/issues/1003 MDEyOklzc3VlQ29tbWVudDcwNjI3MzIxMQ== simonw 9599 2020-10-09T16:16:38Z 2020-10-09T16:16:38Z OWNER I'm not a huge fan of `from_json` as the name for this. Some other options: - Expose `json` directly so templates can do `json.loads()` and `json.dumps()` - this allows for outputting JSON too, which is useful. But is there anything else on the `json` module that shouldn't be exposed in templates? - `json_dumps()` and `json_loads()` template functions. I quite like that. - Something else? {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} from_json jinja2 filter 718238967  
706281451 https://github.com/simonw/datasette/issues/1003#issuecomment-706281451 https://api.github.com/repos/simonw/datasette/issues/1003 MDEyOklzc3VlQ29tbWVudDcwNjI4MTQ1MQ== simonw 9599 2020-10-09T16:33:01Z 2020-10-09T16:33:01Z OWNER I think `json_dumps()` and `json_loads()` as aliases for `json.dumps()` and `json.loads()` is the way to go here. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} from_json jinja2 filter 718238967  
706473306 https://github.com/simonw/datasette/issues/1005#issuecomment-706473306 https://api.github.com/repos/simonw/datasette/issues/1005 MDEyOklzc3VlQ29tbWVudDcwNjQ3MzMwNg== simonw 9599 2020-10-10T02:45:46Z 2020-10-10T02:45:46Z OWNER https://github.com/simonw/datasette/blob/9f6dd985bc0eff70f8a9ce65c6578bc43d2e172b/tests/test_html.py#L145-L149 https://github.com/simonw/datasette/blob/9f6dd985bc0eff70f8a9ce65c6578bc43d2e172b/tests/test_html.py#L539-L564 https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/tests/test_api.py#L1190-L1194 https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/tests/test_api.py#L742-L746 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Remove xfail tests when new httpx is released 718259202  
747209115 https://github.com/simonw/datasette/issues/1005#issuecomment-747209115 https://api.github.com/repos/simonw/datasette/issues/1005 MDEyOklzc3VlQ29tbWVudDc0NzIwOTExNQ== simonw 9599 2020-12-17T05:11:04Z 2020-12-17T05:11:04Z OWNER Tracking ticket for the next HTTPX release is https://github.com/encode/httpx/pull/1403 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Remove xfail tests when new httpx is released 718259202  
787536267 https://github.com/simonw/datasette/issues/1005#issuecomment-787536267 https://api.github.com/repos/simonw/datasette/issues/1005 MDEyOklzc3VlQ29tbWVudDc4NzUzNjI2Nw== simonw 9599 2021-02-28T22:30:37Z 2021-02-28T22:30:37Z OWNER It's out! https://github.com/encode/httpx/releases/tag/0.17.0 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Remove xfail tests when new httpx is released 718259202  
706270877 https://github.com/simonw/datasette/issues/1006#issuecomment-706270877 https://api.github.com/repos/simonw/datasette/issues/1006 MDEyOklzc3VlQ29tbWVudDcwNjI3MDg3Nw== simonw 9599 2020-10-09T16:12:09Z 2020-10-09T16:12:09Z OWNER This can become a section on https://docs.datasette.io/en/stable/internals.html {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Documentation for datasette.client 718264811  
706305601 https://github.com/simonw/datasette/issues/1006#issuecomment-706305601 https://api.github.com/repos/simonw/datasette/issues/1006 MDEyOklzc3VlQ29tbWVudDcwNjMwNTYwMQ== simonw 9599 2020-10-09T17:22:31Z 2020-10-09T17:22:31Z OWNER https://docs.datasette.io/en/latest/internals.html#client {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Documentation for datasette.client 718264811  
706276831 https://github.com/simonw/datasette/issues/1007#issuecomment-706276831 https://api.github.com/repos/simonw/datasette/issues/1007 MDEyOklzc3VlQ29tbWVudDcwNjI3NjgzMQ== simonw 9599 2020-10-09T16:23:51Z 2020-10-09T16:23:51Z OWNER I don't appear to be using these anywhere, not sure why I spotted a warning (which I now can't find). {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} set-env and add-path commands have been deprecated 718272593  
706450418 https://github.com/simonw/datasette/issues/1010#issuecomment-706450418 https://api.github.com/repos/simonw/datasette/issues/1010 MDEyOklzc3VlQ29tbWVudDcwNjQ1MDQxOA== simonw 9599 2020-10-10T00:08:27Z 2020-10-10T00:09:13Z OWNER This commit broke it: https://github.com/simonw/datasette/commit/8f97b9b58e77f82fef1f10e9c9f6754b993544b6#diff-ed4bb3a97b7efa4d45be6a6ea6f4636cL114-R165 <img width="1106" alt="Banners_and_Alerts_and_datasette_client_internal_requests_mechanism_·_simonw_datasette_8f97b9b" src="https://user-images.githubusercontent.com/9599/95640559-25a37f00-0a52-11eb-838e-9f771ab5a413.png"> {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} json / CSV links are broken in Datasette 0.50 718484082  
706455202 https://github.com/simonw/datasette/issues/1010#issuecomment-706455202 https://api.github.com/repos/simonw/datasette/issues/1010 MDEyOklzc3VlQ29tbWVudDcwNjQ1NTIwMg== simonw 9599 2020-10-10T00:33:46Z 2020-10-10T00:33:46Z OWNER Since this is an urgent fix that needs to go out I've marked the `base_url` tests as `xfail`. I need to solve those separately, see #900 and various other bug reports. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} json / CSV links are broken in Datasette 0.50 718484082  
706455505 https://github.com/simonw/datasette/issues/1010#issuecomment-706455505 https://api.github.com/repos/simonw/datasette/issues/1010 MDEyOklzc3VlQ29tbWVudDcwNjQ1NTUwNQ== simonw 9599 2020-10-10T00:35:41Z 2020-10-10T00:35:41Z OWNER I missed the links on the query page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} json / CSV links are broken in Datasette 0.50 718484082  
706480763 https://github.com/simonw/datasette/issues/1011#issuecomment-706480763 https://api.github.com/repos/simonw/datasette/issues/1011 MDEyOklzc3VlQ29tbWVudDcwNjQ4MDc2Mw== simonw 9599 2020-10-10T03:42:58Z 2020-10-10T03:42:58Z OWNER Oh no! It's the same bug as #1010. Thanks for the report, I shall push out 0.50.2 as quickly as possible. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} column name links broken in 0.50.1 718521469  
706480791 https://github.com/simonw/datasette/issues/1011#issuecomment-706480791 https://api.github.com/repos/simonw/datasette/issues/1011 MDEyOklzc3VlQ29tbWVudDcwNjQ4MDc5MQ== simonw 9599 2020-10-10T03:43:13Z 2020-10-10T03:43:13Z OWNER Demo: click column headers on https://latest.datasette.io/fixtures/compound_three_primary_keys {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} column name links broken in 0.50.1 718521469  
706482542 https://github.com/simonw/datasette/issues/1011#issuecomment-706482542 https://api.github.com/repos/simonw/datasette/issues/1011 MDEyOklzc3VlQ29tbWVudDcwNjQ4MjU0Mg== simonw 9599 2020-10-10T03:58:58Z 2020-10-10T03:58:58Z OWNER The fix is live on the demo: https://latest.datasette.io/fixtures/compound_three_primary_keys?_sort_desc=pk2 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} column name links broken in 0.50.1 718521469  
706483680 https://github.com/simonw/datasette/issues/1011#issuecomment-706483680 https://api.github.com/repos/simonw/datasette/issues/1011 MDEyOklzc3VlQ29tbWVudDcwNjQ4MzY4MA== simonw 9599 2020-10-10T04:09:32Z 2020-10-10T04:09:32Z OWNER OK, Datasette 0.50.2 is live on PyPI now. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} column name links broken in 0.50.1 718521469  
969602825 https://github.com/simonw/datasette/issues/1012#issuecomment-969602825 https://api.github.com/repos/simonw/datasette/issues/1012 IC_kwDOBm6k_c45yvcJ simonw 9599 2021-11-16T01:21:14Z 2021-11-16T01:21:14Z OWNER I'd been wondering how to get new classifiers into Trove - thanks, I'll give this a go. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} For 1.0 update trove classifier in setup.py 718540751  
969613166 https://github.com/simonw/datasette/issues/1012#issuecomment-969613166 https://api.github.com/repos/simonw/datasette/issues/1012 IC_kwDOBm6k_c45yx9u simonw 9599 2021-11-16T01:27:25Z 2021-11-16T01:27:25Z OWNER Requested here: - https://github.com/pypa/trove-classifiers/pull/85 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} For 1.0 update trove classifier in setup.py 718540751  
706626796 https://github.com/simonw/datasette/issues/1014#issuecomment-706626796 https://api.github.com/repos/simonw/datasette/issues/1014 MDEyOklzc3VlQ29tbWVudDcwNjYyNjc5Ng== simonw 9599 2020-10-10T23:47:23Z 2020-10-10T23:48:26Z OWNER The GitHub API has these: `link: <https://api.github.com/repositories/107914493/commits?page=2>; rel="next", <https://api.github.com/repositories/107914493/commits?page=46>; rel="last"` WordPress API: https://css-tricks.com/wp-json/wp/v2/posts `link: <https://css-tricks.com/wp-json/wp/v2/posts?page=2>; rel="next"` It also has these: ``` x-wp-total: 5162 x-wp-totalpages: 517 ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Add Link: pagination HTTP headers 718723543  
706626913 https://github.com/simonw/datasette/issues/1014#issuecomment-706626913 https://api.github.com/repos/simonw/datasette/issues/1014 MDEyOklzc3VlQ29tbWVudDcwNjYyNjkxMw== simonw 9599 2020-10-10T23:49:08Z 2020-10-10T23:49:08Z OWNER This should be documented on https://docs.datasette.io/en/stable/json_api.html {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Add Link: pagination HTTP headers 718723543  
706626934 https://github.com/simonw/datasette/issues/1014#issuecomment-706626934 https://api.github.com/repos/simonw/datasette/issues/1014 MDEyOklzc3VlQ29tbWVudDcwNjYyNjkzNA== simonw 9599 2020-10-10T23:49:25Z 2020-10-10T23:49:25Z OWNER I'm just going to implement the 'next' one. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Add Link: pagination HTTP headers 718723543  
706626971 https://github.com/simonw/datasette/issues/1014#issuecomment-706626971 https://api.github.com/repos/simonw/datasette/issues/1014 MDEyOklzc3VlQ29tbWVudDcwNjYyNjk3MQ== simonw 9599 2020-10-10T23:49:57Z 2020-10-10T23:49:57Z OWNER I won't offer a total-number header. That's expensive to calculate - no need to calculate it unless it's explicitly asked for. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Add Link: pagination HTTP headers 718723543  
706627035 https://github.com/simonw/datasette/issues/1014#issuecomment-706627035 https://api.github.com/repos/simonw/datasette/issues/1014 MDEyOklzc3VlQ29tbWVudDcwNjYyNzAzNQ== simonw 9599 2020-10-10T23:50:40Z 2020-10-10T23:50:40Z OWNER First example page to add this to: `?_shape=array`, e.g. https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Add Link: pagination HTTP headers 718723543  
706631006 https://github.com/simonw/datasette/issues/1014#issuecomment-706631006 https://api.github.com/repos/simonw/datasette/issues/1014 MDEyOklzc3VlQ29tbWVudDcwNjYzMTAwNg== simonw 9599 2020-10-11T00:36:43Z 2020-10-11T00:36:43Z OWNER Demo using [paginate-json](https://github.com/simonw/paginate-json): ``` % paginate-json 'https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array' | jq '. | length' https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cd%2Cv http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ch%2Cr http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cl%2Cn http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cp%2Cj http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ct%2Cf http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cx%2Cb http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ca%2Cx http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ce%2Ct http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ci%2Cp http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Cm%2Cl 1001 ``` New documentation: https://docs.datasette.io/en/latest/json_api.html#pagination {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Add Link: pagination HTTP headers 718723543  
706756879 https://github.com/simonw/datasette/issues/1015#issuecomment-706756879 https://api.github.com/repos/simonw/datasette/issues/1015 MDEyOklzc3VlQ29tbWVudDcwNjc1Njg3OQ== simonw 9599 2020-10-11T19:35:03Z 2020-10-11T19:35:03Z OWNER Since plugins are installed via pip this would require Datasette to be restarted. This StackOverflow thread looks relevant to that: https://stackoverflow.com/questions/11329917/restart-python-script-from-within-itself This recipe looks promising: ```python import os import sys import psutil import logging def restart_program(): """Restarts the current program, with file objects and descriptors cleanup """ try: p = psutil.Process(os.getpid()) for handler in p.get_open_files() + p.connections(): os.close(handler.fd) except Exception, e: logging.error(e) python = sys.executable os.execl(python, python, *sys.argv) ``` https://docs.python.org/3/library/os.html#os.execl says about `os.execl`: > These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Research: could Datasette install its own plugins? 718910318  
706788010 https://github.com/simonw/datasette/issues/1016#issuecomment-706788010 https://api.github.com/repos/simonw/datasette/issues/1016 MDEyOklzc3VlQ29tbWVudDcwNjc4ODAxMA== simonw 9599 2020-10-11T23:50:39Z 2020-10-11T23:50:39Z OWNER For consistency can reuse the icon used on selected facets: <img width="721" alt="evernote__notes__8_rows_where_where_application-data____ACD4B543-4663-40B2-BAF5-666FA54D4204__and_source-url_is_not_blank_sorted_by_id" src="https://user-images.githubusercontent.com/9599/95693564-e0ae5280-0be1-11eb-881b-fa0cec8397f5.png"> {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Add a "delete" icon next to filters (in addition to "remove filter") 718953669  
706821561 https://github.com/simonw/datasette/issues/1016#issuecomment-706821561 https://api.github.com/repos/simonw/datasette/issues/1016 MDEyOklzc3VlQ29tbWVudDcwNjgyMTU2MQ== simonw 9599 2020-10-12T02:33:52Z 2020-10-12T02:33:52Z OWNER Should this do the equivalent of clicking Apply for you? I don't think so - that wouldn't fit with how the rest of that block of controls works. But if it makes the row vanish it may not be clear to the user that they still need to click Apply. Easiest thing would be for it to select the "remove filter" column and blank out the boxes. I will try that first. The code can go in `table.js`. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Add a "delete" icon next to filters (in addition to "remove filter") 718953669  
706828375 https://github.com/simonw/datasette/issues/1016#issuecomment-706828375 https://api.github.com/repos/simonw/datasette/issues/1016 MDEyOklzc3VlQ29tbWVudDcwNjgyODM3NQ== simonw 9599 2020-10-12T03:01:58Z 2020-10-12T03:01:58Z OWNER Demo: https://latest.datasette.io/fixtures/facetable?_facet=created&created=2019-01-17+08%3A00%3A00&_facet=state&state=MI ![x-button](https://user-images.githubusercontent.com/9599/95700744-9b4b4e80-0bfc-11eb-85a3-6da82d9e0cbf.gif) {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Add a "delete" icon next to filters (in addition to "remove filter") 718953669  
708113989 https://github.com/simonw/datasette/issues/1019#issuecomment-708113989 https://api.github.com/repos/simonw/datasette/issues/1019 MDEyOklzc3VlQ29tbWVudDcwODExMzk4OQ== simonw 9599 2020-10-14T02:22:40Z 2020-10-14T02:22:40Z OWNER I've found myself wanting this too. It should respect the allow-sql permission so it doesn't display if the user isn't able to run custom SQL. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} "Edit SQL" button on canned queries 721050815  
708114636 https://github.com/simonw/datasette/issues/1019#issuecomment-708114636 https://api.github.com/repos/simonw/datasette/issues/1019 MDEyOklzc3VlQ29tbWVudDcwODExNDYzNg== simonw 9599 2020-10-14T02:24:56Z 2020-10-14T02:24:56Z OWNER So it would appear on this page: https://latest.datasette.io/fixtures/neighborhood_search#fragment-goes-here Probably as a blue "Edit SQL" link next to those buttons. ![575FB160-4F75-43B2-B2AB-D75E60F10477](https://user-images.githubusercontent.com/9599/95936143-c2835680-0d89-11eb-9cf3-06f86d923103.jpeg) {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} "Edit SQL" button on canned queries 721050815  
708127294 https://github.com/simonw/datasette/issues/1019#issuecomment-708127294 https://api.github.com/repos/simonw/datasette/issues/1019 MDEyOklzc3VlQ29tbWVudDcwODEyNzI5NA== simonw 9599 2020-10-14T03:08:13Z 2020-10-14T03:08:13Z OWNER Maybe like this: <img width="1385" alt="fixtures__select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________order_by_neighborhood_" src="https://user-images.githubusercontent.com/9599/95938930-ce721700-0d8f-11eb-87c8-f67ea4e57d55.png"> {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} "Edit SQL" button on canned queries 721050815  
708128286 https://github.com/simonw/datasette/issues/1019#issuecomment-708128286 https://api.github.com/repos/simonw/datasette/issues/1019 MDEyOklzc3VlQ29tbWVudDcwODEyODI4Ng== simonw 9599 2020-10-14T03:11:33Z 2020-10-14T03:14:07Z OWNER Another edge-case: https://latest.datasette.io/fixtures/pragma_cache_size (`PRAGMA cache_size;`) isn't an allowed query usually, so linking to "Edit SQL" for it would link to an error page: https://latest.datasette.io/fixtures?sql=PRAGMA+cache_size%3B Can use `datasette.utils.validate_sql_select(sql)` to check for that - it raises `datasette.utils.InvalidSql` if there's a problem. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} "Edit SQL" button on canned queries 721050815  
708130290 https://github.com/simonw/datasette/issues/1019#issuecomment-708130290 https://api.github.com/repos/simonw/datasette/issues/1019 MDEyOklzc3VlQ29tbWVudDcwODEzMDI5MA== simonw 9599 2020-10-14T03:17:57Z 2020-10-14T03:17:57Z OWNER One last edge-case (I think) - magic parameters aren't supported for non-canned-queries, so if a query includes those it shouldn't show an Edit SQL link: https://latest.datasette.io/fixtures/magic_parameters I don't have a clean way of detecting if a query contains defined magic parameters, but I can instead do a dumb substring match for `:_` and skip showing the Edit SQL link if that is present. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} "Edit SQL" button on canned queries 721050815  
708139822 https://github.com/simonw/datasette/issues/1019#issuecomment-708139822 https://api.github.com/repos/simonw/datasette/issues/1019 MDEyOklzc3VlQ29tbWVudDcwODEzOTgyMg== simonw 9599 2020-10-14T03:53:21Z 2020-10-14T03:53:21Z OWNER Demos: - https://latest.datasette.io/fixtures/neighborhood_search has the link - https://latest.datasette.io/fixtures/neighborhood_search?text=ber has the link, and the link passes through the parameter that has already been entered - https://latest.datasette.io/fixtures/%F0%9D%90%9C%F0%9D%90%A2%F0%9D%90%AD%F0%9D%90%A2%F0%9D%90%9E%F0%9D%90%AC has the link - https://latest.datasette.io/fixtures/pragma_cache_size does not have the link (`PRAGMA` is not allowed in regular editable queries) - https://latest.datasette.io/fixtures/magic_parameters does not have the link (magic parameters) {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} "Edit SQL" button on canned queries 721050815  
754192267 https://github.com/simonw/datasette/issues/102#issuecomment-754192267 https://api.github.com/repos/simonw/datasette/issues/102 MDEyOklzc3VlQ29tbWVudDc1NDE5MjI2Nw== simonw 9599 2021-01-04T20:13:19Z 2021-01-04T20:13:19Z OWNER I'm more likely to do Lambda than Elastic Beanstalk, especially now the size limit for Lambdas has been increased as part of their support for Docker. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} datasette publish elasticbeanstalk 274264175  
708669178 https://github.com/simonw/datasette/issues/1020#issuecomment-708669178 https://api.github.com/repos/simonw/datasette/issues/1020 MDEyOklzc3VlQ29tbWVudDcwODY2OTE3OA== simonw 9599 2020-10-14T21:26:37Z 2020-10-14T21:26:37Z OWNER One option: add an optional request=... parameter which can be passed the current request, and will use that to populate the mock request with the exception of the bits that are passed explicitly (like the path): ```python response = await datasette.client.get("/db/table.json", request=request) ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Method for datasette.client() to forward on authentication 721068929  
708669778 https://github.com/simonw/datasette/issues/1020#issuecomment-708669778 https://api.github.com/repos/simonw/datasette/issues/1020 MDEyOklzc3VlQ29tbWVudDcwODY2OTc3OA== simonw 9599 2020-10-14T21:27:58Z 2020-10-14T21:27:58Z OWNER Maybe these internal requests should have some kind of flag that lets the underlying code tell that it's being called internally. One option: add a `x-internal: 1` request header - and ensure that any requests from outside Datasette have that header stripped. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Method for datasette.client() to forward on authentication 721068929  
708670392 https://github.com/simonw/datasette/issues/1020#issuecomment-708670392 https://api.github.com/repos/simonw/datasette/issues/1020 MDEyOklzc3VlQ29tbWVudDcwODY3MDM5Mg== simonw 9599 2020-10-14T21:29:22Z 2020-10-14T21:29:22Z OWNER I should also verify (and probably unit-test) that things like the `?_trace=1` mechanism work across the internal request boundary. `/-/permissions` appears to work across this boundary, but again a test would be useful confirmation. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Method for datasette.client() to forward on authentication 721068929  
712481568 https://github.com/simonw/datasette/issues/1020#issuecomment-712481568 https://api.github.com/repos/simonw/datasette/issues/1020 MDEyOklzc3VlQ29tbWVudDcxMjQ4MTU2OA== simonw 9599 2020-10-19T22:41:59Z 2020-10-19T22:41:59Z OWNER It turns out this works just fine: ```python response = await datasette.client.get(path, cookies=request.cookies) ``` So I don't need a mechanism for this. I'm going to add this to the documentation instead. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Method for datasette.client() to forward on authentication 721068929  
712482015 https://github.com/simonw/datasette/issues/1020#issuecomment-712482015 https://api.github.com/repos/simonw/datasette/issues/1020 MDEyOklzc3VlQ29tbWVudDcxMjQ4MjAxNQ== simonw 9599 2020-10-19T22:43:24Z 2020-10-19T22:43:24Z OWNER ... unless I want to support authentication mechanisms that work based on incoming IP address instead, in which case there's an argument for copying more over from the incoming request. Tailscale is a good example of a system where authentication based on IP address can actually work well, so this is worth doing. Also, there might be authentication mechanisms which work by setting a custom header on the incoming request (not to mention the `Authorization` header). {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Method for datasette.client() to forward on authentication 721068929  
712482504 https://github.com/simonw/datasette/issues/1020#issuecomment-712482504 https://api.github.com/repos/simonw/datasette/issues/1020 MDEyOklzc3VlQ29tbWVudDcxMjQ4MjUwNA== simonw 9599 2020-10-19T22:45:01Z 2020-10-19T22:45:01Z OWNER I'm having trouble coming up with the syntax for this. Here's one option: ```python response = await datasette.client.get(path, request=request) ``` But this feels confusing to me. We're not using the `request=` argument as a request - we're using it as a source of some default request values (the cookies and incoming headers, but not the path). We're essentially combining that request with the other arguments passed to `.get()`. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Method for datasette.client() to forward on authentication 721068929  
708718475 https://github.com/simonw/datasette/pull/1022#issuecomment-708718475 https://api.github.com/repos/simonw/datasette/issues/1022 MDEyOklzc3VlQ29tbWVudDcwODcxODQ3NQ== simonw 9599 2020-10-14T23:46:38Z 2020-10-14T23:46:38Z OWNER Good catch, thanks! {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Fix table name in spatialite example command 721830990  
712604541 https://github.com/simonw/datasette/issues/1023#issuecomment-712604541 https://api.github.com/repos/simonw/datasette/issues/1023 MDEyOklzc3VlQ29tbWVudDcxMjYwNDU0MQ== simonw 9599 2020-10-20T05:39:44Z 2020-10-20T05:39:44Z OWNER Here's the alpha with most of this work ready for people to preview: https://github.com/simonw/datasette/releases/tag/0.51a0 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Fix issues relating to base_url 722673818  
712607608 https://github.com/simonw/datasette/issues/1023#issuecomment-712607608 https://api.github.com/repos/simonw/datasette/issues/1023 MDEyOklzc3VlQ29tbWVudDcxMjYwNzYwOA== simonw 9599 2020-10-20T05:47:42Z 2020-10-20T05:47:42Z OWNER Requested alpha testers in https://github.com/simonw/datasette/issues/838#issuecomment-712604364 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Fix issues relating to base_url 722673818  
719986922 https://github.com/simonw/datasette/issues/1023#issuecomment-719986922 https://api.github.com/repos/simonw/datasette/issues/1023 MDEyOklzc3VlQ29tbWVudDcxOTk4NjkyMg== simonw 9599 2020-10-31T20:51:01Z 2020-10-31T20:51:01Z OWNER This should all be working correctly now. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Fix issues relating to base_url 722673818  
709589297 https://github.com/simonw/datasette/issues/1024#issuecomment-709589297 https://api.github.com/repos/simonw/datasette/issues/1024 MDEyOklzc3VlQ29tbWVudDcwOTU4OTI5Nw== simonw 9599 2020-10-15T21:04:31Z 2020-10-15T21:04:31Z OWNER I think nginx or Apache would be the best tools for this. I'm inclined to try with nginx first since I know it better. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Figure out how to run an environment that exercises the base_url proxy setting 722674708  
709590337 https://github.com/simonw/datasette/issues/1024#issuecomment-709590337 https://api.github.com/repos/simonw/datasette/issues/1024 MDEyOklzc3VlQ29tbWVudDcwOTU5MDMzNw== simonw 9599 2020-10-15T21:06:24Z 2020-10-15T21:07:19Z OWNER From https://stackoverflow.com/questions/32549684/nginx-proxy-and-remove-proxy-pass-prefix/32550251 it looks like the config I should use is: ``` server { listen 80; server_name example.com; location /datasette/ { proxy_pass http://127.0.0.1:8001; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 90; } } ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Figure out how to run an environment that exercises the base_url proxy setting 722674708  
709590941 https://github.com/simonw/datasette/issues/1024#issuecomment-709590941 https://api.github.com/repos/simonw/datasette/issues/1024 MDEyOklzc3VlQ29tbWVudDcwOTU5MDk0MQ== simonw 9599 2020-10-15T21:07:47Z 2020-10-15T21:07:47Z OWNER On macOS I ran `brew install nginx`. I'm going to try running it on port 8000 so I don't have to run it as root. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Figure out how to run an environment that exercises the base_url proxy setting 722674708  
709595960 https://github.com/simonw/datasette/issues/1024#issuecomment-709595960 https://api.github.com/repos/simonw/datasette/issues/1024 MDEyOklzc3VlQ29tbWVudDcwOTU5NTk2MA== simonw 9599 2020-10-15T21:18:14Z 2020-10-15T21:18:14Z OWNER Typing `nginx` starts it running as a daemon listening on port `http-alt` aka 8080. It uses the config file from ` /usr/local/etc/nginx/nginx.conf`. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Figure out how to run an environment that exercises the base_url proxy setting 722674708  
709597589 https://github.com/simonw/datasette/issues/1024#issuecomment-709597589 https://api.github.com/repos/simonw/datasette/issues/1024 MDEyOklzc3VlQ29tbWVudDcwOTU5NzU4OQ== simonw 9599 2020-10-15T21:21:53Z 2020-10-15T21:23:25Z OWNER Here's a recipe for running nginx against a custom config file: https://gist.github.com/simonw/35f0ebf9c1d6df158759 ``` daemon off; events { worker_connections 1024; } http { access_log /dev/stdout; error_log /dev/stderr; types { text/html html htm shtml; text/css css; image/gif gif; image/jpeg jpeg jpg; application/javascript js; } server { listen 8002; index index.html; root app; } } ``` ``` nginx -p `pwd` -c `pwd`/nginx.conf ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Figure out how to run an environment that exercises the base_url proxy setting 722674708  
709598324 https://github.com/simonw/datasette/issues/1024#issuecomment-709598324 https://api.github.com/repos/simonw/datasette/issues/1024 MDEyOklzc3VlQ29tbWVudDcwOTU5ODMyNA== simonw 9599 2020-10-15T21:23:33Z 2020-10-15T21:26:55Z OWNER Combining these two examples, here's the config file I am going to use for this. I'll save this as `nginx.conf`: ``` daemon off; events { worker_connections 1024; } http { server { listen 8000; location /datasette { proxy_pass http://127.0.0.1:8001; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } } ``` Then start the server with: ``` nginx -p `pwd` -c `pwd`/nginx.conf ``` And start Datasette like this: ``` datasette fixtures.db --config base_url:/datasette/ ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Figure out how to run an environment that exercises the base_url proxy setting 722674708  
709600335 https://github.com/simonw/datasette/issues/1024#issuecomment-709600335 https://api.github.com/repos/simonw/datasette/issues/1024 MDEyOklzc3VlQ29tbWVudDcwOTYwMDMzNQ== simonw 9599 2020-10-15T21:28:02Z 2020-10-15T22:25:43Z OWNER This is working OK so far: <img width="897" alt="fixtures__compound_three_primary_keys__1_001_rows_where_sorted_by_pk2" src="https://user-images.githubusercontent.com/9599/96187878-96371980-0ef2-11eb-875e-5b1ae9b445fd.png"> I'll try crawling it with `wget -r` to see if I get any errors. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Figure out how to run an environment that exercises the base_url proxy setting 722674708  
709622973 https://github.com/simonw/datasette/issues/1024#issuecomment-709622973 https://api.github.com/repos/simonw/datasette/issues/1024 MDEyOklzc3VlQ29tbWVudDcwOTYyMjk3Mw== simonw 9599 2020-10-15T22:27:31Z 2020-10-15T22:27:31Z OWNER Here's how I tested it: ``` time wget -r 'http://localhost:8000/datasette/' 2>&1 | grep -i -C 5 "failed\|error" > /tmp/errors.txt ``` This wrote out any errors (plus context) to the `errors.txt` log - and reported that the full crawl took 33s. Here's what I got in `errors.txt`: ``` 0K . 71.6M=0s 2020-10-15 15:23:09 (71.6 MB/s) - ‘localhost:8000/datasette/index.html’ saved [1276] Loading robots.txt; please ignore errors. --2020-10-15 15:23:09-- http://localhost:8000/robots.txt Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 404 Not Found -- --2020-10-15 15:23:09-- http://localhost:8000/robots.txt Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 404 Not Found 2020-10-15 15:23:09 ERROR 404: Not Found. --2020-10-15 15:23:09-- http://localhost:8000/datasette/-/static/app.css?b576be Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 200 OK Length: 8563 (8.4K) [text/css] -- -- 2020-10-15 15:23:13 (7.90 MB/s) - ‘localhost:8000/datasette/fixtures/primary_key_multiple_columns_explicit_label.json?_shape=object’ saved [58] --2020-10-15 15:23:13-- http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 404 Not Found 2020-10-15 15:23:13 ERROR 404: Not Found. --2020-10-15 15:23:13-- http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+limit+101 Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] -- -- 2020-10-15 15:23:13 (84.3 MB/s) - ‘localhost:8000/datasette/fixtures/roadside_attractions.json?_shape=object’ saved [619] --2020-10-15 15:23:13-- http://localhost:8000/datasette/fixtures/%5C%22https://www.openstreetmap.org/copyright%5C%… {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Figure out how to run an environment that exercises the base_url proxy setting 722674708  
709625063 https://github.com/simonw/datasette/issues/1024#issuecomment-709625063 https://api.github.com/repos/simonw/datasette/issues/1024 MDEyOklzc3VlQ29tbWVudDcwOTYyNTA2Mw== simonw 9599 2020-10-15T22:33:22Z 2020-10-15T22:33:22Z OWNER Of those errors... `http://localhost:8000/robots.txt` 404 is fine. `http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22` looks to me like a `wget` parsing bug where it got confused by this JavaScript: ``` window.DATASETTE_CLUSTER_MAP_TILE_LAYER_OPTIONS = {"maxZoom": 19, "detectRetina": true, "attribution": "&copy; <a href=\"https://www.openstreetmap.org/copyright\">OpenStreetMap</a> contributors"};</script> ``` `http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js` is a real bug. It's a bug in `datasette-cluster-map` but also requires me to solve #988 - mechanism for plugins to construct URLs that obey `base_url`. I'm not sure why I'm getting a hit to `http://localhost:8000/` since I wouldn't expect to link to `/` anywhere. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Figure out how to run an environment that exercises the base_url proxy setting 722674708  
709629920 https://github.com/simonw/datasette/issues/1025#issuecomment-709629920 https://api.github.com/repos/simonw/datasette/issues/1025 MDEyOklzc3VlQ29tbWVudDcwOTYyOTkyMA== simonw 9599 2020-10-15T22:48:20Z 2020-10-15T22:48:20Z OWNER Also these: ``` datasette % git grep '"/' -- '*.html' ':(exclude)*/patterns.html' datasette/templates/allow_debug.html:<form action="/-/allow-debug" method="get"> datasette/templates/base.html: <form action="/-/logout" method="post"> datasette/templates/error.html: <a href="/">home</a> datasette/templates/logout.html:<form action="/-/logout" method="post"> datasette/templates/messages_debug.html:<form action="/-/messages" method="post"> datasette/templates/query.html: <a href="/">home</a> / ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Fix last remaining links to "/" that do not respect base_url 722724086  
709632136 https://github.com/simonw/datasette/issues/1025#issuecomment-709632136 https://api.github.com/repos/simonw/datasette/issues/1025 MDEyOklzc3VlQ29tbWVudDcwOTYzMjEzNg== simonw 9599 2020-10-15T22:55:44Z 2020-10-15T22:55:44Z OWNER It looks like there are also some generated redirect responses that don't take `base_url` into account: ``` datasette % git grep '"/' -- '*.py' ':(exclude)*test_*.py' ':(exclude)datasette/app.py' datasette/_version.py: for i in cfg.versionfile_source.split("/"): datasette/utils/asgi.py: path="/", datasette/views/base.py: should_redirect = "/{}-{}".format(name, expected) datasette/views/base.py: should_redirect += "/" + urllib.parse.quote_plus(kwargs["table"]) datasette/views/base.py: should_redirect += "/" + kwargs["pk_path"] datasette/views/special.py: response = Response.redirect("/") datasette/views/special.py: return Response.redirect("/") datasette/views/special.py: response = Response.redirect("/") datasette/views/special.py: return Response.redirect("/") ``` {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Fix last remaining links to "/" that do not respect base_url 722724086  
709632314 https://github.com/simonw/datasette/issues/1025#issuecomment-709632314 https://api.github.com/repos/simonw/datasette/issues/1025 MDEyOklzc3VlQ29tbWVudDcwOTYzMjMxNA== simonw 9599 2020-10-15T22:56:25Z 2020-10-15T22:56:34Z OWNER That `utils/asgi.py` line is the default path for setting cookies. That should likely take `base_url` into account too: https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/utils/asgi.py#L331-L342 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Fix last remaining links to "/" that do not respect base_url 722724086  
712481127 https://github.com/simonw/datasette/issues/1025#issuecomment-712481127 https://api.github.com/repos/simonw/datasette/issues/1025 MDEyOklzc3VlQ29tbWVudDcxMjQ4MTEyNw== simonw 9599 2020-10-19T22:40:37Z 2020-10-20T01:21:36Z OWNER Was blocked on #904 - now unblocked. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Fix last remaining links to "/" that do not respect base_url 722724086  
712525557 https://github.com/simonw/datasette/issues/1025#issuecomment-712525557 https://api.github.com/repos/simonw/datasette/issues/1025 MDEyOklzc3VlQ29tbWVudDcxMjUyNTU1Nw== simonw 9599 2020-10-20T01:07:02Z 2020-10-20T01:07:02Z OWNER I fixed the `queries.html` one. I'm not going to fix these two: ``` datasette/templates/error.html: <a href="/">home</a> datasette/templates/patterns.html: <a href="/">home</a> / ``` Because the `error.html` one does not get passed a context (which makes sense since an error has occurred) and the pattern portfolio doesn't need to link to anywhere in particular. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Fix last remaining links to "/" that do not respect base_url 722724086  
712579674 https://github.com/simonw/datasette/issues/1025#issuecomment-712579674 https://api.github.com/repos/simonw/datasette/issues/1025 MDEyOklzc3VlQ29tbWVudDcxMjU3OTY3NA== simonw 9599 2020-10-20T04:24:10Z 2020-10-20T04:24:10Z OWNER Changed my mind, `error.html` needs access to `urls` in order to link to its CSS file. Passing it after all (it already got passed `ds.config("base_url")` so `ds` was available previously). {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Fix last remaining links to "/" that do not respect base_url 722724086  
712593790 https://github.com/simonw/datasette/issues/1025#issuecomment-712593790 https://api.github.com/repos/simonw/datasette/issues/1025 MDEyOklzc3VlQ29tbWVudDcxMjU5Mzc5MA== simonw 9599 2020-10-20T05:12:36Z 2020-10-20T05:12:36Z OWNER I'm going to leave the cookies code setting cookies to default to the `"/"` top level. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} Fix last remaining links to "/" that do not respect base_url 722724086  
709636372 https://github.com/simonw/datasette/issues/1026#issuecomment-709636372 https://api.github.com/repos/simonw/datasette/issues/1026 MDEyOklzc3VlQ29tbWVudDcwOTYzNjM3Mg== simonw 9599 2020-10-15T23:09:34Z 2020-10-15T23:09:34Z OWNER I'm inclined to say that internal requests should ignore `base_url` - since that seems like the right thing for plugins that need to access default Datasette APIs. The one catch here is plugins that might want to proxy the current incoming URL for some reason - where that incoming `request.path` could include the `base_url`. Actually those should be fine - because it will have been stripped off earlier: https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/app.py#L963-L968 {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} How should datasette.client interact with base_url 722738988  
712607227 https://github.com/simonw/datasette/issues/1026#issuecomment-712607227 https://api.github.com/repos/simonw/datasette/issues/1026 MDEyOklzc3VlQ29tbWVudDcxMjYwNzIyNw== simonw 9599 2020-10-20T05:46:44Z 2020-10-20T05:46:44Z OWNER We have a solution for this now: `datasette.urls` from #1033 can be used by plugins to assemble the correct URLs to pass to `.get()` and friends. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} How should datasette.client interact with base_url 722738988  
712959034 https://github.com/simonw/datasette/issues/1026#issuecomment-712959034 https://api.github.com/repos/simonw/datasette/issues/1026 MDEyOklzc3VlQ29tbWVudDcxMjk1OTAzNA== simonw 9599 2020-10-20T16:03:33Z 2020-10-20T16:03:55Z OWNER Reconsidering this: I think the `.get()` etc methods should automatically add the `base_url` prefix for you, since these APIs are only intended to make internal calls. The clincher on this is when I went to add a section to the `datasette.client` documentation recommending you use `datasette.urls.path()` for every call to them that you make. But there's a problem: to handle table name escaping users are likely to want to use `datasette.urls.table()` anyway, like this: response = await datasette.client.get(datasette.urls.table("db", "table") + ".json") This risks adding the `base_url` prefix twice. Maybe the `.table()` method could return a string-like object that is marked as already having the `base_url` prefix added, so the `client.get()` method knows not to add it again. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} How should datasette.client interact with base_url 722738988  
712962517 https://github.com/simonw/datasette/issues/1026#issuecomment-712962517 https://api.github.com/repos/simonw/datasette/issues/1026 MDEyOklzc3VlQ29tbWVudDcxMjk2MjUxNw== simonw 9599 2020-10-20T16:09:12Z 2020-10-20T16:09:12Z OWNER That `datasette.urls.table("db", "table") + ".json"` example is bad because if the table name contains a `.` it should be `?_format=json` instead. Maybe `.table()` should have a `format="json"` option that knows how to do this. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} How should datasette.client interact with base_url 722738988  
719959754 https://github.com/simonw/datasette/issues/1026#issuecomment-719959754 https://api.github.com/repos/simonw/datasette/issues/1026 MDEyOklzc3VlQ29tbWVudDcxOTk1OTc1NA== simonw 9599 2020-10-31T16:56:35Z 2020-10-31T16:56:35Z OWNER #1041 can also benefit from the string subclass that shows that `base_url` has been added. {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} How should datasette.client interact with base_url 722738988  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);