issue_comments
36 rows where issue = 1496652622
This data as json, CSV (advanced)
Suggested facets: user, author_association, created_at (date), updated_at (date)
| id ▼ | html_url | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1352643049 | https://github.com/simonw/datasette/issues/1955#issuecomment-1352643049 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qn7Hp | simonw 9599 | 2022-12-15T07:07:10Z | 2022-12-15T07:07:10Z | OWNER | This is definitely a regression: Datasette is meant to work in those environments, and I didn't think to test them when I added the `invoke_startup()` hook. Coincidentally I actually built a plugin for running Datasette with Gunicorn just a couple of months ago: https://datasette.io/plugins/datasette-gunicorn And I just tested and it has the same bug you describe here! Filed: - https://github.com/simonw/datasette-gunicorn/issues/5 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1352643333 | https://github.com/simonw/datasette/issues/1955#issuecomment-1352643333 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qn7MF | simonw 9599 | 2022-12-15T07:07:29Z | 2022-12-15T07:07:29Z | OWNER | Datasette 0.63 is the release that broke this, thanks to this issue: - https://github.com/simonw/datasette/issues/1809 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1352674924 | https://github.com/simonw/datasette/issues/1955#issuecomment-1352674924 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QoC5s | simonw 9599 | 2022-12-15T07:46:36Z | 2022-12-15T07:46:36Z | OWNER | It's possible the fix for this might be for the first incoming HTTP request to trigger `invoke_startup()` if it hasn't been called yet - similar to the hack I put in place for `datasette.client.get()` in tests: https://github.com/simonw/datasette/blob/e054704fb64d1f23154ec43b81b6c9481ff8202f/datasette/app.py#L1728-L1731 This would be a much more elegant fix, I could remove those multiple `invoke_startup()` calls entirely - and remove this tip from the documentation too: https://docs.datasette.io/en/0.63.2/testing_plugins.html#setting-up-a-datasette-test-instance | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1353423584 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353423584 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qq5rg | simonw 9599 | 2022-12-15T17:13:18Z | 2022-12-15T17:22:59Z | OWNER | Wow, just spotted this in the code - it turns out I solved this problem a different (and better) way long before i introduced `invoke_startup()`! https://github.com/simonw/datasette/blob/e054704fb64d1f23154ec43b81b6c9481ff8202f/datasette/app.py#L1416-L1440 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1353443718 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353443718 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qq-mG | simonw 9599 | 2022-12-15T17:23:12Z | 2022-12-15T17:23:55Z | OWNER | That may not be the best fix here. It turns out this pattern: ```python async def get(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.get(self._fix(path), **kwargs) ``` Doesn't trigger that `AsgiLifespan` class. I wrote about that previously in this TIL: https://til.simonwillison.net/asgi/lifespan-test-httpx | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1353448095 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353448095 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qq_qf | simonw 9599 | 2022-12-15T17:25:05Z | 2022-12-15T17:25:05Z | OWNER | So actually that `setup_db()` function I wrote back in 2019 has not been executing for most of Datasette's tests. Which seems bad. I'm inclined to ditch `AsgiLifespan` entirely in favour of the mechanism I described above, where `invoke_startup()` is called for every request on the first request processed by the server. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1353473086 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353473086 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrFw- | simonw 9599 | 2022-12-15T17:43:08Z | 2022-12-15T17:43:08Z | OWNER | It looks like that fix _almost_ works... except it seems to push the tests into an infinite loop or similar? They're not finishing their runs from what I can see. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1353473571 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353473571 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrF4j | simonw 9599 | 2022-12-15T17:43:28Z | 2022-12-15T17:43:48Z | OWNER | Running: pytest -n auto -x -v On may laptop to see if I can replicate. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1353509776 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353509776 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrOuQ | simonw 9599 | 2022-12-15T18:09:26Z | 2022-12-15T18:09:26Z | OWNER | I added this to `conftest.py`: ```python @pytest.fixture(autouse=True) def log_name_of_test_before_test(request): # To help identify tests that are hanging name = str(request.node) with open("/tmp/test.log", "a") as f: f.write(name + "\n") yield ``` This logs out the name of each test to `/tmp/test.log` before running the test - so I can wait until it hangs and see which test it was that caused that. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1353512099 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353512099 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrPSj | simonw 9599 | 2022-12-15T18:11:27Z | 2022-12-15T18:11:27Z | OWNER | This is surprising!  The logs suggest that the test suite hung running this test here: https://github.com/simonw/datasette/blob/dc18f62089e5672d03176f217d7840cdafa5c447/tests/test_utils.py#L55-L58 I find that very hard to believe. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1353516572 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353516572 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrQYc | simonw 9599 | 2022-12-15T18:15:28Z | 2022-12-15T18:15:28Z | OWNER | I added `return` to the first line of that test to disable it, then ran again - and now it's hanging at about the same progress point through the tests but in a different test:  So this time it was hanging at `test_urlsafe_components()`. So it's clearly not the individual tests themselves that are the problem - something about running the entire test suite in one go is incompatible with this change for some reason. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1353680261 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353680261 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qr4WF | simonw 9599 | 2022-12-15T20:39:19Z | 2022-12-15T20:39:19Z | OWNER | When I hit `Ctr+C` here's the traceback I get: ``` ^C^CException ignored in: <module 'threading' from '/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py'> Traceback (most recent call last): File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1530, in _shutdown !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py:324: KeyboardInterrupt (to show a full traceback on KeyboardInterrupt use --full-trace) Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/pytest", line 8, in <module> atexit_call() File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/concurrent/futures/thread.py", line 31, in _python_exit sys.exit(console_main()) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py", line 187, in console_main t.join() File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1089, in join self._wait_for_tstate_lock() File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock if lock.acquire(block, timeout): KeyboardInterrupt: code = main() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py", line 164, in main ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main( File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/Users/simon/.local/share/virtualenvs/datasette-AW… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1353683238 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353683238 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qr5Em | simonw 9599 | 2022-12-15T20:42:18Z | 2022-12-15T20:42:18Z | OWNER | Possibly related issue: - https://github.com/pytest-dev/pytest-xdist/issues/60 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1353694582 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353694582 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qr712 | simonw 9599 | 2022-12-15T20:52:46Z | 2022-12-15T20:52:46Z | OWNER | Just noticed this: https://github.com/simonw/datasette/actions/runs/3706504228/jobs/6281796135 <img width="746" alt="image" src="https://user-images.githubusercontent.com/9599/207964564-46c31685-db7a-494f-9aab-a611d25e5511.png"> This suggests that the regular tests passed in CI fine, but the non-serial ones failed. I'm going to try running everything using `pytest -n auto` without splitting serial and non-serial tests. Maybe the serial thing isn't needed any more? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1353701674 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353701674 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qr9kq | simonw 9599 | 2022-12-15T21:00:51Z | 2022-12-15T21:00:51Z | OWNER | OK, I've broken the test suite here. I'm going to revert these two commits: - https://github.com/simonw/datasette/commit/dc18f62089e5672d03176f217d7840cdafa5c447 - https://github.com/simonw/datasette/commit/51ee8caa4a697fa3f4120e93b1c205b714a6cdc7 Then I'll do a bunch of work making the test suite more robust before I try this again. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356487139 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356487139 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q2lnj | simonw 9599 | 2022-12-17T22:16:52Z | 2022-12-17T22:16:52Z | OWNER | I'm trying this fix again, after a bunch of work on the test suite in: - #1959 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356489200 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356489200 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q2mHw | simonw 9599 | 2022-12-17T22:29:51Z | 2022-12-17T22:29:51Z | OWNER | No, it still causes the tests to hang (I let them run for 12 minutes): <img width="664" alt="image" src="https://user-images.githubusercontent.com/9599/208268223-7fb40e3f-833b-4204-9acb-079adce6452f.png"> Interesting that the regular tests passed an then the `pytest -m serial` ones seem to have failed. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356595665 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356595665 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3AHR | simonw 9599 | 2022-12-18T00:58:16Z | 2022-12-18T00:58:16Z | OWNER | `pytest -m serial` on my Mac laptop also freezes: ``` (datasette) datasette % pytest -m serial ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items / 1264 deselected / 31 selected tests/test_package.py . [ 3%] tests/test_cli_serve_server.py ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356596740 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356596740 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3AYE | simonw 9599 | 2022-12-18T00:59:47Z | 2022-12-18T00:59:47Z | OWNER | Hitting `Ctrl+C` while using `--full-trace` gave me more clues: ``` % pytest -m serial tests/test_cli_serve_server.py --full-trace ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 3 items tests/test_cli_serve_server.py ^C^C ====================================================== no tests ran in 3.49s ======================================================= Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions yield File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/backends/sync.py", line 86, in connect_tcp sock = socket.create_connection( File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 845, in create_connection raise err File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 61] Connection refused [...] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356599930 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356599930 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3BJ6 | simonw 9599 | 2022-12-18T01:01:47Z | 2022-12-18T01:01:47Z | OWNER | I think that's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L6-L13 Using this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L155-L175 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356600917 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356600917 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3BZV | simonw 9599 | 2022-12-18T01:02:26Z | 2022-12-18T01:02:26Z | OWNER | This bit here looks like it could hang! ```python # Loop until port 8041 serves traffic while True: try: httpx.get("http://localhost:8041/") break except httpx.ConnectError: time.sleep(0.1) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356609095 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356609095 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3DZH | simonw 9599 | 2022-12-18T01:10:43Z | 2022-12-18T01:10:43Z | OWNER | Improved version of that fixture: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 44c44f87..69dee68b 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -27,6 +27,17 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None +def wait_until_responds(url, timeout=5.0, client=httpx, **kwargs): + start = time.time() + while time.time() - start < timeout: + try: + client.get(url, **kwargs) + return + except httpx.ConnectError: + time.sleep(0.1) + raise AssertionError("Timed out waiting for {} to respond".format(url)) + + @pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette @@ -161,13 +172,7 @@ def ds_localhost_http_server(): # Avoid FileNotFoundError: [Errno 2] No such file or directory: cwd=tempfile.gettempdir(), ) - # Loop until port 8041 serves traffic - while True: - try: - httpx.get("http://localhost:8041/") - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost:8041/") # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc @@ -202,12 +207,7 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - while True: - try: - httpx.get("https://localhost:8042/", verify=client_cert) - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost:8042/", verify=client_cert) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert @@ -231,12 +231,7 @@ def ds_unix_domain_socket_server(tmp_path_factory): # Poll until available transport = httpx.HTTPTransport(uds=uds) client = httpx.Client(transport=transport) - while T… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356610089 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356610089 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3Dop | simonw 9599 | 2022-12-18T01:12:39Z | 2022-12-18T01:12:39Z | OWNER | ... and it turns out those tests saved me. Because I forgot to check if `datasette` would actually start a server correctly! ``` % datasette fixtures.db -p 8852 INFO: Started server process [3538] INFO: Waiting for application startup. ERROR: Exception in 'lifespan' protocol Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/uvicorn/lifespan/on.py", line 86, in main await app(scope, self.receive, self.send) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__ return await self.app(scope, receive, send) File "/Users/simon/Dropbox/Development/datasette/datasette/utils/asgi.py", line 437, in __call__ return await self.asgi(scope, receive, send) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/asgi_csrf.py", line 39, in app_wrapped_with_csrf await app(scope, receive, send) File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1457, in __call__ path = scope["path"] KeyError: 'path' ERROR: Application startup failed. Exiting. ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356618913 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356618913 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3Fyh | simonw 9599 | 2022-12-18T01:29:05Z | 2022-12-18T01:29:05Z | OWNER | Now the only failure is in the `https` test - which fails like this (in CI and on my laptop): ``` message = str(exc) > raise mapped_exc(message) from exc E httpx.RemoteProtocolError: Server disconnected without sending a response. /opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/site-packages/httpx/_transports/default.py:77: RemoteProtocolError =========================== short test summary info ============================ ERROR tests/test_cli_serve_server.py::test_serve_localhost_https - httpx.RemoteProtocolError: Server disconnected without sending a response. ================= 30 passed, 1264 deselected, 1 error in 6.15s ================= ``` That's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L16-L24 And this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L178-L215 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356620233 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356620233 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3GHJ | simonw 9599 | 2022-12-18T01:31:10Z | 2022-12-18T01:31:10Z | OWNER | During the polling loop it constantly raises: `httpx.RemoteProtocolError`: Server disconnected without sending a response | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356625556 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356625556 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3HaU | simonw 9599 | 2022-12-18T02:00:18Z | 2022-12-18T02:00:18Z | OWNER | Maybe the reason the ASGI lifespan stuff broke was this line: https://github.com/simonw/datasette/blob/8b73fc6b47dffd8836f5c58aae1e57c1f66a5754/datasette/cli.py#L630-L632 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356625642 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356625642 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3Hbq | simonw 9599 | 2022-12-18T02:00:57Z | 2022-12-18T02:00:57Z | OWNER | I added the TLS support here: - https://github.com/simonw/datasette/issues/1221 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356626334 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356626334 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3Hme | simonw 9599 | 2022-12-18T02:04:01Z | 2022-12-18T02:04:07Z | OWNER | I used the steps to test manually from this comment: https://github.com/simonw/datasette/issues/1221#issuecomment-777901052 In one terminal: ``` cd /tmp python -m trustme datasette --memory --ssl-keyfile=/tmp/server.key --ssl-certfile=/tmp/server.pem -p 8003 ``` Then in another terminal: ``` curl --cacert /tmp/client.pem 'https://localhost:8003/_memory.json' ``` This worked correctly, outputting the expected JSON. So the feature still works, it's just the test that is broken for some reason. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356627331 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356627331 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3H2D | simonw 9599 | 2022-12-18T02:11:17Z | 2022-12-18T02:11:17Z | OWNER | This issue might be relevant, but I tried the suggested fix in there (`Connection: close` on the incoming requests) and it didn't fix my problem: - https://github.com/encode/httpx/discussions/2056 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356627931 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356627931 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3H_b | simonw 9599 | 2022-12-18T02:13:01Z | 2022-12-18T02:13:01Z | OWNER | Rather than continue to bang my head against this, I'm tempted to rewrite this test to happen outside of Python world - in a bash script run by GitHub Actions, for example. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356629783 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356629783 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3IcX | simonw 9599 | 2022-12-18T02:18:43Z | 2022-12-18T02:18:43Z | OWNER | Various attempts at a fix which didn't work: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 69dee68b..899d36fd 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,4 +1,3 @@ -import asyncio import httpx import os import pathlib @@ -6,6 +5,7 @@ import pytest import pytest_asyncio import re import subprocess +import sys import tempfile import time import trustme @@ -27,13 +27,23 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None -def wait_until_responds(url, timeout=5.0, client=httpx, **kwargs): +def wait_until_responds(url, timeout=5.0, client=None, **kwargs): + client = client or httpx.Client(**kwargs) start = time.time() while time.time() - start < timeout: try: - client.get(url, **kwargs) + if "verify" in kwargs: + print(kwargs["verify"]) + print( + "Contents of verify file: {}".format( + open(kwargs.get("verify")).read() + ) + ) + print("client = {}, kwargs = {}".format(client, kwargs)) + client.get(url) return - except httpx.ConnectError: + except (httpx.ConnectError, httpx.RemoteProtocolError) as ex: + print(ex) time.sleep(0.1) raise AssertionError("Timed out waiting for {} to respond".format(url)) @@ -166,7 +176,7 @@ def check_permission_actions_are_documented(): @pytest.fixture(scope="session") def ds_localhost_http_server(): ds_proc = subprocess.Popen( - ["datasette", "--memory", "-p", "8041"], + [sys.executable, "-m", "datasette", "--memory", "-p", "8041"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, # Avoid FileNotFoundError: [Errno 2] No such file or directory: @@ -180,7 +190,7 @@ def ds_localhost_http_server(): ds_proc.terminate() -@pytest.fixture(scope="session") +@pytest.fixture def ds_localhost_https_serve… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356630092 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356630092 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3IhM | simonw 9599 | 2022-12-18T02:20:01Z | 2022-12-18T02:20:01Z | OWNER | Asked ChatGPT: > Write a bash script which starts a server in the background using "datasette -p 8002", then uses curl to make a test request against it, then shuts the server down again at the end It gave me: ```bash #!/bin/bash # Start the server in the background datasette -p 8002 & # Store the background process ID in a variable server_pid=$! # Make a test request using curl curl http://localhost:8002 # Shut down the server kill $server_pid ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356633937 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356633937 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3JdR | simonw 9599 | 2022-12-18T02:28:22Z | 2022-12-18T02:28:22Z | OWNER | This `bash` script does the job: ```bash #!/bin/bash # Generate certificates python -m trustme # This creates server.pem, server.key, client.pem # Start the server in the background datasette --memory \ --ssl-keyfile=server.key \ --ssl-certfile=server.pem \ -p 8152 & # Store the background process ID in a variable server_pid=$! # Wait for the server to start sleep 2 # Make a test request using curl curl -f --cacert client.pem 'https://localhost:8152/_memory.json' # Save curl's exit code (-f option causes it to return one on HTTP errors) curl_exit_code=$? # Shut down the server kill $server_pid sleep 1 # Clean up the certificates rm server.pem server.key client.pem echo $curl_exit_code exit $curl_exit_code ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356640266 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356640266 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3LAK | simonw 9599 | 2022-12-18T02:43:00Z | 2022-12-18T02:43:00Z | OWNER | https://github.com/simonw/datasette/actions/runs/3722908296/jobs/6314093163 shows that new test passing in CI: ``` Generated a certificate for 'localhost', '127.0.0.1', '::1' Configure your server to use the following files: cert=/home/runner/work/datasette/datasette/server.pem key=/home/runner/work/datasette/datasette/server.key Configure your client to use the following files: cert=/home/runner/work/datasette/datasette/client.pem INFO: Started server process [4036] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on https://127.0.0.1:8152/ (Press CTRL+C to quit) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed INFO: 127.0.0.1:56726 - "GET /_memory.json HTTP/1.1" 200 OK 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 213 0 213 0 0 11542 0 --:--:-- --:--:-- --:--:-- 11833 INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [4036] {"database": "_memory", "private": false, "path": "/_memory", "size": 0, "tables": [], "hidden_count": 0, "views": [], "queries": [], "allow_execute_sql": true, "table_columns": {}, "query_ms": 1.4545189999921604}0 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1356640463 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356640463 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3LDP | simonw 9599 | 2022-12-18T02:45:18Z | 2022-12-18T02:45:18Z | OWNER | ... and with this change, the following now works correctly: ``` % datasette install datasette-gunicorn % datasette gunicorn fixtures.db -p 8855 [2022-12-17 18:44:29 -0800] [7651] [INFO] Starting gunicorn 20.1.0 [2022-12-17 18:44:29 -0800] [7651] [INFO] Listening at: http://127.0.0.1:8855 (7651) [2022-12-17 18:44:29 -0800] [7651] [INFO] Using worker: uvicorn.workers.UvicornWorker [2022-12-17 18:44:29 -0800] [7653] [INFO] Booting worker with pid: 7653 [2022-12-17 18:44:29 -0800] [7653] [INFO] Started server process [7653] [2022-12-17 18:44:29 -0800] [7653] [INFO] Waiting for application startup. [2022-12-17 18:44:29 -0800] [7653] [INFO] Application startup complete. ``` So this issue is now fixed! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
| 1357084279 | https://github.com/simonw/datasette/issues/1955#issuecomment-1357084279 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q43Z3 | andrewdotn 178162 | 2022-12-19T04:34:16Z | 2022-12-19T04:34:16Z | NONE | You were super-close on the python version of the test here, changing `http` to `https` on 8b73fc6b47dffd8836f5c58aae1e57c1f66a5754 is enough to pass the test: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 69dee68b4a3f..ba07a11d37f6 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -207,7 +207,7 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - wait_until_responds("http://localhost:8042/", verify=client_cert) + wait_until_responds("https://localhost:8042/", verify=client_cert) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert ``` My speculation about what was happening here: when `wait_until_responds()` would time out due to SSL connection problems, because `.terminate()` isn’t in a `finally`, the datasette process wouldn’t get killed. That could (1) hang CI and (2) cause all your future local test runs to mysteriously fail because they’d be secretly talking to that old datasette process still hanging around from a past test run with an old temporary server certificate, and that old server cert wouldn’t validate against your newly-created ca cert. A `finally` for `.terminate()` would help; a fancier version could be a context manager for running the external `datasette` process that could: - ensure the process always exited when no longer needed - if you want to be fancy, call `terminate()`, `wait()` for a short timeout for the process to exit, then try `kill()` and `wait()` again; raise an exception complaining about the seemingly-unkillable process if all that fails - raise an error if the process exited with a non-zero error code; here it’s likely that some `datasette` executions were secretly failing with `[Errno 48] error while attempting to bind on address ('127.0.0.1', 8042): address already in use` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] (
[html_url] TEXT,
[issue_url] TEXT,
[id] INTEGER PRIMARY KEY,
[node_id] TEXT,
[user] INTEGER REFERENCES [users]([id]),
[created_at] TEXT,
[updated_at] TEXT,
[author_association] TEXT,
[body] TEXT,
[reactions] TEXT,
[issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
ON [issue_comments] ([user]);