html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/issues/1955#issuecomment-1352643049,https://api.github.com/repos/simonw/datasette/issues/1955,1352643049,IC_kwDOBm6k_c5Qn7Hp,9599,2022-12-15T07:07:10Z,2022-12-15T07:07:10Z,OWNER,"This is definitely a regression: Datasette is meant to work in those environments, and I didn't think to test them when I added the `invoke_startup()` hook. Coincidentally I actually built a plugin for running Datasette with Gunicorn just a couple of months ago: https://datasette.io/plugins/datasette-gunicorn And I just tested and it has the same bug you describe here! Filed: - https://github.com/simonw/datasette-gunicorn/issues/5 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1352643333,https://api.github.com/repos/simonw/datasette/issues/1955,1352643333,IC_kwDOBm6k_c5Qn7MF,9599,2022-12-15T07:07:29Z,2022-12-15T07:07:29Z,OWNER,"Datasette 0.63 is the release that broke this, thanks to this issue: - https://github.com/simonw/datasette/issues/1809","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1352674924,https://api.github.com/repos/simonw/datasette/issues/1955,1352674924,IC_kwDOBm6k_c5QoC5s,9599,2022-12-15T07:46:36Z,2022-12-15T07:46:36Z,OWNER,"It's possible the fix for this might be for the first incoming HTTP request to trigger `invoke_startup()` if it hasn't been called yet - similar to the hack I put in place for `datasette.client.get()` in tests: https://github.com/simonw/datasette/blob/e054704fb64d1f23154ec43b81b6c9481ff8202f/datasette/app.py#L1728-L1731 This would be a much more elegant fix, I could remove those multiple `invoke_startup()` calls entirely - and remove this tip from the documentation too: https://docs.datasette.io/en/0.63.2/testing_plugins.html#setting-up-a-datasette-test-instance","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353423584,https://api.github.com/repos/simonw/datasette/issues/1955,1353423584,IC_kwDOBm6k_c5Qq5rg,9599,2022-12-15T17:13:18Z,2022-12-15T17:22:59Z,OWNER,"Wow, just spotted this in the code - it turns out I solved this problem a different (and better) way long before i introduced `invoke_startup()`! https://github.com/simonw/datasette/blob/e054704fb64d1f23154ec43b81b6c9481ff8202f/datasette/app.py#L1416-L1440","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353443718,https://api.github.com/repos/simonw/datasette/issues/1955,1353443718,IC_kwDOBm6k_c5Qq-mG,9599,2022-12-15T17:23:12Z,2022-12-15T17:23:55Z,OWNER,"That may not be the best fix here. It turns out this pattern: ```python async def get(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.get(self._fix(path), **kwargs) ``` Doesn't trigger that `AsgiLifespan` class. I wrote about that previously in this TIL: https://til.simonwillison.net/asgi/lifespan-test-httpx","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353448095,https://api.github.com/repos/simonw/datasette/issues/1955,1353448095,IC_kwDOBm6k_c5Qq_qf,9599,2022-12-15T17:25:05Z,2022-12-15T17:25:05Z,OWNER,"So actually that `setup_db()` function I wrote back in 2019 has not been executing for most of Datasette's tests. Which seems bad. I'm inclined to ditch `AsgiLifespan` entirely in favour of the mechanism I described above, where `invoke_startup()` is called for every request on the first request processed by the server.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353473086,https://api.github.com/repos/simonw/datasette/issues/1955,1353473086,IC_kwDOBm6k_c5QrFw-,9599,2022-12-15T17:43:08Z,2022-12-15T17:43:08Z,OWNER,It looks like that fix _almost_ works... except it seems to push the tests into an infinite loop or similar? They're not finishing their runs from what I can see.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353473571,https://api.github.com/repos/simonw/datasette/issues/1955,1353473571,IC_kwDOBm6k_c5QrF4j,9599,2022-12-15T17:43:28Z,2022-12-15T17:43:48Z,OWNER,"Running: pytest -n auto -x -v On may laptop to see if I can replicate.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353509776,https://api.github.com/repos/simonw/datasette/issues/1955,1353509776,IC_kwDOBm6k_c5QrOuQ,9599,2022-12-15T18:09:26Z,2022-12-15T18:09:26Z,OWNER,"I added this to `conftest.py`: ```python @pytest.fixture(autouse=True) def log_name_of_test_before_test(request): # To help identify tests that are hanging name = str(request.node) with open(""/tmp/test.log"", ""a"") as f: f.write(name + ""\n"") yield ``` This logs out the name of each test to `/tmp/test.log` before running the test - so I can wait until it hangs and see which test it was that caused that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353512099,https://api.github.com/repos/simonw/datasette/issues/1955,1353512099,IC_kwDOBm6k_c5QrPSj,9599,2022-12-15T18:11:27Z,2022-12-15T18:11:27Z,OWNER,"This is surprising! ![Image](https://user-images.githubusercontent.com/9599/207935885-e1f51983-0621-4490-86a6-fafd4c876f41.png) The logs suggest that the test suite hung running this test here: https://github.com/simonw/datasette/blob/dc18f62089e5672d03176f217d7840cdafa5c447/tests/test_utils.py#L55-L58 I find that very hard to believe.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353516572,https://api.github.com/repos/simonw/datasette/issues/1955,1353516572,IC_kwDOBm6k_c5QrQYc,9599,2022-12-15T18:15:28Z,2022-12-15T18:15:28Z,OWNER,"I added `return` to the first line of that test to disable it, then ran again - and now it's hanging at about the same progress point through the tests but in a different test: ![Image](https://user-images.githubusercontent.com/9599/207936587-30ebf780-c0da-4e62-b20b-e274e0adaa19.png) So this time it was hanging at `test_urlsafe_components()`. So it's clearly not the individual tests themselves that are the problem - something about running the entire test suite in one go is incompatible with this change for some reason.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353680261,https://api.github.com/repos/simonw/datasette/issues/1955,1353680261,IC_kwDOBm6k_c5Qr4WF,9599,2022-12-15T20:39:19Z,2022-12-15T20:39:19Z,OWNER,"When I hit `Ctr+C` here's the traceback I get: ``` ^C^CException ignored in: Traceback (most recent call last): File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py"", line 1530, in _shutdown !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py:324: KeyboardInterrupt (to show a full traceback on KeyboardInterrupt use --full-trace) Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/pytest"", line 8, in atexit_call() File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/concurrent/futures/thread.py"", line 31, in _python_exit sys.exit(console_main()) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py"", line 187, in console_main t.join() File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py"", line 1089, in join self._wait_for_tstate_lock() File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py"", line 1109, in _wait_for_tstate_lock if lock.acquire(block, timeout): KeyboardInterrupt: code = main() File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py"", line 164, in main ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main( File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_hooks.py"", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_manager.py"", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py"", line 60, in _multicall return outcome.get_result() File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_result.py"", line 60, in get_result raise ex[1].with_traceback(ex[2]) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py"", line 39, in _multicall res = hook_impl.function(*args) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/main.py"", line 315, in pytest_cmdline_main return wrap_session(config, _main) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/main.py"", line 303, in wrap_session config.hook.pytest_sessionfinish( File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_hooks.py"", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_manager.py"", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py"", line 55, in _multicall gen.send(outcome) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/terminal.py"", line 798, in pytest_sessionfinish outcome.get_result() File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_result.py"", line 60, in get_result raise ex[1].with_traceback(ex[2]) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py"", line 39, in _multicall res = hook_impl.function(*args) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/xdist/dsession.py"", line 88, in pytest_sessionfinish nm.teardown_nodes() File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/xdist/workermanage.py"", line 79, in teardown_nodes self.group.terminate(self.EXIT_TIMEOUT) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/multi.py"", line 215, in terminate safe_terminate( File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/multi.py"", line 311, in safe_terminate reply.get() File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/gateway_base.py"", line 206, in get self.waitfinish(timeout) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/gateway_base.py"", line 213, in waitfinish if not self._result_ready.wait(timeout): File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py"", line 600, in wait signaled = self._cond.wait(timeout) File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py"", line 320, in wait waiter.acquire() KeyboardInterrupt ``` It looks to me like this relates to `pytest-xdist` istelf - it's waiting on some locks but `site-packages/xdist/workermanage.py` shows up in that track.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353683238,https://api.github.com/repos/simonw/datasette/issues/1955,1353683238,IC_kwDOBm6k_c5Qr5Em,9599,2022-12-15T20:42:18Z,2022-12-15T20:42:18Z,OWNER,"Possibly related issue: - https://github.com/pytest-dev/pytest-xdist/issues/60","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353694582,https://api.github.com/repos/simonw/datasette/issues/1955,1353694582,IC_kwDOBm6k_c5Qr712,9599,2022-12-15T20:52:46Z,2022-12-15T20:52:46Z,OWNER,"Just noticed this: https://github.com/simonw/datasette/actions/runs/3706504228/jobs/6281796135 This suggests that the regular tests passed in CI fine, but the non-serial ones failed. I'm going to try running everything using `pytest -n auto` without splitting serial and non-serial tests. Maybe the serial thing isn't needed any more?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1353701674,https://api.github.com/repos/simonw/datasette/issues/1955,1353701674,IC_kwDOBm6k_c5Qr9kq,9599,2022-12-15T21:00:51Z,2022-12-15T21:00:51Z,OWNER,"OK, I've broken the test suite here. I'm going to revert these two commits: - https://github.com/simonw/datasette/commit/dc18f62089e5672d03176f217d7840cdafa5c447 - https://github.com/simonw/datasette/commit/51ee8caa4a697fa3f4120e93b1c205b714a6cdc7 Then I'll do a bunch of work making the test suite more robust before I try this again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356487139,https://api.github.com/repos/simonw/datasette/issues/1955,1356487139,IC_kwDOBm6k_c5Q2lnj,9599,2022-12-17T22:16:52Z,2022-12-17T22:16:52Z,OWNER,"I'm trying this fix again, after a bunch of work on the test suite in: - #1959","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356489200,https://api.github.com/repos/simonw/datasette/issues/1955,1356489200,IC_kwDOBm6k_c5Q2mHw,9599,2022-12-17T22:29:51Z,2022-12-17T22:29:51Z,OWNER,"No, it still causes the tests to hang (I let them run for 12 minutes): Interesting that the regular tests passed an then the `pytest -m serial` ones seem to have failed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356595665,https://api.github.com/repos/simonw/datasette/issues/1955,1356595665,IC_kwDOBm6k_c5Q3AHR,9599,2022-12-18T00:58:16Z,2022-12-18T00:58:16Z,OWNER,"`pytest -m serial` on my Mac laptop also freezes: ``` (datasette) datasette % pytest -m serial ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items / 1264 deselected / 31 selected tests/test_package.py . [ 3%] tests/test_cli_serve_server.py ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356596740,https://api.github.com/repos/simonw/datasette/issues/1955,1356596740,IC_kwDOBm6k_c5Q3AYE,9599,2022-12-18T00:59:47Z,2022-12-18T00:59:47Z,OWNER,"Hitting `Ctrl+C` while using `--full-trace` gave me more clues: ``` % pytest -m serial tests/test_cli_serve_server.py --full-trace ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 3 items tests/test_cli_serve_server.py ^C^C ====================================================== no tests ran in 3.49s ======================================================= Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/_exceptions.py"", line 8, in map_exceptions yield File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/backends/sync.py"", line 86, in connect_tcp sock = socket.create_connection( File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py"", line 845, in create_connection raise err File ""/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py"", line 833, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 61] Connection refused [...] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356599930,https://api.github.com/repos/simonw/datasette/issues/1955,1356599930,IC_kwDOBm6k_c5Q3BJ6,9599,2022-12-18T01:01:47Z,2022-12-18T01:01:47Z,OWNER,"I think that's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L6-L13 Using this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L155-L175","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356600917,https://api.github.com/repos/simonw/datasette/issues/1955,1356600917,IC_kwDOBm6k_c5Q3BZV,9599,2022-12-18T01:02:26Z,2022-12-18T01:02:26Z,OWNER,"This bit here looks like it could hang! ```python # Loop until port 8041 serves traffic while True: try: httpx.get(""http://localhost:8041/"") break except httpx.ConnectError: time.sleep(0.1) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356609095,https://api.github.com/repos/simonw/datasette/issues/1955,1356609095,IC_kwDOBm6k_c5Q3DZH,9599,2022-12-18T01:10:43Z,2022-12-18T01:10:43Z,OWNER,"Improved version of that fixture: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 44c44f87..69dee68b 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -27,6 +27,17 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None +def wait_until_responds(url, timeout=5.0, client=httpx, **kwargs): + start = time.time() + while time.time() - start < timeout: + try: + client.get(url, **kwargs) + return + except httpx.ConnectError: + time.sleep(0.1) + raise AssertionError(""Timed out waiting for {} to respond"".format(url)) + + @pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette @@ -161,13 +172,7 @@ def ds_localhost_http_server(): # Avoid FileNotFoundError: [Errno 2] No such file or directory: cwd=tempfile.gettempdir(), ) - # Loop until port 8041 serves traffic - while True: - try: - httpx.get(""http://localhost:8041/"") - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds(""http://localhost:8041/"") # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode(""utf-8"") yield ds_proc @@ -202,12 +207,7 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - while True: - try: - httpx.get(""https://localhost:8042/"", verify=client_cert) - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds(""http://localhost:8042/"", verify=client_cert) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode(""utf-8"") yield ds_proc, client_cert @@ -231,12 +231,7 @@ def ds_unix_domain_socket_server(tmp_path_factory): # Poll until available transport = httpx.HTTPTransport(uds=uds) client = httpx.Client(transport=transport) - while True: - try: - client.get(""http://localhost/_memory.json"") - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds(""http://localhost/_memory.json"", client=client) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode(""utf-8"") yield ds_proc, uds ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356610089,https://api.github.com/repos/simonw/datasette/issues/1955,1356610089,IC_kwDOBm6k_c5Q3Dop,9599,2022-12-18T01:12:39Z,2022-12-18T01:12:39Z,OWNER,"... and it turns out those tests saved me. Because I forgot to check if `datasette` would actually start a server correctly! ``` % datasette fixtures.db -p 8852 INFO: Started server process [3538] INFO: Waiting for application startup. ERROR: Exception in 'lifespan' protocol Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/uvicorn/lifespan/on.py"", line 86, in main await app(scope, self.receive, self.send) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py"", line 78, in __call__ return await self.app(scope, receive, send) File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/asgi.py"", line 437, in __call__ return await self.asgi(scope, receive, send) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/asgi_csrf.py"", line 39, in app_wrapped_with_csrf await app(scope, receive, send) File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 1457, in __call__ path = scope[""path""] KeyError: 'path' ERROR: Application startup failed. Exiting. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356618913,https://api.github.com/repos/simonw/datasette/issues/1955,1356618913,IC_kwDOBm6k_c5Q3Fyh,9599,2022-12-18T01:29:05Z,2022-12-18T01:29:05Z,OWNER,"Now the only failure is in the `https` test - which fails like this (in CI and on my laptop): ``` message = str(exc) > raise mapped_exc(message) from exc E httpx.RemoteProtocolError: Server disconnected without sending a response. /opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/site-packages/httpx/_transports/default.py:77: RemoteProtocolError =========================== short test summary info ============================ ERROR tests/test_cli_serve_server.py::test_serve_localhost_https - httpx.RemoteProtocolError: Server disconnected without sending a response. ================= 30 passed, 1264 deselected, 1 error in 6.15s ================= ``` That's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L16-L24 And this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L178-L215 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356620233,https://api.github.com/repos/simonw/datasette/issues/1955,1356620233,IC_kwDOBm6k_c5Q3GHJ,9599,2022-12-18T01:31:10Z,2022-12-18T01:31:10Z,OWNER,"During the polling loop it constantly raises: `httpx.RemoteProtocolError`: Server disconnected without sending a response","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356625556,https://api.github.com/repos/simonw/datasette/issues/1955,1356625556,IC_kwDOBm6k_c5Q3HaU,9599,2022-12-18T02:00:18Z,2022-12-18T02:00:18Z,OWNER,Maybe the reason the ASGI lifespan stuff broke was this line: https://github.com/simonw/datasette/blob/8b73fc6b47dffd8836f5c58aae1e57c1f66a5754/datasette/cli.py#L630-L632,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356625642,https://api.github.com/repos/simonw/datasette/issues/1955,1356625642,IC_kwDOBm6k_c5Q3Hbq,9599,2022-12-18T02:00:57Z,2022-12-18T02:00:57Z,OWNER,"I added the TLS support here: - https://github.com/simonw/datasette/issues/1221","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356626334,https://api.github.com/repos/simonw/datasette/issues/1955,1356626334,IC_kwDOBm6k_c5Q3Hme,9599,2022-12-18T02:04:01Z,2022-12-18T02:04:07Z,OWNER,"I used the steps to test manually from this comment: https://github.com/simonw/datasette/issues/1221#issuecomment-777901052 In one terminal: ``` cd /tmp python -m trustme datasette --memory --ssl-keyfile=/tmp/server.key --ssl-certfile=/tmp/server.pem -p 8003 ``` Then in another terminal: ``` curl --cacert /tmp/client.pem 'https://localhost:8003/_memory.json' ``` This worked correctly, outputting the expected JSON. So the feature still works, it's just the test that is broken for some reason.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356627331,https://api.github.com/repos/simonw/datasette/issues/1955,1356627331,IC_kwDOBm6k_c5Q3H2D,9599,2022-12-18T02:11:17Z,2022-12-18T02:11:17Z,OWNER,"This issue might be relevant, but I tried the suggested fix in there (`Connection: close` on the incoming requests) and it didn't fix my problem: - https://github.com/encode/httpx/discussions/2056","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356627931,https://api.github.com/repos/simonw/datasette/issues/1955,1356627931,IC_kwDOBm6k_c5Q3H_b,9599,2022-12-18T02:13:01Z,2022-12-18T02:13:01Z,OWNER,"Rather than continue to bang my head against this, I'm tempted to rewrite this test to happen outside of Python world - in a bash script run by GitHub Actions, for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356629783,https://api.github.com/repos/simonw/datasette/issues/1955,1356629783,IC_kwDOBm6k_c5Q3IcX,9599,2022-12-18T02:18:43Z,2022-12-18T02:18:43Z,OWNER,"Various attempts at a fix which didn't work: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 69dee68b..899d36fd 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,4 +1,3 @@ -import asyncio import httpx import os import pathlib @@ -6,6 +5,7 @@ import pytest import pytest_asyncio import re import subprocess +import sys import tempfile import time import trustme @@ -27,13 +27,23 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None -def wait_until_responds(url, timeout=5.0, client=httpx, **kwargs): +def wait_until_responds(url, timeout=5.0, client=None, **kwargs): + client = client or httpx.Client(**kwargs) start = time.time() while time.time() - start < timeout: try: - client.get(url, **kwargs) + if ""verify"" in kwargs: + print(kwargs[""verify""]) + print( + ""Contents of verify file: {}"".format( + open(kwargs.get(""verify"")).read() + ) + ) + print(""client = {}, kwargs = {}"".format(client, kwargs)) + client.get(url) return - except httpx.ConnectError: + except (httpx.ConnectError, httpx.RemoteProtocolError) as ex: + print(ex) time.sleep(0.1) raise AssertionError(""Timed out waiting for {} to respond"".format(url)) @@ -166,7 +176,7 @@ def check_permission_actions_are_documented(): @pytest.fixture(scope=""session"") def ds_localhost_http_server(): ds_proc = subprocess.Popen( - [""datasette"", ""--memory"", ""-p"", ""8041""], + [sys.executable, ""-m"", ""datasette"", ""--memory"", ""-p"", ""8041""], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, # Avoid FileNotFoundError: [Errno 2] No such file or directory: @@ -180,7 +190,7 @@ def ds_localhost_http_server(): ds_proc.terminate() -@pytest.fixture(scope=""session"") +@pytest.fixture def ds_localhost_https_server(tmp_path_factory): cert_directory = tmp_path_factory.mktemp(""certs"") ca = trustme.CA() @@ -194,6 +204,8 @@ def ds_localhost_https_server(tmp_path_factory): ca.cert_pem.write_to_path(path=client_cert) ds_proc = subprocess.Popen( [ + sys.executable, + ""-m"", ""datasette"", ""--memory"", ""-p"", @@ -207,7 +219,11 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - wait_until_responds(""http://localhost:8042/"", verify=client_cert) + wait_until_responds( + ""http://localhost:8042/_memory.json"", + verify=client_cert, + headers={""Connection"": ""close""}, + ) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode(""utf-8"") yield ds_proc, client_cert diff --git a/tests/test_cli_serve_server.py b/tests/test_cli_serve_server.py index 1c31e2a3..9320b623 100644 --- a/tests/test_cli_serve_server.py +++ b/tests/test_cli_serve_server.py @@ -16,7 +16,11 @@ def test_serve_localhost_http(ds_localhost_http_server): @pytest.mark.serial def test_serve_localhost_https(ds_localhost_https_server): _, client_cert = ds_localhost_https_server - response = httpx.get(""https://localhost:8042/_memory.json"", verify=client_cert) + response = httpx.get( + ""https://localhost:8042/_memory.json"", + verify=client_cert, + headers={""Connection"": ""close""}, + ) assert { ""database"": ""_memory"", ""path"": ""/_memory"", ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356630092,https://api.github.com/repos/simonw/datasette/issues/1955,1356630092,IC_kwDOBm6k_c5Q3IhM,9599,2022-12-18T02:20:01Z,2022-12-18T02:20:01Z,OWNER,"Asked ChatGPT: > Write a bash script which starts a server in the background using ""datasette -p 8002"", then uses curl to make a test request against it, then shuts the server down again at the end It gave me: ```bash #!/bin/bash # Start the server in the background datasette -p 8002 & # Store the background process ID in a variable server_pid=$! # Make a test request using curl curl http://localhost:8002 # Shut down the server kill $server_pid ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356633937,https://api.github.com/repos/simonw/datasette/issues/1955,1356633937,IC_kwDOBm6k_c5Q3JdR,9599,2022-12-18T02:28:22Z,2022-12-18T02:28:22Z,OWNER,"This `bash` script does the job: ```bash #!/bin/bash # Generate certificates python -m trustme # This creates server.pem, server.key, client.pem # Start the server in the background datasette --memory \ --ssl-keyfile=server.key \ --ssl-certfile=server.pem \ -p 8152 & # Store the background process ID in a variable server_pid=$! # Wait for the server to start sleep 2 # Make a test request using curl curl -f --cacert client.pem 'https://localhost:8152/_memory.json' # Save curl's exit code (-f option causes it to return one on HTTP errors) curl_exit_code=$? # Shut down the server kill $server_pid sleep 1 # Clean up the certificates rm server.pem server.key client.pem echo $curl_exit_code exit $curl_exit_code ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356640266,https://api.github.com/repos/simonw/datasette/issues/1955,1356640266,IC_kwDOBm6k_c5Q3LAK,9599,2022-12-18T02:43:00Z,2022-12-18T02:43:00Z,OWNER,"https://github.com/simonw/datasette/actions/runs/3722908296/jobs/6314093163 shows that new test passing in CI: ``` Generated a certificate for 'localhost', '127.0.0.1', '::1' Configure your server to use the following files: cert=/home/runner/work/datasette/datasette/server.pem key=/home/runner/work/datasette/datasette/server.key Configure your client to use the following files: cert=/home/runner/work/datasette/datasette/client.pem INFO: Started server process [4036] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on https://127.0.0.1:8152/ (Press CTRL+C to quit) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed INFO: 127.0.0.1:56726 - ""GET /_memory.json HTTP/1.1"" 200 OK 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 213 0 213 0 0 11542 0 --:--:-- --:--:-- --:--:-- 11833 INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [4036] {""database"": ""_memory"", ""private"": false, ""path"": ""/_memory"", ""size"": 0, ""tables"": [], ""hidden_count"": 0, ""views"": [], ""queries"": [], ""allow_execute_sql"": true, ""table_columns"": {}, ""query_ms"": 1.4545189999921604}0 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1356640463,https://api.github.com/repos/simonw/datasette/issues/1955,1356640463,IC_kwDOBm6k_c5Q3LDP,9599,2022-12-18T02:45:18Z,2022-12-18T02:45:18Z,OWNER,"... and with this change, the following now works correctly: ``` % datasette install datasette-gunicorn % datasette gunicorn fixtures.db -p 8855 [2022-12-17 18:44:29 -0800] [7651] [INFO] Starting gunicorn 20.1.0 [2022-12-17 18:44:29 -0800] [7651] [INFO] Listening at: http://127.0.0.1:8855 (7651) [2022-12-17 18:44:29 -0800] [7651] [INFO] Using worker: uvicorn.workers.UvicornWorker [2022-12-17 18:44:29 -0800] [7653] [INFO] Booting worker with pid: 7653 [2022-12-17 18:44:29 -0800] [7653] [INFO] Started server process [7653] [2022-12-17 18:44:29 -0800] [7653] [INFO] Waiting for application startup. [2022-12-17 18:44:29 -0800] [7653] [INFO] Application startup complete. ``` So this issue is now fixed!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622, https://github.com/simonw/datasette/issues/1955#issuecomment-1357084279,https://api.github.com/repos/simonw/datasette/issues/1955,1357084279,IC_kwDOBm6k_c5Q43Z3,178162,2022-12-19T04:34:16Z,2022-12-19T04:34:16Z,NONE,"You were super-close on the python version of the test here, changing `http` to `https` on 8b73fc6b47dffd8836f5c58aae1e57c1f66a5754 is enough to pass the test: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 69dee68b4a3f..ba07a11d37f6 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -207,7 +207,7 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - wait_until_responds(""http://localhost:8042/"", verify=client_cert) + wait_until_responds(""https://localhost:8042/"", verify=client_cert) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode(""utf-8"") yield ds_proc, client_cert ``` My speculation about what was happening here: when `wait_until_responds()` would time out due to SSL connection problems, because `.terminate()` isn’t in a `finally`, the datasette process wouldn’t get killed. That could (1) hang CI and (2) cause all your future local test runs to mysteriously fail because they’d be secretly talking to that old datasette process still hanging around from a past test run with an old temporary server certificate, and that old server cert wouldn’t validate against your newly-created ca cert. A `finally` for `.terminate()` would help; a fancier version could be a context manager for running the external `datasette` process that could: - ensure the process always exited when no longer needed - if you want to be fancy, call `terminate()`, `wait()` for a short timeout for the process to exit, then try `kill()` and `wait()` again; raise an exception complaining about the seemingly-unkillable process if all that fails - raise an error if the process exited with a non-zero error code; here it’s likely that some `datasette` executions were secretly failing with `[Errno 48] error while attempting to bind on address ('127.0.0.1', 8042): address already in use`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1496652622,