html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/datasette/issues/1958#issuecomment-1353977605,https://api.github.com/repos/simonw/datasette/issues/1958,1353977605,IC_kwDOBm6k_c5QtA8F,9599,simonw,2022-12-16T00:38:23Z,2022-12-16T00:38:23Z,OWNER,"Confirmed, I just got the same result: ``` - % docker run datasetteproject/datasette pip install datasette-upload-csvs ~ % docker commit $(docker ps -lq) datasette-with-plugins sha256:8cde4a6357b9221d6f9e15887a314f2b4d9d1b87b517764d207ccbaec7c0a69f ~ % docker run -p 8001:8001 -v $(pwd):/mnt datasette-with-plugins datasette --root -p 8001 -h 0.0.0.0 INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ^CINFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [1] http://0.0.0.0:8001/-/auth-token?token=4bd70fdbca215ea55c874eaf889adf8c09f2a00231f7e5e6d0470f3176407a98 ``` Note how the auth-token URL is only displayed after you hit `Ctrl+C`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,datasette --root running in Docker doesn't reliably show the magic URL, https://github.com/simonw/datasette/issues/1958#issuecomment-1354008688,https://api.github.com/repos/simonw/datasette/issues/1958,1354008688,IC_kwDOBm6k_c5QtIhw,9599,simonw,2022-12-16T00:45:40Z,2022-12-16T00:45:40Z,OWNER,The fix may just be to switch to `click.echo()`. https://click.palletsprojects.com/en/8.1.x/api/#click.echo,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,datasette --root running in Docker doesn't reliably show the magic URL, https://github.com/simonw/datasette/issues/1958#issuecomment-1354019543,https://api.github.com/repos/simonw/datasette/issues/1958,1354019543,IC_kwDOBm6k_c5QtLLX,9599,simonw,2022-12-16T00:54:11Z,2022-12-16T00:54:11Z,OWNER,"To test the fix, I did a fresh checkout of `simonw/datasette` and edited `Dockerfile` to look like this: ```dockerfile FROM python:3.11.0-slim-bullseye as build RUN apt-get update && \ apt-get install -y --no-install-recommends libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/apt && \ rm -rf /var/lib/dpkg/info/* RUN pip install https://github.com/simonw/datasette/archive/refs/heads/main.zip && \ find /usr/local/lib -name '__pycache__' | xargs rm -r && \ rm -rf /root/.cache/pip EXPOSE 8001 CMD [""datasette""] ``` Then I built it like this: ``` docker build -f Dockerfile -t datasette-pre-click . ``` And ran like this: ``` docker run -p 8001:8001 -v $(pwd):/mnt datasette-pre-click datasette --root -p 8001 -h 0.0.0.0 ``` This exhibited the same problem. Then I pushed a changed to branch and changed the line to: ``` RUN pip install https://github.com/simonw/datasette/archive/refs/heads/click-echo-root.zip && \ ``` Ran this: ``` docker build -f Dockerfile -t datasette-post-click . ``` And this: ``` docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0 ``` It fixed the problem! ``` datasette % docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0 http://0.0.0.0:8001/-/auth-token?token=6542dcf5c8f34f8d13f4af9ce728359c602469efb54029098562bd06c87ad26d INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,datasette --root running in Docker doesn't reliably show the magic URL, https://github.com/simonw/datasette/issues/1958#issuecomment-1354023960,https://api.github.com/repos/simonw/datasette/issues/1958,1354023960,IC_kwDOBm6k_c5QtMQY,9599,simonw,2022-12-16T00:58:14Z,2022-12-16T00:58:19Z,OWNER,"This is in the `0.63.x` branch now, ready to go out in a bug fix release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,datasette --root running in Docker doesn't reliably show the magic URL, https://github.com/simonw/datasette/issues/1958#issuecomment-1354025319,https://api.github.com/repos/simonw/datasette/issues/1958,1354025319,IC_kwDOBm6k_c5QtMln,11729897,davidhaley,2022-12-16T00:59:12Z,2022-12-16T00:59:12Z,NONE,Awesome. Thank you @simonw.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1497909798,datasette --root running in Docker doesn't reliably show the magic URL, https://github.com/simonw/datasette/pull/1960#issuecomment-1354036967,https://api.github.com/repos/simonw/datasette/issues/1960,1354036967,IC_kwDOBm6k_c5QtPbn,9599,simonw,2022-12-16T01:10:12Z,2022-12-16T01:10:12Z,OWNER,"If it does turn out that I can't use `scope=""session""` on this fixture it might not actually be a showstopper: I can take advantage of the fact that `memory_name=""...""` databases stay present in memory for the duration of the process, so I could have it such that each test that uses the `ds_client` fixture DOES construct a fresh `Datasette` instance, but doesn't need to populate the database since they can re-use the in-memory database from the previous object.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1354046627,https://api.github.com/repos/simonw/datasette/issues/1960,1354046627,IC_kwDOBm6k_c5QtRyj,9599,simonw,2022-12-16T01:23:21Z,2022-12-16T01:23:21Z,OWNER,"This does seem to help: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 1306c407..af9c7696 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -25,12 +25,7 @@ UNDOCUMENTED_PERMISSIONS = { } -@pytest.fixture(scope=""session"") -def event_loop(): - return asyncio.get_event_loop() - - -@pytest_asyncio.fixture(scope=""session"") +@pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette from .fixtures import METADATA, PLUGINS_DIR @@ -53,10 +48,11 @@ async def ds_client(): db = ds.add_memory_database(""fixtures"") def prepare(conn): - conn.executescript(TABLES) - for sql, params in TABLE_PARAMETERIZED_SQL: - with conn: - conn.execute(sql, params) + if not conn.execute(""select count(*) from sqlite_master"").fetchone()[0]: + conn.executescript(TABLES) + for sql, params in TABLE_PARAMETERIZED_SQL: + with conn: + conn.execute(sql, params) await db.execute_write_fn(prepare) return ds.client diff --git a/tests/plugins/my_plugin_2.py b/tests/plugins/my_plugin_2.py index 4f7bf08c..d588342c 100644 --- a/tests/plugins/my_plugin_2.py +++ b/tests/plugins/my_plugin_2.py @@ -117,7 +117,12 @@ def actor_from_request(datasette, request): def permission_allowed(datasette, actor, action): # Testing asyncio version of permission_allowed async def inner(): - assert 2 == (await datasette.get_database().execute(""select 1 + 1"")).first()[0] + assert ( + 2 + == ( + await datasette.get_database(""_internal"").execute(""select 1 + 1"") + ).first()[0] + ) if action == ""this_is_allowed_async"": return True elif action == ""this_is_denied_async"": ``` `pytest -m ds_client` now passes 134 tests. Need to get `pytest -n auto` passing too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1354053151,https://api.github.com/repos/simonw/datasette/issues/1960,1354053151,IC_kwDOBm6k_c5QtTYf,9599,simonw,2022-12-16T01:33:22Z,2022-12-16T01:33:22Z,OWNER,"The thing with `Datasette(memory=False)` is tripping me up. The problem is that the tests written against `app_client` - which I want to replace - all assume that there is no `_memory` database, because when you start Datasette with at least one database file it doesn't enable `_memory` unless you explicitly tell it to. But the new `ds_client` fixture works by creating a named in-memory database called `fixtures`, which it does with a call to `ds.add_memory_database(""fixtures"")` after the object has been instantiated. This results in a datasette instance that DOES have a `_memory` database, when we didn't want one. My initial solution attempt was a huge hack - I decided that if you pass `memory=False` to the `Datasette` constructor it should mean ""don't add a `_memory` database even though I didn't pass any files"". I set a the default `memory` argument to `None`. This is weird and surprising (`memory=False` no does something different from `memory=None`?) and I found other tests that it broke, like this one: ```python def test_sql_errors_logged_to_stderr(): runner = CliRunner(mix_stderr=False) result = runner.invoke(cli, [""--get"", ""/_memory.json?sql=select+blah""]) assert result.exit_code == 1 assert ""sql = 'select blah', params = {}: no such column: blah\n"" in result.stderr ``` It ended up with no `_memory` database because it turns out `datasette serve ...` passes `memory=False` without me realizing it. So I'm going to undo that hack and teach the fixture to do this instead: ```python db = ds.add_memory_database(""fixtures"") ds.remove_database(""_memory"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1354061440,https://api.github.com/repos/simonw/datasette/issues/1960,1354061440,IC_kwDOBm6k_c5QtVaA,9599,simonw,2022-12-16T01:45:38Z,2022-12-16T01:45:38Z,OWNER,"I'm going to do `test_table_html.py` next. Currently: 68 passed in 17.20s Will this speed it up?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1354062939,https://api.github.com/repos/simonw/datasette/issues/1960,1354062939,IC_kwDOBm6k_c5QtVxb,22429695,codecov[bot],2022-12-16T01:48:06Z,2022-12-17T21:40:43Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.19**% // Head: **92.05**% // Decreases project coverage by **`-0.13%`** :warning: > Coverage data is based on head [(`770879a`)](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`0b68996`)](https://codecov.io/gh/simonw/datasette/commit/0b68996cc511b3a801f0cd0157bd66332d75f46f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 100.00% of modified lines in pull request are covered. > :exclamation: Current head 770879a differs from pull request most recent head f42bca8. Consider uploading reports for the commit f42bca8 to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1960 +/- ## ========================================== - Coverage 92.19% 92.05% -0.14% ========================================== Files 38 38 Lines 5521 5527 +6 ========================================== - Hits 5090 5088 -2 - Misses 431 439 +8 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.13% <100.00%> (-0.57%)` | :arrow_down: | | [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.83% <100.00%> (+0.24%)` | :arrow_up: | | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: | | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.57% <0.00%> (-0.61%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1354072344,https://api.github.com/repos/simonw/datasette/issues/1960,1354072344,IC_kwDOBm6k_c5QtYEY,9599,simonw,2022-12-16T02:00:07Z,2022-12-16T02:00:07Z,OWNER,"It did NOT speed it up: 68 passed in 26.26s","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1354148139,https://api.github.com/repos/simonw/datasette/issues/1960,1354148139,IC_kwDOBm6k_c5Qtqkr,9599,simonw,2022-12-16T03:32:52Z,2022-12-16T03:32:52Z,OWNER,"Got that done to: 68 passed in 14.92s By implementing my own global variable - since `pytest` won't use a global fixture for me, I decided to do it for myself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354160286,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1354160286,IC_kwDOCGYnMM5Qttie,9020979,hydrosquall,2022-12-16T03:51:19Z,2022-12-16T03:52:13Z,NONE,"Hi @ryascott, thanks for sharing this! How did you upgrade your sqlite3 version? I'm running into this issue (also on an m1) with `Python ==3.10.7 sqlite3.sqlite_version==3.37.0 sqlite-utils==3.30` Unfortunately, `3.10.8` isn't listed in `pyenv` so I'm unable to install it. For me, the trigger is trying to use the `add-foreign-key` command on its own: ```bash sqlite-utils add-foreign-key library.db book_creators creator_id creators id ``` Some stackoverflow searching suggests that brew installing sqlite may fix it ( https://stackoverflow.com/questions/26345972/how-do-i-upgrade-the-sqlite-version-used-by-pythons-sqlite3-module-on-mac ), but I don't want to risk breaking the version of sqlite used by some other system, I'd only like to upgrade sqlite3 inside my current virtual environment. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354192168,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1354192168,IC_kwDOCGYnMM5Qt1Uo,9020979,hydrosquall,2022-12-16T04:35:30Z,2022-12-16T04:35:38Z,NONE,"A related historical problem: https://github.com/tekartik/sqflite/issues/525#issuecomment-714500720 I wonder if the version of Sqlite or Python for Intel chip have defensive mode disabled by default, whereas M1 chips versions have it enabled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/datasette/pull/1960#issuecomment-1354269873,https://api.github.com/repos/simonw/datasette/issues/1960,1354269873,IC_kwDOBm6k_c5QuISx,9599,simonw,2022-12-16T06:11:43Z,2022-12-16T06:11:43Z,OWNER,"This is quite fast: ``` % pytest -m ds_client -n auto ================================================================================== test session starts ================================================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict gw0 [291] / gw1 [291] / gw2 [291] / gw3 [291] / gw4 [291] / gw5 [291] ................................................................................................................................................................................. [ 60%] .................................................................................................................. [100%] ================================================================================== 291 passed in 6.30s ================================================================================== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355300217,https://api.github.com/repos/simonw/datasette/issues/1960,1355300217,IC_kwDOBm6k_c5QyD15,9599,simonw,2022-12-16T17:44:55Z,2022-12-16T17:44:55Z,OWNER,That's enough for this round. I'll get the tests passing and land this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355313058,https://api.github.com/repos/simonw/datasette/issues/1960,1355313058,IC_kwDOBm6k_c5QyG-i,9599,simonw,2022-12-16T17:53:17Z,2022-12-16T17:53:17Z,OWNER,"Got some surprising test failures here: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336626 ``` > assert response.json() == [{""id"": 1, ""name"": ""San Francisco""}] E AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}] E Full diff: E - [{'id': 1, 'name': 'San Francisco'}] E + {'error': 'Table not found: 𝐜𝐢𝐭𝐢𝐞𝐬', 'ok': False, 'status': 404, 'title': None} ``` A hunch: this failure suggests that maybe the fixtures tables were not correctly created when this test run. Maybe that can happen when `python -n auto` runs a bunch of separate processes and hence one of the tests randomly gets run in a fresh process and executes before the in-memory fixtures database has been fully populated. I tried adding `invoke_startup()` to the `ds_client()` fixture to see if that would fix this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355317369,https://api.github.com/repos/simonw/datasette/issues/1960,1355317369,IC_kwDOBm6k_c5QyIB5,9599,simonw,2022-12-16T17:57:14Z,2022-12-16T17:57:14Z,OWNER,"Opened a follow-up issue here: - #1962","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/issues/1959#issuecomment-1355317782,https://api.github.com/repos/simonw/datasette/issues/1959,1355317782,IC_kwDOBm6k_c5QyIIW,9599,simonw,2022-12-16T17:57:25Z,2022-12-16T17:57:25Z,OWNER,"Opened a follow-up issue here: - #1962","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499081664,Refactor test suite to use mostly `async def` tests, https://github.com/simonw/datasette/pull/1960#issuecomment-1355319541,https://api.github.com/repos/simonw/datasette/issues/1960,1355319541,IC_kwDOBm6k_c5QyIj1,9599,simonw,2022-12-16T17:58:24Z,2022-12-16T17:58:46Z,OWNER,"> I tried adding `invoke_startup()` to the `ds_client()` fixture to see if that would fix this. It did not: I'm still seeing those same failures. Frustrating: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336884 ====== 11 failed, 1252 passed, 1 skipped, 1 warning in 185.77s (0:03:05) =======","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355325426,https://api.github.com/repos/simonw/datasette/issues/1960,1355325426,IC_kwDOBm6k_c5QyJ_y,9599,simonw,2022-12-16T18:00:40Z,2022-12-16T18:00:40Z,OWNER,Many of the failing tests pass on my laptop but fail in CI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355445710,https://api.github.com/repos/simonw/datasette/issues/1960,1355445710,IC_kwDOBm6k_c5QynXO,9599,simonw,2022-12-16T19:06:49Z,2022-12-16T19:09:35Z,OWNER,"This would be much easier to debug if I could use Docker to run the GitHub Actions image directly on my own laptop. https://github.com/actions/runner-images/blob/releases/ubuntu22/20221212/images/linux/Ubuntu2204-Readme.md is the README for their most recent image. Not sure if there's an easy way to run it in Docker though. https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md is instructions for building them locally - looks fiddly though, involves https://www.packer.io/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355471341,https://api.github.com/repos/simonw/datasette/issues/1960,1355471341,IC_kwDOBm6k_c5Qytnt,9599,simonw,2022-12-16T19:23:33Z,2022-12-16T19:23:33Z,OWNER,"Trying this instead: ``` docker run -it ubuntu:22.04 /bin/bash ``` Then in that shell: ``` apt-get update apt-get install python3.11 python3.11-venv git -y cd /tmp git clone https://github.com/simonw/datasette cd datasette git checkout async-tests python3.11 -m venv venv pip install -e '.[test]' pytest ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355475671,https://api.github.com/repos/simonw/datasette/issues/1960,1355475671,IC_kwDOBm6k_c5QyurX,9599,simonw,2022-12-16T19:26:08Z,2022-12-16T19:26:08Z,OWNER,"Great news! The test failures I got running on my laptop (with that fresh Ubuntu Docker image) look like they match the failures I saw in CI: ``` ======== short test summary info ======== FAILED tests/test_table_api.py::test_paginate_tables_and_views[/fixtures/paginated_view.json-201-9] - assert 9 == 5 FAILED tests/test_table_api.py::test_sortable_columns_metadata - KeyError: 'error' FAILED tests/test_table_api.py::test_searchable_views[/fixtures/searchable_view_configured_by_metadata.json?_search=weasel-expected_rows0] - AssertionError: assert [[2, 'terry d...sel', 'puma']] == [[1, 'barry c...sel', 'puma']] FAILED tests/test_table_api.py::test_unit_filters - KeyError: 'distance' FAILED tests/test_table_api.py::test_custom_query_with_unicode_characters - AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}] FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...] FAILED tests/test_table_html.py::test_table_html_no_primary_key - AttributeError: 'NoneType' object has no attribute 'strip' FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['']] == [['1']] FAILED tests/test_table_html.py::test_view_html - assert upper_content is None FAILED tests/test_table_html.py::test_table_metadata - AssertionError: assert 'This <em&...t; is escaped' == 'simple_primary_key' FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id\xa0▼', 'name'] FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk\xa0▼', 'name'] FAILED tests/test_table_html.py::test_column_metadata - AttributeError: 'NoneType' object has no attribute 'findAll' ======== 13 failed, 1279 passed, 3 skipped, 57 warnings in 572.40s (0:09:32) ======== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355478743,https://api.github.com/repos/simonw/datasette/issues/1960,1355478743,IC_kwDOBm6k_c5QyvbX,9599,simonw,2022-12-16T19:27:12Z,2022-12-16T19:27:12Z,OWNER,"Bad news: they're definitely caused by tests that are subtly affected by other tests. This passes without errors: pytest -k test_paginate_tables_and_views","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355685828,https://api.github.com/repos/simonw/datasette/issues/1960,1355685828,IC_kwDOBm6k_c5Qzh_E,9599,simonw,2022-12-16T21:50:01Z,2022-12-16T21:50:01Z,OWNER,"Looks like that `@pytest.mark.ds_client` mark I've been using isn't necessary - I added that so I could easily run `pytest -m ds_client` to execute all tests that I had ported to the new feature, but actually this achieves the same thing: pytest -k ds_client So I'm going to remove the mark.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/issues/1962#issuecomment-1355691595,https://api.github.com/repos/simonw/datasette/issues/1962,1355691595,IC_kwDOBm6k_c5QzjZL,9599,simonw,2022-12-16T21:53:45Z,2022-12-16T21:55:29Z,OWNER,"Also need an alternative mechanism to this convenience for getting CSRF tokens before a POST: https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/datasette/utils/testing.py#L90-L103 One option would be adding that mechanism to `datasette.client.post(...)` - as a `_csrf_from=` parameter (with an underscore because it's mainly intended for use in tests, though perhaps that's a weird convention that I shouldn't introduce).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1500636982,"Alternative, async-friendly pattern for `make_app_client()` and similar - fully retire `TestClient`", https://github.com/simonw/datasette/pull/1960#issuecomment-1355823260,https://api.github.com/repos/simonw/datasette/issues/1960,1355823260,IC_kwDOBm6k_c5Q0Dic,9599,simonw,2022-12-16T23:36:07Z,2022-12-16T23:36:07Z,OWNER,"I ran `pytest --pdb -x` to drop into the debugger on the first failing test. ``` assert expected_rows == len(fetched) > assert expected_pages == count E assert 9 == 5 ``` That's in `test_paginate_tables_and_views()`. On a hunch, I checked the current settings: ``` (Pdb) ds_client.ds._settings {'default_page_size': 50, 'max_returned_rows': 100, 'max_insert_rows': 100, 'num_sql_threads': 1, 'sql_time_limit_ms': 200, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'allow_facet': True, 'allow_download': True, 'allow_signed_tokens': True, 'max_signed_tokens_ttl': 0, 'suggest_facets': True, 'default_cache_ttl': 5, 'cache_size_kb': 0, 'allow_csv_stream': True, 'max_csv_mb': 100, 'truncate_cells_html': 2048, 'force_https_urls': False, 'template_debug': False, 'trace_debug': False, 'base_url': '/'} ``` Looks like something changed `default_page_size` to 50 and forgot to change it back!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client,