{"html_url": "https://github.com/simonw/datasette/issues/1958#issuecomment-1353977605", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1958", "id": 1353977605, "node_id": "IC_kwDOBm6k_c5QtA8F", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T00:38:23Z", "updated_at": "2022-12-16T00:38:23Z", "author_association": "OWNER", "body": "Confirmed, I just got the same result:\r\n```\r\n- % docker run datasetteproject/datasette pip install datasette-upload-csvs\r\n~ % docker commit $(docker ps -lq) datasette-with-plugins\r\nsha256:8cde4a6357b9221d6f9e15887a314f2b4d9d1b87b517764d207ccbaec7c0a69f\r\n~ % docker run -p 8001:8001 -v $(pwd):/mnt datasette-with-plugins datasette --root -p 8001 -h 0.0.0.0\r\nINFO: Started server process [1]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nINFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)\r\n\r\n\r\n\r\n^CINFO: Shutting down\r\nINFO: Waiting for application shutdown.\r\nINFO: Application shutdown complete.\r\nINFO: Finished server process [1]\r\nhttp://0.0.0.0:8001/-/auth-token?token=4bd70fdbca215ea55c874eaf889adf8c09f2a00231f7e5e6d0470f3176407a98\r\n```\r\nNote how the auth-token URL is only displayed after you hit `Ctrl+C`!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1497909798, "label": "datasette --root running in Docker doesn't reliably show the magic URL"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1958#issuecomment-1354008688", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1958", "id": 1354008688, "node_id": "IC_kwDOBm6k_c5QtIhw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T00:45:40Z", "updated_at": "2022-12-16T00:45:40Z", "author_association": "OWNER", "body": "The fix may just be to switch to `click.echo()`. https://click.palletsprojects.com/en/8.1.x/api/#click.echo", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1497909798, "label": "datasette --root running in Docker doesn't reliably show the magic URL"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1958#issuecomment-1354019543", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1958", "id": 1354019543, "node_id": "IC_kwDOBm6k_c5QtLLX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T00:54:11Z", "updated_at": "2022-12-16T00:54:11Z", "author_association": "OWNER", "body": "To test the fix, I did a fresh checkout of `simonw/datasette` and edited `Dockerfile` to look like this:\r\n```dockerfile\r\nFROM python:3.11.0-slim-bullseye as build\r\n\r\nRUN apt-get update && \\\r\n apt-get install -y --no-install-recommends libsqlite3-mod-spatialite && \\\r\n apt clean && \\\r\n rm -rf /var/lib/apt && \\\r\n rm -rf /var/lib/dpkg/info/*\r\n\r\nRUN pip install https://github.com/simonw/datasette/archive/refs/heads/main.zip && \\\r\n find /usr/local/lib -name '__pycache__' | xargs rm -r && \\\r\n rm -rf /root/.cache/pip\r\n\r\nEXPOSE 8001\r\nCMD [\"datasette\"]\r\n```\r\nThen I built it like this:\r\n```\r\ndocker build -f Dockerfile -t datasette-pre-click .\r\n```\r\nAnd ran like this:\r\n```\r\ndocker run -p 8001:8001 -v $(pwd):/mnt datasette-pre-click datasette --root -p 8001 -h 0.0.0.0\r\n```\r\nThis exhibited the same problem.\r\n\r\nThen I pushed a changed to branch and changed the line to:\r\n```\r\nRUN pip install https://github.com/simonw/datasette/archive/refs/heads/click-echo-root.zip && \\\r\n```\r\nRan this:\r\n```\r\ndocker build -f Dockerfile -t datasette-post-click .\r\n```\r\nAnd this:\r\n```\r\ndocker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0\r\n```\r\nIt fixed the problem!\r\n```\r\ndatasette % docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0\r\nhttp://0.0.0.0:8001/-/auth-token?token=6542dcf5c8f34f8d13f4af9ce728359c602469efb54029098562bd06c87ad26d\r\nINFO: Started server process [1]\r\nINFO: Waiting for application startup.\r\nINFO: Application startup complete.\r\nINFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1497909798, "label": "datasette --root running in Docker doesn't reliably show the magic URL"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1958#issuecomment-1354023960", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1958", "id": 1354023960, "node_id": "IC_kwDOBm6k_c5QtMQY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T00:58:14Z", "updated_at": "2022-12-16T00:58:19Z", "author_association": "OWNER", "body": "This is in the `0.63.x` branch now, ready to go out in a bug fix release.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1497909798, "label": "datasette --root running in Docker doesn't reliably show the magic URL"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1958#issuecomment-1354025319", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1958", "id": 1354025319, "node_id": "IC_kwDOBm6k_c5QtMln", "user": {"value": 11729897, "label": "davidhaley"}, "created_at": "2022-12-16T00:59:12Z", "updated_at": "2022-12-16T00:59:12Z", "author_association": "NONE", "body": "Awesome. Thank you @simonw.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1497909798, "label": "datasette --root running in Docker doesn't reliably show the magic URL"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1354036967", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1354036967, "node_id": "IC_kwDOBm6k_c5QtPbn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T01:10:12Z", "updated_at": "2022-12-16T01:10:12Z", "author_association": "OWNER", "body": "If it does turn out that I can't use `scope=\"session\"` on this fixture it might not actually be a showstopper: I can take advantage of the fact that `memory_name=\"...\"` databases stay present in memory for the duration of the process, so I could have it such that each test that uses the `ds_client` fixture DOES construct a fresh `Datasette` instance, but doesn't need to populate the database since they can re-use the in-memory database from the previous object.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1354046627", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1354046627, "node_id": "IC_kwDOBm6k_c5QtRyj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T01:23:21Z", "updated_at": "2022-12-16T01:23:21Z", "author_association": "OWNER", "body": "This does seem to help:\r\n```diff\r\ndiff --git a/tests/conftest.py b/tests/conftest.py\r\nindex 1306c407..af9c7696 100644\r\n--- a/tests/conftest.py\r\n+++ b/tests/conftest.py\r\n@@ -25,12 +25,7 @@ UNDOCUMENTED_PERMISSIONS = {\r\n }\r\n \r\n \r\n-@pytest.fixture(scope=\"session\")\r\n-def event_loop():\r\n- return asyncio.get_event_loop()\r\n-\r\n-\r\n-@pytest_asyncio.fixture(scope=\"session\")\r\n+@pytest_asyncio.fixture\r\n async def ds_client():\r\n from datasette.app import Datasette\r\n from .fixtures import METADATA, PLUGINS_DIR\r\n@@ -53,10 +48,11 @@ async def ds_client():\r\n db = ds.add_memory_database(\"fixtures\")\r\n \r\n def prepare(conn):\r\n- conn.executescript(TABLES)\r\n- for sql, params in TABLE_PARAMETERIZED_SQL:\r\n- with conn:\r\n- conn.execute(sql, params)\r\n+ if not conn.execute(\"select count(*) from sqlite_master\").fetchone()[0]:\r\n+ conn.executescript(TABLES)\r\n+ for sql, params in TABLE_PARAMETERIZED_SQL:\r\n+ with conn:\r\n+ conn.execute(sql, params)\r\n \r\n await db.execute_write_fn(prepare)\r\n return ds.client\r\ndiff --git a/tests/plugins/my_plugin_2.py b/tests/plugins/my_plugin_2.py\r\nindex 4f7bf08c..d588342c 100644\r\n--- a/tests/plugins/my_plugin_2.py\r\n+++ b/tests/plugins/my_plugin_2.py\r\n@@ -117,7 +117,12 @@ def actor_from_request(datasette, request):\r\n def permission_allowed(datasette, actor, action):\r\n # Testing asyncio version of permission_allowed\r\n async def inner():\r\n- assert 2 == (await datasette.get_database().execute(\"select 1 + 1\")).first()[0]\r\n+ assert (\r\n+ 2\r\n+ == (\r\n+ await datasette.get_database(\"_internal\").execute(\"select 1 + 1\")\r\n+ ).first()[0]\r\n+ )\r\n if action == \"this_is_allowed_async\":\r\n return True\r\n elif action == \"this_is_denied_async\":\r\n```\r\n`pytest -m ds_client` now passes 134 tests.\r\n\r\nNeed to get `pytest -n auto` passing too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1354053151", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1354053151, "node_id": "IC_kwDOBm6k_c5QtTYf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T01:33:22Z", "updated_at": "2022-12-16T01:33:22Z", "author_association": "OWNER", "body": "The thing with `Datasette(memory=False)` is tripping me up.\r\n\r\nThe problem is that the tests written against `app_client` - which I want to replace - all assume that there is no `_memory` database, because when you start Datasette with at least one database file it doesn't enable `_memory` unless you explicitly tell it to.\r\n\r\nBut the new `ds_client` fixture works by creating a named in-memory database called `fixtures`, which it does with a call to `ds.add_memory_database(\"fixtures\")` after the object has been instantiated.\r\n\r\nThis results in a datasette instance that DOES have a `_memory` database, when we didn't want one.\r\n\r\nMy initial solution attempt was a huge hack - I decided that if you pass `memory=False` to the `Datasette` constructor it should mean \"don't add a `_memory` database even though I didn't pass any files\". I set a the default `memory` argument to `None`.\r\n\r\nThis is weird and surprising (`memory=False` no does something different from `memory=None`?) and I found other tests that it broke, like this one:\r\n\r\n```python\r\ndef test_sql_errors_logged_to_stderr():\r\n runner = CliRunner(mix_stderr=False)\r\n result = runner.invoke(cli, [\"--get\", \"/_memory.json?sql=select+blah\"])\r\n assert result.exit_code == 1\r\n assert \"sql = 'select blah', params = {}: no such column: blah\\n\" in result.stderr\r\n```\r\nIt ended up with no `_memory` database because it turns out `datasette serve ...` passes `memory=False` without me realizing it.\r\n\r\nSo I'm going to undo that hack and teach the fixture to do this instead:\r\n\r\n```python\r\ndb = ds.add_memory_database(\"fixtures\")\r\nds.remove_database(\"_memory\")\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1354061440", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1354061440, "node_id": "IC_kwDOBm6k_c5QtVaA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T01:45:38Z", "updated_at": "2022-12-16T01:45:38Z", "author_association": "OWNER", "body": "I'm going to do `test_table_html.py` next.\r\n\r\nCurrently: 68 passed in 17.20s\r\n\r\nWill this speed it up?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1354062939", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1354062939, "node_id": "IC_kwDOBm6k_c5QtVxb", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-12-16T01:48:06Z", "updated_at": "2022-12-17T21:40:43Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nBase: **92.19**% // Head: **92.05**% // Decreases project coverage by **`-0.13%`** :warning:\n> Coverage data is based on head [(`770879a`)](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`0b68996`)](https://codecov.io/gh/simonw/datasette/commit/0b68996cc511b3a801f0cd0157bd66332d75f46f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> Patch coverage: 100.00% of modified lines in pull request are covered.\n\n> :exclamation: Current head 770879a differs from pull request most recent head f42bca8. Consider uploading reports for the commit f42bca8 to get more accurate results\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #1960 +/- ##\n==========================================\n- Coverage 92.19% 92.05% -0.14% \n==========================================\n Files 38 38 \n Lines 5521 5527 +6 \n==========================================\n- Hits 5090 5088 -2 \n- Misses 431 439 +8 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.13% <100.00%> (-0.57%)` | :arrow_down: |\n| [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.83% <100.00%> (+0.24%)` | :arrow_up: |\n| [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: |\n| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.57% <0.00%> (-0.61%)` | :arrow_down: |\n\nHelp us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n
\n\n[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1354072344", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1354072344, "node_id": "IC_kwDOBm6k_c5QtYEY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T02:00:07Z", "updated_at": "2022-12-16T02:00:07Z", "author_association": "OWNER", "body": "It did NOT speed it up:\r\n\r\n68 passed in 26.26s", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1354148139", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1354148139, "node_id": "IC_kwDOBm6k_c5Qtqkr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T03:32:52Z", "updated_at": "2022-12-16T03:32:52Z", "author_association": "OWNER", "body": "Got that done to: 68 passed in 14.92s\r\n\r\nBy implementing my own global variable - since `pytest` won't use a global fixture for me, I decided to do it for myself.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354160286", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1354160286, "node_id": "IC_kwDOCGYnMM5Qttie", "user": {"value": 9020979, "label": "hydrosquall"}, "created_at": "2022-12-16T03:51:19Z", "updated_at": "2022-12-16T03:52:13Z", "author_association": "NONE", "body": "Hi @ryascott, thanks for sharing this! How did you upgrade your sqlite3 version? I'm running into this issue (also on an m1) with \r\n\r\n`Python ==3.10.7 sqlite3.sqlite_version==3.37.0 sqlite-utils==3.30`\r\n\r\nUnfortunately, `3.10.8` isn't listed in `pyenv` so I'm unable to install it.\r\n\r\nFor me, the trigger is trying to use the `add-foreign-key` command on its own:\r\n\r\n```bash\r\nsqlite-utils add-foreign-key library.db book_creators creator_id creators id\r\n```\r\n\r\nSome stackoverflow searching suggests that brew installing sqlite may fix it ( https://stackoverflow.com/questions/26345972/how-do-i-upgrade-the-sqlite-version-used-by-pythons-sqlite3-module-on-mac ), but I don't want to risk breaking the version of sqlite used by some other system, I'd only like to upgrade sqlite3 inside my current virtual environment.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354192168", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1354192168, "node_id": "IC_kwDOCGYnMM5Qt1Uo", "user": {"value": 9020979, "label": "hydrosquall"}, "created_at": "2022-12-16T04:35:30Z", "updated_at": "2022-12-16T04:35:38Z", "author_association": "NONE", "body": "A related historical problem:\r\n\r\nhttps://github.com/tekartik/sqflite/issues/525#issuecomment-714500720\r\n\r\nI wonder if the version of Sqlite or Python for Intel chip have defensive mode disabled by default, whereas M1 chips versions have it enabled.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1354269873", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1354269873, "node_id": "IC_kwDOBm6k_c5QuISx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T06:11:43Z", "updated_at": "2022-12-16T06:11:43Z", "author_association": "OWNER", "body": "This is quite fast:\r\n```\r\n% pytest -m ds_client -n auto\r\n================================================================================== test session starts ==================================================================================\r\nplatform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0\r\nSQLite: 3.39.4\r\nrootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini\r\nplugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0\r\nasyncio: mode=strict\r\ngw0 [291] / gw1 [291] / gw2 [291] / gw3 [291] / gw4 [291] / gw5 [291]\r\n................................................................................................................................................................................. [ 60%]\r\n.................................................................................................................. [100%]\r\n================================================================================== 291 passed in 6.30s ==================================================================================\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1355300217", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1355300217, "node_id": "IC_kwDOBm6k_c5QyD15", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T17:44:55Z", "updated_at": "2022-12-16T17:44:55Z", "author_association": "OWNER", "body": "That's enough for this round. I'll get the tests passing and land this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1355313058", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1355313058, "node_id": "IC_kwDOBm6k_c5QyG-i", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T17:53:17Z", "updated_at": "2022-12-16T17:53:17Z", "author_association": "OWNER", "body": "Got some surprising test failures here: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336626\r\n```\r\n> assert response.json() == [{\"id\": 1, \"name\": \"San Francisco\"}]\r\nE AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}]\r\nE Full diff:\r\nE - [{'id': 1, 'name': 'San Francisco'}]\r\nE + {'error': 'Table not found: \ud835\udc1c\ud835\udc22\ud835\udc2d\ud835\udc22\ud835\udc1e\ud835\udc2c', 'ok': False, 'status': 404, 'title': None}\r\n```\r\nA hunch: this failure suggests that maybe the fixtures tables were not correctly created when this test run. Maybe that can happen when `python -n auto` runs a bunch of separate processes and hence one of the tests randomly gets run in a fresh process and executes before the in-memory fixtures database has been fully populated.\r\n\r\nI tried adding `invoke_startup()` to the `ds_client()` fixture to see if that would fix this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1355317369", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1355317369, "node_id": "IC_kwDOBm6k_c5QyIB5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T17:57:14Z", "updated_at": "2022-12-16T17:57:14Z", "author_association": "OWNER", "body": "Opened a follow-up issue here:\r\n- #1962", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1959#issuecomment-1355317782", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1959", "id": 1355317782, "node_id": "IC_kwDOBm6k_c5QyIIW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T17:57:25Z", "updated_at": "2022-12-16T17:57:25Z", "author_association": "OWNER", "body": "Opened a follow-up issue here:\r\n- #1962", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499081664, "label": "Refactor test suite to use mostly `async def` tests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1355319541", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1355319541, "node_id": "IC_kwDOBm6k_c5QyIj1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T17:58:24Z", "updated_at": "2022-12-16T17:58:46Z", "author_association": "OWNER", "body": "> I tried adding `invoke_startup()` to the `ds_client()` fixture to see if that would fix this.\r\n\r\nIt did not: I'm still seeing those same failures. Frustrating: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336884\r\n\r\n ====== 11 failed, 1252 passed, 1 skipped, 1 warning in 185.77s (0:03:05) =======", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1355325426", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1355325426, "node_id": "IC_kwDOBm6k_c5QyJ_y", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T18:00:40Z", "updated_at": "2022-12-16T18:00:40Z", "author_association": "OWNER", "body": "Many of the failing tests pass on my laptop but fail in CI.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1355445710", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1355445710, "node_id": "IC_kwDOBm6k_c5QynXO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T19:06:49Z", "updated_at": "2022-12-16T19:09:35Z", "author_association": "OWNER", "body": "This would be much easier to debug if I could use Docker to run the GitHub Actions image directly on my own laptop.\r\n\r\nhttps://github.com/actions/runner-images/blob/releases/ubuntu22/20221212/images/linux/Ubuntu2204-Readme.md is the README for their most recent image. Not sure if there's an easy way to run it in Docker though.\r\n\r\nhttps://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md is instructions for building them locally - looks fiddly though, involves https://www.packer.io/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1355471341", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1355471341, "node_id": "IC_kwDOBm6k_c5Qytnt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T19:23:33Z", "updated_at": "2022-12-16T19:23:33Z", "author_association": "OWNER", "body": "Trying this instead:\r\n```\r\ndocker run -it ubuntu:22.04 /bin/bash\r\n```\r\nThen in that shell:\r\n```\r\napt-get update\r\napt-get install python3.11 python3.11-venv git -y\r\ncd /tmp\r\ngit clone https://github.com/simonw/datasette\r\ncd datasette\r\ngit checkout async-tests\r\npython3.11 -m venv venv\r\npip install -e '.[test]'\r\npytest\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1355475671", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1355475671, "node_id": "IC_kwDOBm6k_c5QyurX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T19:26:08Z", "updated_at": "2022-12-16T19:26:08Z", "author_association": "OWNER", "body": "Great news! The test failures I got running on my laptop (with that fresh Ubuntu Docker image) look like they match the failures I saw in CI:\r\n\r\n```\r\n======== short test summary info ========\r\nFAILED tests/test_table_api.py::test_paginate_tables_and_views[/fixtures/paginated_view.json-201-9] - assert 9 == 5\r\nFAILED tests/test_table_api.py::test_sortable_columns_metadata - KeyError: 'error'\r\nFAILED tests/test_table_api.py::test_searchable_views[/fixtures/searchable_view_configured_by_metadata.json?_search=weasel-expected_rows0] - AssertionError: assert [[2, 'terry d...sel', 'puma']] == [[1, 'barry c...sel', 'puma']]\r\nFAILED tests/test_table_api.py::test_unit_filters - KeyError: 'distance'\r\nFAILED tests/test_table_api.py::test_custom_query_with_unicode_characters - AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}]\r\nFAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...]\r\nFAILED tests/test_table_html.py::test_table_html_no_primary_key - AttributeError: 'NoneType' object has no attribute 'strip'\r\nFAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['']] == [['1']]\r\nFAILED tests/test_table_html.py::test_view_html - assert upper_content is None\r\nFAILED tests/test_table_html.py::test_table_metadata - AssertionError: assert 'This <em&...t; is escaped' == 'simple_primary_key'\r\nFAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\\xa0\u25bc'] == ['id\\xa0\u25bc', 'name']\r\nFAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\\xa0\u25b2', 'name'] == ['pk\\xa0\u25bc', 'name']\r\nFAILED tests/test_table_html.py::test_column_metadata - AttributeError: 'NoneType' object has no attribute 'findAll'\r\n======== 13 failed, 1279 passed, 3 skipped, 57 warnings in 572.40s (0:09:32) ========\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1355478743", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1355478743, "node_id": "IC_kwDOBm6k_c5QyvbX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T19:27:12Z", "updated_at": "2022-12-16T19:27:12Z", "author_association": "OWNER", "body": "Bad news: they're definitely caused by tests that are subtly affected by other tests.\r\n\r\nThis passes without errors:\r\n\r\n pytest -k test_paginate_tables_and_views", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1355685828", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1355685828, "node_id": "IC_kwDOBm6k_c5Qzh_E", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T21:50:01Z", "updated_at": "2022-12-16T21:50:01Z", "author_association": "OWNER", "body": "Looks like that `@pytest.mark.ds_client` mark I've been using isn't necessary - I added that so I could easily run `pytest -m ds_client` to execute all tests that I had ported to the new feature, but actually this achieves the same thing:\r\n\r\n pytest -k ds_client\r\n\r\nSo I'm going to remove the mark.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1962#issuecomment-1355691595", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1962", "id": 1355691595, "node_id": "IC_kwDOBm6k_c5QzjZL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T21:53:45Z", "updated_at": "2022-12-16T21:55:29Z", "author_association": "OWNER", "body": "Also need an alternative mechanism to this convenience for getting CSRF tokens before a POST:\r\n\r\nhttps://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/datasette/utils/testing.py#L90-L103\r\n\r\nOne option would be adding that mechanism to `datasette.client.post(...)` - as a `_csrf_from=` parameter (with an underscore because it's mainly intended for use in tests, though perhaps that's a weird convention that I shouldn't introduce).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1500636982, "label": "Alternative, async-friendly pattern for `make_app_client()` and similar - fully retire `TestClient`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1960#issuecomment-1355823260", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1960", "id": 1355823260, "node_id": "IC_kwDOBm6k_c5Q0Dic", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-12-16T23:36:07Z", "updated_at": "2022-12-16T23:36:07Z", "author_association": "OWNER", "body": "I ran `pytest --pdb -x` to drop into the debugger on the first failing test.\r\n\r\n```\r\n assert expected_rows == len(fetched)\r\n> assert expected_pages == count\r\nE assert 9 == 5\r\n```\r\nThat's in `test_paginate_tables_and_views()`.\r\n\r\nOn a hunch, I checked the current settings:\r\n\r\n```\r\n(Pdb) ds_client.ds._settings\r\n{'default_page_size': 50, 'max_returned_rows': 100, 'max_insert_rows': 100, 'num_sql_threads': 1, 'sql_time_limit_ms': 200, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'allow_facet': True, 'allow_download': True, 'allow_signed_tokens': True, 'max_signed_tokens_ttl': 0, 'suggest_facets': True, 'default_cache_ttl': 5, 'cache_size_kb': 0, 'allow_csv_stream': True, 'max_csv_mb': 100, 'truncate_cells_html': 2048, 'force_https_urls': False, 'template_debug': False, 'trace_debug': False, 'base_url': '/'}\r\n```\r\nLooks like something changed `default_page_size` to 50 and forgot to change it back!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1499150951, "label": "Port as many tests as possible to async def tests against ds_client"}, "performed_via_github_app": null}