{"html_url": "https://github.com/simonw/datasette/pull/1537#issuecomment-981631026", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1537", "id": 981631026, "node_id": "IC_kwDOBm6k_c46goAy", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-11-29T13:23:20Z", "updated_at": "2021-11-29T13:23:20Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#1537](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fcfaec1) into [main](https://codecov.io/gh/simonw/datasette/commit/48f11998b73350057b74fe6ab464d4ac3071637c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (48f1199) will **decrease** coverage by `0.06%`.\n> The diff coverage is `n/a`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1537/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@            Coverage Diff             @@\n##             main    #1537      +/-   ##\n==========================================\n- Coverage   91.90%   91.83%   -0.07%     \n==========================================\n  Files          34       34              \n  Lines        4434     4434              \n==========================================\n- Hits         4075     4072       -3     \n- Misses        359      362       +3     \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1537/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.42% <0.00%> (-1.79%)` | :arrow_down: |\n| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1537/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <0.00%> (-0.75%)` | :arrow_down: |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute <relative> (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [48f1199...fcfaec1](https://codecov.io/gh/simonw/datasette/pull/1537?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066023866, "label": "Update aiofiles requirement from <0.8,>=0.4 to >=0.4,<0.9"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1538#issuecomment-981849494", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1538", "id": 981849494, "node_id": "IC_kwDOBm6k_c46hdWW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T17:23:52Z", "updated_at": "2021-11-29T17:23:52Z", "author_association": "OWNER", "body": "Just trying to use `git_history_file` produces this error:\r\n\r\n> `TypeError: Attempted to convert a callback into a command twice.`\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066288689, "label": "Research pattern for re-registering existing Click tools with register_commands"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1538#issuecomment-981852280", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1538", "id": 981852280, "node_id": "IC_kwDOBm6k_c46heB4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T17:27:12Z", "updated_at": "2021-11-29T17:27:12Z", "author_association": "OWNER", "body": "Thanks to https://stackoverflow.com/a/45514541/6083 I found the right pattern:\r\n```python\r\nfrom datasette import hookimpl\r\nfrom git_history.cli import cli as git_history_cli\r\n\r\n@hookimpl\r\ndef register_commands(cli):\r\n    cli.add_command(git_history_cli, name=\"git-history\")\r\n```\r\nI think this is a little bit too obscure to add to the Datasette documentation - I'll turn it into a TIL instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066288689, "label": "Research pattern for re-registering existing Click tools with register_commands"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1538#issuecomment-981856895", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1538", "id": 981856895, "node_id": "IC_kwDOBm6k_c46hfJ_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T17:32:44Z", "updated_at": "2021-11-29T17:32:44Z", "author_association": "OWNER", "body": "TIL: https://til.simonwillison.net/datasette/reuse-click-for-register-commands", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066288689, "label": "Research pattern for re-registering existing Click tools with register_commands"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1532#issuecomment-981966693", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1532", "id": 981966693, "node_id": "IC_kwDOBm6k_c46h59l", "user": {"value": 30934, "label": "20after4"}, "created_at": "2021-11-29T19:56:52Z", "updated_at": "2021-11-29T19:56:52Z", "author_association": "NONE", "body": "FWIW I've written some web components that consume the json api and I think it's a really nice way to work with datasette. I like the combination with datasette+sqlite as a back-end feeding data to a front-end that's entirely javascript + html.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1065429936, "label": "Use datasette-table Web Component to guide the design of the JSON API for 1.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1304#issuecomment-981980048", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1304", "id": 981980048, "node_id": "IC_kwDOBm6k_c46h9OQ", "user": {"value": 30934, "label": "20after4"}, "created_at": "2021-11-29T20:13:53Z", "updated_at": "2021-11-29T20:14:11Z", "author_association": "NONE", "body": "There isn't any way to do this with sqlite as far as I know.  The only option is to insert the right number of ? placeholders into the sql template and then provide an array of values.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 863884805, "label": "Document how to send multiple values for \"Named parameters\" "}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-981997973", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 981997973, "node_id": "IC_kwDOCGYnMM46iBmV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T20:33:52Z", "updated_at": "2021-11-29T20:33:52Z", "author_association": "OWNER", "body": "From that page:\r\n\r\n> If you try to open a database containing the STRICT keyword in an earlier version of SQLite, it will not recognize the keyword and will report an error (except as noted below.\r\n>\r\n> [...]\r\n>\r\n> Because of a quirk in the SQL language parser, versions of SQLite prior to 3.37.0 can still read and write STRICT tables if they set \"PRAGMA writable_schema=ON\" immediately after opening the database file, prior to doing anything else that requires knowing the schema.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-981999025", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 981999025, "node_id": "IC_kwDOCGYnMM46iB2x", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T20:34:38Z", "updated_at": "2021-11-29T20:35:58Z", "author_association": "OWNER", "body": "I'm going to build my own `pysqlite3` wheel with the latest SQLite to try this out, following the instructions on https://github.com/coleifer/pysqlite3", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982006544", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 982006544, "node_id": "IC_kwDOCGYnMM46iDsQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T20:44:37Z", "updated_at": "2021-11-29T20:48:43Z", "author_association": "OWNER", "body": "This worked:\r\n```\r\ncd /tmp\r\nmkdir sqlite-3.37\r\ncd sqlite-3.37\r\nwget 'https://www.sqlite.org/2021/sqlite-amalgamation-3370000.zip'\r\nunzip sqlite-amalgamation-3370000.zip\r\ngit clone https://github.com/coleifer/pysqlite3/\r\ncp sqlite-amalgamation-3370000/sqlite3.[ch] pysqlite3\r\ncd pysqlite3\r\npython3 setup.py build_static build bdist_wheel\r\n```\r\nThis gave me a file here:\r\n\r\n```\r\npysqlite3 % ls -l dist  \r\ntotal 1872\r\n-rw-r--r--  1 simon  wheel  956557 Nov 29 12:38 pysqlite3-0.4.6-cp39-cp39-macosx_10_15_x86_64.whl\r\n```\r\nThat wheel only works when installed for Python 3.9 (it failed to install in a Python 3.10 virtual environment) - but `pip install /tmp/sqlite-3.37/pysqlite3/dist/pysqlite3-0.4.6-cp39-cp39-macosx_10_15_x86_64.whl` gave me a working `pysqlite3` - and the following worked:\r\n\r\n```pycon\r\n>>> import pysqlite3\r\n>>> pysqlite3.connect(\":memory:\").execute(\"select sqlite_version()\").fetchall()\r\n[('3.37.0',)]\r\n```\r\nAnd if I install `sqlite-utils` in the same virtual environment this works:\r\n```\r\n% sqlite-utils memory 'select sqlite_version()'\r\n[{\"sqlite_version()\": \"3.37.0\"}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 982014776, "node_id": "IC_kwDOCGYnMM46iFs4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T20:55:19Z", "updated_at": "2021-11-29T20:55:19Z", "author_association": "OWNER", "body": "There are a few places that the `strict=True` option could go:\r\n\r\n- `table.create()` and `table.create_table_sql()`\r\n- The `Database()` constructor, to turn it on for all created tables\r\n- The `.insert()` / `.insert_all()` etc family of methods that can implicitly create tables\r\n\r\nI'll definitely implement the first one, and likely the second one too. I'm on the fence with regards to the third one.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982016594", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 982016594, "node_id": "IC_kwDOCGYnMM46iGJS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T20:57:42Z", "updated_at": "2021-11-29T20:57:42Z", "author_association": "OWNER", "body": "What should happen if you attempt to use `strict=True` against a SQLite version prior to 3.37.0?\r\n\r\nAn obvious error would be best... but how about silently ignoring it on older versions instead? That would match how we handle `deterministic=True` for registering functions:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/126703706ea153f63e6134ad14e5712e4bbcb8ae/sqlite_utils/db.py#L372-L380\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/93c7fd9868fed3193a1732b39bfac539e5812b0b/tests/test_register_function.py#L34-L37\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982017994", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 982017994, "node_id": "IC_kwDOCGYnMM46iGfK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T20:59:37Z", "updated_at": "2021-11-29T20:59:37Z", "author_association": "OWNER", "body": "I'm leaning towards silently ignore if SQLite version doesn't support it. That means that the first time you attempt to use `strict=True` we will need to run a test against the database connection to see what version of SQLite it uses, then cache the result to avoid making the same call again for the same connection.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982018304", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 982018304, "node_id": "IC_kwDOCGYnMM46iGkA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T21:00:02Z", "updated_at": "2021-11-29T21:00:02Z", "author_association": "OWNER", "body": "Is there a need for an introspection function for telling if a SQLite table is in strict mode or not? How would that work?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982020757", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 982020757, "node_id": "IC_kwDOCGYnMM46iHKV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T21:03:34Z", "updated_at": "2021-11-29T21:03:34Z", "author_association": "OWNER", "body": "From the STRICT docs:\r\n\r\n> The SQLite parser accepts a comma-separated list of table options after the final close parenthesis in a CREATE TABLE statement. As of this writing (2021-08-23) only two options are recognized:\r\n> \r\n> -   STRICT\r\n> -   [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html)\r\n\r\nSo I think I need to read the `CREATE TABLE` statement from the `sqlite_master` table, split on the last `)`, split those tokens on `,` and see if `create` is in there (case insensitive).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982026918", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 982026918, "node_id": "IC_kwDOCGYnMM46iIqm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T21:11:42Z", "updated_at": "2021-11-29T21:16:31Z", "author_association": "OWNER", "body": "Made myself a test strict table like so:\r\n\r\n```pycon\r\n>>> import pysqlite3\r\n>>> conn = pysqlite3.connect(\"/tmp/strict-table.db\")\r\n>>> conn.execute(\"create table foo (id integer primary key, name text, weight real) strict\")\r\n<pysqlite3.dbapi2.Cursor object at 0x104317340>\r\n>>> cursor = conn.cursor()\r\n>>> with conn:\r\n...     cursor.execute(\"insert into foo (name, weight) values ('Lila', '2.31')\")\r\n<pysqlite3.dbapi2.Cursor object at 0x10331e1f0>\r\n>>> conn.close()\r\n```\r\nUploaded that to: https://static.simonwillison.net/static/2021/strict-table.db", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982048918", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 982048918, "node_id": "IC_kwDOCGYnMM46iOCW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T21:40:42Z", "updated_at": "2021-11-29T21:40:42Z", "author_association": "OWNER", "body": "Here's a function that detects if `strict` is supported or not:\r\n```python\r\nimport secrets\r\nimport sqlite3\r\n\r\ndef supports_strict_tables(db = None):\r\n    if db is None:\r\n        db = sqlite3.connect(\":memory:\")\r\n    try:\r\n        table_name = 't{}'.format(secrets.token_hex(16))\r\n        with db:\r\n            db.execute(\"create table {} (name text) strict\".format(table_name))\r\n            db.execute(\"drop table {}\".format(table_name))\r\n        return True\r\n    except:\r\n        return False\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982049148", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 982049148, "node_id": "IC_kwDOCGYnMM46iOF8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T21:40:59Z", "updated_at": "2021-11-29T21:40:59Z", "author_association": "OWNER", "body": "I'm going to add that as `db.supports_strict`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982076702", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 982076702, "node_id": "IC_kwDOCGYnMM46iU0e", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T22:20:22Z", "updated_at": "2021-11-29T22:20:22Z", "author_association": "OWNER", "body": "I haven't documented `db.supports_strict` yet (I documented `table.strict`) because there wasn't an obvious section of the documentation for it.\r\n\r\nI need to remember to document it once I add documentation for the `strict=True` parameter.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982076924", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/344", "id": 982076924, "node_id": "IC_kwDOCGYnMM46iU38", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T22:20:44Z", "updated_at": "2021-11-29T22:20:44Z", "author_association": "OWNER", "body": "Need to figure out a good pattern for testing this in CI too - it will currently skip the new tests if it doesn't have SQLite 3.37 or higher.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066474200, "label": "Support STRICT tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982077873", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/346", "id": 982077873, "node_id": "IC_kwDOCGYnMM46iVGx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T22:22:05Z", "updated_at": "2021-11-29T22:22:05Z", "author_association": "OWNER", "body": "Ideally I'd like an extra set of matrix options for different versions of SQLite.\r\n\r\nI can use `pysqlite3` for this, but it isn't a completely compatible drop-in replacement - turns out it doesn't support the `iterdump()` method for example, see https://github.com/coleifer/pysqlite3/issues/24", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066563554, "label": "Way to test SQLite 3.37 (and potentially other versions) in CI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982078527", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/346", "id": 982078527, "node_id": "IC_kwDOCGYnMM46iVQ_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T22:23:03Z", "updated_at": "2021-11-29T22:23:03Z", "author_association": "OWNER", "body": "Here's a modified version of the `dump` command that works even with `pysqlite3`:\r\n\r\n```python\r\n@cli.command()\r\n@click.argument(\r\n    \"path\",\r\n    type=click.Path(exists=True, file_okay=True, dir_okay=False, allow_dash=False),\r\n    required=True,\r\n)\r\n@load_extension_option\r\ndef dump(path, load_extension):\r\n    \"\"\"Output a SQL dump of the schema and full contents of the database\"\"\"\r\n    db = sqlite_utils.Database(path)\r\n    _load_extensions(db, load_extension)\r\n    # pysqlite3 doesn't implement .iterdump()\r\n    from sqlite3.dump import _iterdump\r\n\r\n    for line in _iterdump(db.conn):\r\n        click.echo(line)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066563554, "label": "Way to test SQLite 3.37 (and potentially other versions) in CI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/345#issuecomment-982090895", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/345", "id": 982090895, "node_id": "IC_kwDOCGYnMM46iYSP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T22:44:36Z", "updated_at": "2021-11-29T22:44:36Z", "author_association": "OWNER", "body": "Documented here: https://sqlite-utils.datasette.io/en/latest/python-api.html#strict", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066501534, "label": "`table.strict` introspection boolean for identifying STRICT mode tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/345#issuecomment-982091363", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/345", "id": 982091363, "node_id": "IC_kwDOCGYnMM46iYZj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T22:45:26Z", "updated_at": "2021-11-29T22:45:26Z", "author_association": "OWNER", "body": "This is the implementation: https://github.com/simonw/sqlite-utils/blob/213a0ff177f23a35f3b235386366ff132eb879f1/sqlite_utils/db.py#L1236-L1241", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066501534, "label": "`table.strict` introspection boolean for identifying STRICT mode tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982094020", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/346", "id": 982094020, "node_id": "IC_kwDOCGYnMM46iZDE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T22:50:11Z", "updated_at": "2021-11-29T22:50:11Z", "author_association": "OWNER", "body": "For the moment I think I'll combine two problems into one, and just add a single matrix alternative that uses `pysqlite3` running SQLite 3.37.0 - only on macOS and Linux so I don't have to figure out how to compile it for Windows.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066563554, "label": "Way to test SQLite 3.37 (and potentially other versions) in CI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982094370", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/346", "id": 982094370, "node_id": "IC_kwDOCGYnMM46iZIi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T22:50:49Z", "updated_at": "2021-11-29T22:50:49Z", "author_association": "OWNER", "body": "I have a working recipe for compiling it for macOS here: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982006544", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066563554, "label": "Way to test SQLite 3.37 (and potentially other versions) in CI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982111751", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/346", "id": 982111751, "node_id": "IC_kwDOCGYnMM46idYH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:11:17Z", "updated_at": "2021-11-29T23:12:49Z", "author_association": "OWNER", "body": "To keep things simple for the moment I'm only going to add one extra thing to the matrix, and it will be a run of the tests against SQLite 3.37.0 using pysqlite3 on Linux only.\r\n\r\nI can use this mechanism: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#example-including-new-combinations", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066563554, "label": "Way to test SQLite 3.37 (and potentially other versions) in CI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982126665", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982126665, "node_id": "IC_kwDOCGYnMM46ihBJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:26:01Z", "updated_at": "2021-11-29T23:33:48Z", "author_association": "OWNER", "body": "https://github.com/simonw/sqlite-utils/blob/93b059dd230eae9eaae472b7fbabd4a66feeb79d/.github/workflows/test.yml#L11-L20\r\n\r\nThis configuration means that the numpy=0, Python=3.10, os=Ubuntu build will additionally use `pysqlite3` with the SQLite 3.37.0.\r\n\r\nIt's failing right now: https://github.com/simonw/sqlite-utils/runs/4360593156 - because `pysqlite3` doesn't provide `.iterdump()`.\r\n\r\nI can use the workaround from this comment: https://github.com/coleifer/pysqlite3/issues/24#issuecomment-982081267", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982129218", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982129218, "node_id": "IC_kwDOCGYnMM46ihpC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:31:02Z", "updated_at": "2021-11-29T23:31:02Z", "author_association": "OWNER", "body": "Here's the test run that's installing `pysqlite3` and that version of SQLite: https://github.com/simonw/sqlite-utils/runs/4360663292", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982129727", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982129727, "node_id": "IC_kwDOCGYnMM46ihw_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:31:58Z", "updated_at": "2021-11-29T23:31:58Z", "author_association": "OWNER", "body": "It failed on other Python versions with `mypy`:\r\n```\r\nsqlite_utils/utils.py:8: error: Cannot find implementation or library stub for module named \"sqlite3.dump\"\r\nsqlite_utils/utils.py:8: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982133970", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982133970, "node_id": "IC_kwDOCGYnMM46iizS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:41:17Z", "updated_at": "2021-11-29T23:41:17Z", "author_association": "OWNER", "body": "Took a bit of experimenting to get both `mypy` AND `flake8` to ignore the same line. The incantation that worked was this one:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/f990e134aa8219b687ff6c261330f36824b5df36/sqlite_utils/utils.py#L8\r\n\r\nOrder here matters - this did NOT work for both tools:\r\n\r\n```python\r\n from sqlite3.dump import _iterdump as iterdump   # noqa: F401 # type: ignore\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982136747", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982136747, "node_id": "IC_kwDOCGYnMM46ijer", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:48:05Z", "updated_at": "2021-11-29T23:48:05Z", "author_association": "OWNER", "body": "Some interesting test failures in the version that runs with `pysqlite3`:\r\n```\r\n=========================== short test summary info ============================\r\nFAILED tests/test_cli.py::test_enable_wal - assert 0 == 1\r\nFAILED tests/test_cli.py::test_disable_wal - pysqlite3.dbapi2.OperationalErro...\r\nFAILED tests/test_fts.py::test_rebuild_fts[searchable] - pysqlite3.dbapi2.Dat...\r\nFAILED tests/test_fts.py::test_rebuild_fts[searchable_fts] - pysqlite3.dbapi2...\r\nFAILED tests/test_wal.py::test_enable_disable_wal - pysqlite3.dbapi2.Operatio...\r\n================== 5 failed, 750 passed, 3 skipped in 15.20s ===================\r\n```\r\nhttps://github.com/simonw/sqlite-utils/runs/4360759085\r\n\r\nThe WAL errors look like this:\r\n```\r\nE           pysqlite3.dbapi2.OperationalError: cannot change into wal mode from within a transaction\r\n```\r\nTriggered by a call to `db.enable_wal()`\r\n\r\nThe FTS errors are caused by tests that try to deliberately corrupt the FTS index by running `fresh_db[\"searchable_fts_data\"].delete_where()` - and then rebuilding it using `rebuild_fts()`:\r\n```\r\n    @pytest.mark.parametrize(\"table_to_fix\", [\"searchable\", \"searchable_fts\"])\r\n    def test_rebuild_fts(fresh_db, table_to_fix):\r\n        table = fresh_db[\"searchable\"]\r\n        table.insert(search_records[0])\r\n        table.enable_fts([\"text\", \"country\"])\r\n        # Run a search\r\n        rows = list(table.search(\"tanuki\"))\r\n        assert len(rows) == 1\r\n        assert {\r\n            \"rowid\": 1,\r\n            \"text\": \"tanuki are running tricksters\",\r\n            \"country\": \"Japan\",\r\n            \"not_searchable\": \"foo\",\r\n        }.items() <= rows[0].items()\r\n        # Delete from searchable_fts_data\r\n        fresh_db[\"searchable_fts_data\"].delete_where()\r\n        # This should have broken the index\r\n        with pytest.raises(sqlite3.DatabaseError):\r\n            list(table.search(\"tanuki\"))\r\n        # Running rebuild_fts() should fix it\r\n>       fresh_db[table_to_fix].rebuild_fts()\r\n\r\ntests/test_fts.py:277: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\nsqlite_utils/db.py:1947: in rebuild_fts\r\n    self.db.execute(\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n\r\nself = <Database <pysqlite3.dbapi2.Connection object at 0x7fd95d299c30>>\r\nsql = \"INSERT INTO [searchable_fts]([searchable_fts]) VALUES('rebuild');\"\r\nparameters = None\r\n\r\n    def execute(\r\n        self, sql: str, parameters: Optional[Union[Iterable, dict]] = None\r\n    ) -> sqlite3.Cursor:\r\n        \"Execute SQL query and return a ``sqlite3.Cursor``.\"\r\n        if self._tracer:\r\n            self._tracer(sql, parameters)\r\n        if parameters is not None:\r\n            return self.conn.execute(sql, parameters)\r\n        else:\r\n>           return self.conn.execute(sql)\r\nE           pysqlite3.dbapi2.DatabaseError: database disk image is malformed\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982137293", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982137293, "node_id": "IC_kwDOCGYnMM46ijnN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:49:29Z", "updated_at": "2021-11-29T23:49:29Z", "author_association": "OWNER", "body": "A short term fix would be to skip those tests against `pysqlite3` - but longer term it would be good to address the underlying issue, particularly for the WAL ones (the FTS ones aren't too worrying since if you deliberately try and break the FTS table it's not hugely problematic if you corrupt your database).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982137888", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982137888, "node_id": "IC_kwDOCGYnMM46ijwg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:50:54Z", "updated_at": "2021-11-29T23:50:54Z", "author_association": "OWNER", "body": "If I'm going to `skipIf()` those tests I need a way to check if `pysqlite3` is being used.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null}