{"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553528386", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 553528386, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MzUyODM4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-13T18:03:10Z", "updated_at": "2019-11-13T18:03:54Z", "author_association": "OWNER", "body": "Maybe `inplace()` (combining \"insert\" and \"replace\")?\r\n\r\nIt could be an alias for `.insert(..., replace=True)`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553528850", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 553528850, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MzUyODg1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-13T18:04:20Z", "updated_at": "2019-11-13T18:04:20Z", "author_association": "OWNER", "body": "This is going to affect the design of the CLI subcommands as well.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553540146", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 553540146, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MzU0MDE0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-13T18:33:30Z", "updated_at": "2019-11-13T18:33:30Z", "author_association": "OWNER", "body": "Maybe instead of inventing a new term I should tell people to use `.insert(..., replace=True)` directly. That matches `ignore=True`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553574011", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 553574011, "node_id": "MDEyOklzc3VlQ29tbWVudDU1MzU3NDAxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-13T19:53:45Z", "updated_at": "2019-11-13T19:53:45Z", "author_association": "OWNER", "body": "First step: add a `replace=True` argument to `insert()` and `insert_all()` that does the same thing as the current `upsert=True`\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/8dab9fd1ccf571e188eec9ccf606a0c50fccf200/sqlite_utils/db.py#L938-L946", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-554565198", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 554565198, "node_id": "MDEyOklzc3VlQ29tbWVudDU1NDU2NTE5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-15T23:12:28Z", "updated_at": "2019-11-15T23:12:28Z", "author_association": "OWNER", "body": "Urgh this is going to be quite a bit of work, especially in the CLI module which shares an implementation for `upsert` and `insert` in a way that looks like it will have to be unwrapped.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-555690319", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 555690319, "node_id": "MDEyOklzc3VlQ29tbWVudDU1NTY5MDMxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-19T20:10:17Z", "updated_at": "2019-11-19T20:10:17Z", "author_association": "OWNER", "body": "Thinking about this further: I believe every time I've personally used `upsert` in the past (either with the Python library or the CLI tool) I've actually wanted the new behaviour, where \"upsert\" means \"update existing record with these changes, or insert a new record if one does not exist\".\r\n\r\nSo I'm happy with `upsert` doing that, and `insert --replace` being added as an option that does what `upsert` does ta the moment.\r\n\r\nI'll still ship it as version 2.0 since it's technically a breaking change.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569131397", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 569131397, "node_id": "MDEyOklzc3VlQ29tbWVudDU2OTEzMTM5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-12-26T20:49:11Z", "updated_at": "2019-12-26T20:49:11Z", "author_association": "OWNER", "body": "Don't forget to update the documentation. This will be quite an involved task.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569226620", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 569226620, "node_id": "MDEyOklzc3VlQ29tbWVudDU2OTIyNjYyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-12-27T09:05:29Z", "updated_at": "2019-12-27T09:05:36Z", "author_association": "OWNER", "body": "I'm going to start by ignoring the existing `upsert` entirely and implementing `.insert(..., replace=True)` and `$ sqlite-utils insert --replace`. Including updating the tests.\r\n\r\nThen I'll figure out how to implement the new `.upsert()` / `$ sqlite-utils upsert`.\r\n\r\nThen I'll update the documentation, and ship `sqlite-utils` 2.0.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569588216", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 569588216, "node_id": "MDEyOklzc3VlQ29tbWVudDU2OTU4ODIxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-12-30T05:31:45Z", "updated_at": "2019-12-30T05:31:45Z", "author_association": "OWNER", "body": "Last step: update changelog and ship 2.0. Then I can close this issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569844426", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66", "id": 569844426, "node_id": "MDEyOklzc3VlQ29tbWVudDU2OTg0NDQyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-12-31T01:30:20Z", "updated_at": "2019-12-31T01:30:20Z", "author_association": "OWNER", "body": "I shipped 2.0 - release notes here: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v2\r\n\r\nI also wrote about it on my blog: https://simonwillison.net/2019/Dec/30/sqlite-utils-2/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 521868864, "label": "The \".upsert()\" method is misnamed"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/63#issuecomment-549430429", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/63", "id": 549430429, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTQzMDQyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-04T16:20:35Z", "updated_at": "2019-11-04T16:20:35Z", "author_association": "OWNER", "body": "I don't think we need this. We already have a `dogs.create_index([\"name\"], if_not_exists=True)` option.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 517241040, "label": "ensure_index() method"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549418134", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/62", "id": 549418134, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTQxODEzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-04T15:54:18Z", "updated_at": "2019-11-04T15:54:18Z", "author_association": "OWNER", "body": "This is a good idea. The individual row version of it can work like the `.update(pk, ...)` method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 500783373, "label": "[enhancement] Method to delete a row in python"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549425012", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/62", "id": 549425012, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTQyNTAxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-04T16:09:08Z", "updated_at": "2019-11-04T16:09:08Z", "author_association": "OWNER", "body": "Documentation for `table.delete()`: https://github.com/simonw/sqlite-utils/blob/19073d6d972fad9d68dd74c28544cd29083f1c12/docs/python-api.rst#deleting-a-specific-record", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 500783373, "label": "[enhancement] Method to delete a row in python"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549425364", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/62", "id": 549425364, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTQyNTM2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-04T16:09:56Z", "updated_at": "2019-11-04T16:09:56Z", "author_association": "OWNER", "body": "Since we have `table.rows_where(where, where_args)` it makes sense to me to also support `table.delete_where(where, where_args)`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 500783373, "label": "[enhancement] Method to delete a row in python"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549429512", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/62", "id": 549429512, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTQyOTUxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-04T16:18:48Z", "updated_at": "2019-11-04T16:18:48Z", "author_association": "OWNER", "body": "Documentation for `.delete_where()`: https://github.com/simonw/sqlite-utils/blob/169ea455fc1f1d5e5b6e44cb339ba7ffa9d49c31/docs/python-api.rst#deleting-multiple-records", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 500783373, "label": "[enhancement] Method to delete a row in python"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549435364", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/62", "id": 549435364, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTQzNTM2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-04T16:30:34Z", "updated_at": "2019-11-04T16:30:34Z", "author_association": "OWNER", "body": "Released as 1.12.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 500783373, "label": "[enhancement] Method to delete a row in python"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/61#issuecomment-533818697", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/61", "id": 533818697, "node_id": "MDEyOklzc3VlQ29tbWVudDUzMzgxODY5Nw==", "user": {"value": 49260, "label": "amjith"}, "created_at": "2019-09-21T18:09:01Z", "updated_at": "2019-09-21T18:09:28Z", "author_association": "CONTRIBUTOR", "body": "@witeshadow The library version doesn't have helpers around CSV (at least not from what I can see in the code). \r\n\r\nBut here's a snippet that makes it easy to insert from CSV using the library. \r\n\r\n```\r\nimport csv\r\nfrom sqlite_utils import Database\r\n\r\n# CSV Reader\r\n\r\ncsv_file = open(\"filename.csv\") # open the csv file.\r\nreader = csv.reader(csv_file) # Create a CSV reader\r\nheaders = next(reader) # First line is the header\r\ndocs = (dict(zip(headers, row)) for row in reader)\r\n\r\n# Now you can use the `sqlite_utils` library. \r\n\r\ndb = Database(\"my_database.db\")\r\ndb[\"table_name\"].insert_all(docs)\r\n```\r\n\r\nThis snippet is adapted from reading the CLI source code on how it implements the csv option.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 491219910, "label": "importing CSV to SQLite as library"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/61#issuecomment-549432592", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/61", "id": 549432592, "node_id": "MDEyOklzc3VlQ29tbWVudDU0OTQzMjU5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-04T16:25:01Z", "updated_at": "2019-11-04T16:25:01Z", "author_association": "OWNER", "body": "Yeah I've thought about this a bit and I'm OK leaving it out.\r\n\r\nThe core idea of `sqlite-utils` is that if you can create a list, iterator or generator of Python dictionaries you can efficiently insert those into a SQLite table.\r\n\r\nThe `--csv` function is actually implemented as just a few lines of code which turn that incoming CSV into a generator of dictionaries:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/169ea455fc1f1d5e5b6e44cb339ba7ffa9d49c31/sqlite_utils/cli.py#L364-L368\r\n\r\nI could turn this into a reusable helper function but since it wouldn't have anything to do with database inserts (it would just be a helper that turns a CSV into a generator of dictionaries) it doesn't feel like it fits well in the Python library section of this package.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 491219910, "label": "importing CSV to SQLite as library"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/6#issuecomment-457978729", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/6", "id": 457978729, "node_id": "MDEyOklzc3VlQ29tbWVudDQ1Nzk3ODcyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-01-28T02:12:19Z", "updated_at": "2019-01-28T02:12:19Z", "author_association": "OWNER", "body": "Will need to solve #7 for this to become truly efficient.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 403624090, "label": "\"sqlite-utils insert\" should support newline-delimited JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/58#issuecomment-710399593", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/58", "id": 710399593, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMDM5OTU5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-16T18:39:31Z", "updated_at": "2020-10-16T18:39:31Z", "author_association": "OWNER", "body": "I don't think this is valuable enough to justify adding to the library - especially since you can execute FTS search against views by joining to an FTS table built against an underlying table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 488293926, "label": "Support enabling FTS on views"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527091278", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/57", "id": 527091278, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzA5MTI3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-02T10:13:32Z", "updated_at": "2019-09-02T10:13:32Z", "author_association": "OWNER", "body": "This is brilliant!\r\n\r\nOne tiny suggestion: I like `--create-triggers` and `create_triggers=True` better for this as they are shorter but still capture what it dies - especially since the underlying SQL uses `CREATE TRIGGER` statements.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487987958, "label": "Add triggers while enabling FTS"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527211047", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/57", "id": 527211047, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzIxMTA0Nw==", "user": {"value": 49260, "label": "amjith"}, "created_at": "2019-09-02T17:30:43Z", "updated_at": "2019-09-02T17:30:43Z", "author_association": "CONTRIBUTOR", "body": "I have merged the other PR (#56) into this one. \r\n\r\nI have incorporated your suggestions. Cheers!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487987958, "label": "Add triggers while enabling FTS"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527258212", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/57", "id": 527258212, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzI1ODIxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-02T23:40:56Z", "updated_at": "2019-09-02T23:40:56Z", "author_association": "OWNER", "body": "This is fantastic, thanks so much.\r\n\r\nI spotted a Datasette bug triggered by this: Datasette has its own version of the `detect_fts` function - at https://github.com/simonw/datasette/blob/d224ee2c98ac39c2c6e21a0ac0c62e5c3e1ccd11/datasette/utils/__init__.py#L466-L479 - which fails to pick up FTS tables created using the new escaping pattern.\r\n\r\nIt's a bug in Datasette, not sqlite-utils - so I'll open an issue there.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487987958, "label": "Add triggers while enabling FTS"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527266798", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/57", "id": 527266798, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzI2Njc5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-03T01:03:59Z", "updated_at": "2019-09-03T01:03:59Z", "author_association": "OWNER", "body": "Released in 1.11 - thanks again! https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-11", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487987958, "label": "Add triggers while enabling FTS"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638910473", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/567", "id": 1638910473, "node_id": "IC_kwDOCGYnMM5hr8oJ", "user": {"value": 15178711, "label": "asg017"}, "created_at": "2023-07-17T21:27:41Z", "updated_at": "2023-07-17T21:27:41Z", "author_association": "NONE", "body": "Another use-case: I want to make a `sqlite-utils` plugin that'll help me insert data into Datasette. \r\n\r\n```bash\r\nsqlite-utils insert-datasette \\\r\n --token $DATASETTE_API_KEY \\\r\n https://latest.datasette.io/fixtures/my-table \\\r\n 'select ...'\r\n```\r\n\r\n\r\nThis could also be a datasette plugin (ex `datasette upload-data ...`, but you can also think of `sqlite-utils` plugins that upload to S3, a postgres table, other DBMS's, etc.)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1801394744, "label": "Plugin system"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638926655", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/567", "id": 1638926655, "node_id": "IC_kwDOCGYnMM5hsAk_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-17T21:42:37Z", "updated_at": "2023-07-17T21:42:37Z", "author_association": "OWNER", "body": "I really like this. I'm also interested in:\r\n- Plugins that make new custom SQL functions available - similar to this Datasette hook: https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette\r\n- Plugins that register functions that can be used as recipes for `sqlite-utils convert` https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes\r\n\r\nThe upload-data-to-Datasette problem is planned to be solved by a future version of https://github.com/simonw/dclient ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1801394744, "label": "Plugin system"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627597872", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/566", "id": 1627597872, "node_id": "IC_kwDOCGYnMM5hAyww", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-09T04:09:56Z", "updated_at": "2023-07-09T04:09:56Z", "author_association": "OWNER", "body": "Thanks, looks like a bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1795219865, "label": "`--no-headers` doesn't work on most formats"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627598570", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/566", "id": 1627598570, "node_id": "IC_kwDOCGYnMM5hAy7q", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-09T04:13:34Z", "updated_at": "2023-07-09T04:13:34Z", "author_association": "OWNER", "body": "On consulting https://pypi.org/project/tabulate/ it looks like most of those formats don't actually makes sense without headers - so the right thing here might be to raise an error if `--fmt` and `--no-headers` are used at the same time.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1795219865, "label": "`--no-headers` doesn't work on most formats"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1618380888", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/565", "id": 1618380888, "node_id": "IC_kwDOCGYnMM5gdohY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-03T14:09:11Z", "updated_at": "2023-07-03T14:09:31Z", "author_association": "OWNER", "body": "For the CLI:\r\n\r\n```bash\r\nsqlite-utils rename-table data.db old_table_name new_table_name\r\n```\r\n\r\nFor the Python code, should it go on Table or on Database?\r\n```python\r\ndb[\"foo\"].rename_table(\"bar\")\r\n\r\ndb.rename_table(\"foo\", \"bar\")\r\n```\r\nI think I like the second better, it's slightly more clear.\r\n\r\nAlso need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1786258502, "label": "Table renaming utilities"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/563#issuecomment-1617395444", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/563", "id": 1617395444, "node_id": "IC_kwDOCGYnMM5gZ370", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-03T05:44:43Z", "updated_at": "2023-07-03T05:44:43Z", "author_association": "OWNER", "body": "Documentation at the bottom of this section: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-csv-or-tsv-data", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1785360409, "label": "`--empty-null` option when importing CSV"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/562#issuecomment-1616782404", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/562", "id": 1616782404, "node_id": "IC_kwDOCGYnMM5gXiRE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-07-02T19:24:14Z", "updated_at": "2023-07-02T19:26:39Z", "author_association": "OWNER", "body": "[Dataclasses](https://docs.python.org/3/library/dataclasses.html) were added in Python 3.7 and `sqlite-utils` was originally written for Python 3.6 - but both 3.6 and 3.7 are EOL now.\r\n\r\nThe thing that makes Dataclasses particularly interesting is the potential to use type annotations with them to help specify the types of the related SQLite columns.\r\n\r\nExample for https://datasette.io/content/users\r\n\r\n```sql\r\nCREATE TABLE [users] (\r\n [login] TEXT,\r\n [id] INTEGER PRIMARY KEY,\r\n [node_id] TEXT,\r\n [avatar_url] TEXT,\r\n [gravatar_id] TEXT,\r\n [html_url] TEXT,\r\n [type] TEXT,\r\n [site_admin] INTEGER,\r\n [name] TEXT\r\n);\r\n```\r\nAnd the dataclass:\r\n```python\r\nfrom dataclasses import dataclass\r\n\r\n@dataclass\r\nclass User:\r\n id: int\r\n login: str\r\n node_id: str\r\n avatar_url: str\r\n gravatar_id: str\r\n html_url: str\r\n type: str\r\n site_admin: int\r\n name: str\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1784794489, "label": "Explore the intersection between sqlite-utils and dataclasses"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/561#issuecomment-1610040517", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/561", "id": 1610040517, "node_id": "IC_kwDOCGYnMM5f90TF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-27T18:44:31Z", "updated_at": "2023-06-27T18:44:38Z", "author_association": "OWNER", "body": "Got this working:\r\n```bash\r\nsqlite-utils insert /tmp/playground.db Playground_Submission_Data \\\r\n ~/Downloads/Playground_Submission_Data.csv --csv --stop-after 2000\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1777548699, "label": "`--stop-after` option for `insert` and `upsert` commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606237836", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606237836, "node_id": "IC_kwDOCGYnMM5fvT6M", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-06-25T19:49:45Z", "updated_at": "2023-06-26T08:20:59Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage: **`80.55`**% and project coverage change: **`-0.15`** :warning:\n> Comparison is base [(`2747257`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/2747257a3334d55e890b40ec58fada57ae8cfbfd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`5e7d27e`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.22%.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #560 +/- ##\n==========================================\n- Coverage 96.36% 96.22% -0.15% \n==========================================\n Files 6 6 \n Lines 2726 2752 +26 \n==========================================\n+ Hits 2627 2648 +21 \n- Misses 99 104 +5 \n```\n\n\n| [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/utils.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.56% <63.63%> (-0.62%)` | :arrow_down: |\n| [sqlite\\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.33% <86.36%> (-0.20%)` | :arrow_down: |\n| [sqlite\\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <100.00%> (\u00f8)` | |\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270055", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606270055, "node_id": "IC_kwDOCGYnMM5fvbxn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T21:31:56Z", "updated_at": "2023-06-25T21:31:56Z", "author_association": "OWNER", "body": "Lots of failures now that I'm trying to run the tests against `sqlean.py` on macOS and Python 3.10: https://github.com/simonw/sqlite-utils/actions/runs/5371800108/jobs/9744802953\r\n\r\nA bunch of these, because `pysqlite3` chooses not to implement `.iterdump()`:\r\n```\r\n @pytest.fixture\r\n def db_to_analyze_path(db_to_analyze, tmpdir):\r\n path = str(tmpdir / \"test.db\")\r\n db = sqlite3.connect(path)\r\n> db.executescript(\"\\n\".join(db_to_analyze.conn.iterdump()))\r\nE AttributeError: 'sqlean.dbapi2.Connection' object has no attribute 'iterdump'\r\n```\r\nAlso some of these:\r\n```\r\n def test_analyze_whole_database(db):\r\n assert set(db.table_names()) == {\"one_index\", \"two_indexes\"}\r\n db.analyze()\r\n> assert set(db.table_names()) == {\"one_index\", \"two_indexes\", \"sqlite_stat1\"}\r\nE AssertionError: assert {'one_index',...'two_indexes'} == {'one_index',...'two_indexes'}\r\nE Extra items in the left set:\r\nE 'sqlite_stat4'\r\nE Full diff:\r\nE - {'two_indexes', 'sqlite_stat1', 'one_index'}\r\nE + {'two_indexes', 'sqlite_stat1', 'sqlite_stat4', 'one_index'}\r\nE ? ++++++++++++++++\r\n```\r\nApparently `sqlean.py` adds a `sqlite_stat4` table that the tests are not expecting.\r\n\r\nPlus some errors that look like this:\r\n```\r\n def test_enable_wal():\r\n runner = CliRunner()\r\n dbs = [\"test.db\", \"test2.db\"]\r\n with runner.isolated_filesystem():\r\n for dbname in dbs:\r\n db = Database(dbname)\r\n db[\"t\"].create({\"pk\": int}, pk=\"pk\")\r\n assert db.journal_mode == \"delete\"\r\n result = runner.invoke(cli.cli, [\"enable-wal\"] + dbs)\r\n> assert 0 == result.exit_code\r\nE AssertionError: assert 0 == 1\r\nE + where 1 = .exit_code\r\n```\r\nTest summary:\r\n```\r\n============ 13 failed, 909 passed, 16 skipped, 2 errors in 19.29s =============\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270887", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606270887, "node_id": "IC_kwDOCGYnMM5fvb-n", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T21:37:12Z", "updated_at": "2023-06-26T08:21:00Z", "author_association": "OWNER", "body": "On my own laptop I got a crash running the tests - details here:\r\n\r\n- https://github.com/nalgeon/sqlean.py/issues/3", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606273005", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606273005, "node_id": "IC_kwDOCGYnMM5fvcft", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T21:47:47Z", "updated_at": "2023-06-25T21:47:47Z", "author_association": "OWNER", "body": "I can use https://github.com/simonw/sqlite-dump as an optional dependency to handle the missing `.iterdump()` method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606290917", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606290917, "node_id": "IC_kwDOCGYnMM5fvg3l", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T22:32:28Z", "updated_at": "2023-06-25T22:32:28Z", "author_association": "OWNER", "body": "I've fixed most of the test failures, but I still need to fix this one:\r\n\r\n> cannot change into wal mode from within a transaction", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606293382", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606293382, "node_id": "IC_kwDOCGYnMM5fvheG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T22:34:47Z", "updated_at": "2023-06-25T22:34:47Z", "author_association": "OWNER", "body": "```pycon\r\n>>> import sqlite3\r\n>>> db = sqlite3.connect(\"/tmp/1.db\")\r\n>>> db.execute('PRAGMA journal_mode=wal;')\r\n\r\n>>> import sqlean\r\n>>> db2 = sqlean.connect(\"/tmp/2.db\")\r\n>>> db2.execute('PRAGMA journal_mode=wal;')\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\nsqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606294627", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606294627, "node_id": "IC_kwDOCGYnMM5fvhxj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T22:40:10Z", "updated_at": "2023-06-25T22:40:10Z", "author_association": "OWNER", "body": "I suspect this has something to do with `autocommit` mode in `sqlite3` - which I may be able to turn off by setting `con.isolation_level = None`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606297356", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606297356, "node_id": "IC_kwDOCGYnMM5fvicM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T22:42:41Z", "updated_at": "2023-06-25T22:42:41Z", "author_association": "OWNER", "body": "Yes that does seem to do the trick:\r\n```pycon\r\n>>> import sqlean\r\n>>> db = sqlean.connect(\"/tmp/4.db\")\r\n>>> db.execute('PRAGMA journal_mode;').fetchall()\r\n[('delete',)]\r\n>>> db.isolation_level\r\n''\r\n>>> db.execute('PRAGMA journal_mode=wal;')\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\nsqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction\r\n>>> db.isolation_level = None\r\n>>> db.isolation_level\r\n>>> db.execute('PRAGMA journal_mode=wal;')\r\n\r\n```\r\nWeird how `isolation_level` of empty string causes the error, but setting that to `None` fixes the error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606310630", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606310630, "node_id": "IC_kwDOCGYnMM5fvlrm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T23:06:07Z", "updated_at": "2023-06-25T23:06:07Z", "author_association": "OWNER", "body": "Filed an issue about the above with `pysqlite3` (which `sqlean.py` is based on) here:\r\n- https://github.com/coleifer/pysqlite3/issues/58", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606315321", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606315321, "node_id": "IC_kwDOCGYnMM5fvm05", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T23:18:33Z", "updated_at": "2023-06-25T23:18:33Z", "author_association": "OWNER", "body": "Documentation preview: https://sqlite-utils--560.org.readthedocs.build/en/560/installation.html#alternatives-to-sqlite3", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/56#issuecomment-527090411", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/56", "id": 527090411, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzA5MDQxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-02T10:10:48Z", "updated_at": "2019-09-02T10:10:48Z", "author_association": "OWNER", "body": "Good spot, thanks!\r\n\r\nWould be useful to add a test as well, derived from your above example.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487847945, "label": "Escape the table name in populate_fts and search."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/56#issuecomment-527209840", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/56", "id": 527209840, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzIwOTg0MA==", "user": {"value": 49260, "label": "amjith"}, "created_at": "2019-09-02T17:23:21Z", "updated_at": "2019-09-02T17:23:21Z", "author_association": "CONTRIBUTOR", "body": "I have updated the other PR with the changes from this one and added tests. I have also changed the escaping from double quotes to brackets. \r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487847945, "label": "Escape the table name in populate_fts and search."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1577355134", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/557", "id": 1577355134, "node_id": "IC_kwDOCGYnMM5eBId-", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-05T19:26:26Z", "updated_at": "2023-06-05T19:26:26Z", "author_association": "CONTRIBUTOR", "body": "this isn't really actionable... I'm just being a whiny baby. I have tasted the milk of being able to use `upsert_all`, `insert_all`, etc without having to write DDL to create tables. The meat of the issue is that SQLITE doesn't make rowid stable between vacuums so it is not possible to take shortcuts", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1740150327, "label": "Aliased ROWID option for tables created from alter=True commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1590531892", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/557", "id": 1590531892, "node_id": "IC_kwDOCGYnMM5ezZc0", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T06:09:21Z", "updated_at": "2023-06-14T06:09:21Z", "author_association": "CONTRIBUTOR", "body": "I put together a [simple script](https://github.com/chapmanjacobd/library/blob/42129c5ebe15f9d74653c0f5ca4ed0c991d383e0/xklb/scripts/dedupe_db.py) to upsert and remove duplicate rows based on business keys. If anyone has similar problems with above this might help\r\n\r\n```\r\nCREATE TABLE my_table (\r\n id INTEGER PRIMARY KEY,\r\n column1 TEXT,\r\n column2 TEXT,\r\n column3 TEXT\r\n);\r\n\r\nINSERT INTO my_table (column1, column2, column3)\r\nVALUES\r\n ('Value 1', 'Duplicate 1', 'Duplicate A'),\r\n ('Value 2', 'Duplicate 2', 'Duplicate B'),\r\n ('Value 3', 'Duplicate 2', 'Duplicate C'),\r\n ('Value 4', 'Duplicate 3', 'Duplicate D'),\r\n ('Value 5', 'Duplicate 3', 'Duplicate E'),\r\n ('Value 6', 'Duplicate 3', 'Duplicate F');\r\n```\r\n\r\n```\r\nlibrary dedupe-db test.db my_table --bk column2\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1740150327, "label": "Aliased ROWID option for tables created from alter=True commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/556#issuecomment-1575310378", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/556", "id": 1575310378, "node_id": "IC_kwDOCGYnMM5d5VQq", "user": {"value": 601708, "label": "mcint"}, "created_at": "2023-06-04T01:21:15Z", "updated_at": "2023-06-04T01:21:15Z", "author_association": "CONTRIBUTOR", "body": "I've resolved my use, with the line-buffered output and while read loop for line buffered input, but I leave this here so the incremental saving or line-buffered use-case can be explicitly handled or rejected (or deferred).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1740026046, "label": "Support storing incrementally piped values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/555#issuecomment-1592047502", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/555", "id": 1592047502, "node_id": "IC_kwDOCGYnMM5e5LeO", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2023-06-14T22:00:10Z", "updated_at": "2023-06-14T22:01:57Z", "author_association": "CONTRIBUTOR", "body": "You may want to try doing a performance comparison between this and just selecting all the ids with few constraints and then doing the filtering within python.\r\n\r\nThat might seem like a lazy-programmer, inefficient way but queries with large resultsets are a different profile than what databases like SQLITE are designed for. That is not to say that SQLITE is slow or that python is always faster but when you start reading >20% of an index there is an equilibrium that is reached. Especially when adding in writing extra temp tables and stuff to memory/disk. And especially given the `NOT IN` style of query...\r\n\r\nYou may also try chunking like this:\r\n\r\n```py\r\ndef chunks(lst, n) -> Generator:\r\n for i in range(0, len(lst), n):\r\n yield lst[i : i + n]\r\n\r\nSQLITE_PARAM_LIMIT = 32765\r\n\r\ndata = []\r\nchunked = chunks(video_ids, consts.SQLITE_PARAM_LIMIT)\r\nfor ids in chunked:\r\n data.expand(\r\n list(\r\n db.query(\r\n f\"\"\"SELECT * from videos\r\n WHERE id in (\"\"\"\r\n + \",\".join([\"?\"] * len(ids))\r\n + \")\",\r\n (*ids,),\r\n )\r\n )\r\n )\r\n```\r\n\r\nbut that actually won't work with your `NOT IN` requirements. You need to query the full resultset to check any row.\r\n\r\nSince you are doing stuff with files/videos in SQLITE you might be interested in my side project: https://github.com/chapmanjacobd/library", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1733198948, "label": "Filter table by a large bunch of ids"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/554#issuecomment-1557607516", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/554", "id": 1557607516, "node_id": "IC_kwDOCGYnMM5c1zRc", "user": {"value": 1231935, "label": "xavdid"}, "created_at": "2023-05-22T17:18:33Z", "updated_at": "2023-05-22T17:18:33Z", "author_association": "NONE", "body": "Oh and for context - this goes away if I use `.upsert` instead of `insert(..., ignore=True)`, but I don't want to update the value if it's written, just do an insert if it's new. The code is basically:\r\n\r\n```py\r\ndef save_items(table, items):\r\n db[\"users\"].insert(build_user(items[0]), pk=\"id\",ignore=True)\r\n db[table].insert_all(items)\r\n\r\nif comments := fetch_comments():\r\n save_items('comments', comments)\r\n\r\nif posts := fetch_posts():\r\n save_items('posts', posts)\r\n```\r\n\r\nSo either `comments` or `post` could create the relevant user if those items exist. In cases where they _both_ exist, I get this error. I need the `pk` because either call could create the table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1720096994, "label": "`IndexError` when doing `.insert(..., pk='id')` after `insert_all`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556287870", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/553", "id": 1556287870, "node_id": "IC_kwDOCGYnMM5cwxF-", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-05-21T20:45:58Z", "updated_at": "2023-05-21T20:57:08Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch and project coverage have no change.\n> Comparison is base [(`e240133`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/e240133b11588d31dc22c632f7a7ca636c72978d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`0b81794`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36%.\n\n> :exclamation: Current head 0b81794 differs from pull request most recent head 21036a5. Consider uploading reports for the commit 21036a5 to get more accurate results\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #553 +/- ##\n=======================================\n Coverage 96.36% 96.36% \n=======================================\n Files 6 6 \n Lines 2726 2726 \n=======================================\n Hits 2627 2627 \n Misses 99 99 \n```\n\n\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718635018, "label": "Reformatted CLI examples in docs"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556288300", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/553", "id": 1556288300, "node_id": "IC_kwDOCGYnMM5cwxMs", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T20:48:01Z", "updated_at": "2023-05-21T20:48:01Z", "author_association": "OWNER", "body": "If https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries looks good I can merge this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718635018, "label": "Reformatted CLI examples in docs"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/552#issuecomment-1556292204", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/552", "id": 1556292204, "node_id": "IC_kwDOCGYnMM5cwyJs", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T21:05:15Z", "updated_at": "2023-05-21T21:05:15Z", "author_association": "OWNER", "body": "Now live at https://sqlite-utils.datasette.io/en/latest/installation.html#setting-up-shell-completion", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718612569, "label": "Document how to setup shell auto-completion"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556262574", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/551", "id": 1556262574, "node_id": "IC_kwDOCGYnMM5cwq6u", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T19:04:59Z", "updated_at": "2023-05-21T19:04:59Z", "author_association": "OWNER", "body": "I wrote the docs like this because early examples include both the command and its output:\r\n\r\nhttps://sqlite-utils.datasette.io/en/stable/cli.html#returning-json\r\n\r\n\"image\"", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718607907, "label": "Make as many examples in the CLI docs as possible copy-and-pastable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556263182", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/551", "id": 1556263182, "node_id": "IC_kwDOCGYnMM5cwrEO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T19:06:48Z", "updated_at": "2023-05-21T19:06:48Z", "author_association": "OWNER", "body": "I could split them up into two blocks like this:\r\n\r\n\"image\"\r\n\r\nI do miss the visual indication that one of these is the command and one is the output though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718607907, "label": "Make as many examples in the CLI docs as possible copy-and-pastable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556265772", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/551", "id": 1556265772, "node_id": "IC_kwDOCGYnMM5cwrss", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T19:16:15Z", "updated_at": "2023-05-21T19:16:15Z", "author_association": "OWNER", "body": "Another option:\r\n\r\n\"image\"\r\n\r\nThat's using this markup:\r\n\r\n```\r\nNewline-delimited JSON\r\n~~~~~~~~~~~~~~~~~~~~~~\r\n\r\nUse ``--nl`` to get back newline-delimited JSON objects:\r\n\r\n.. code-block:: bash\r\n\r\n sqlite-utils dogs.db \"select * from dogs\" --nl\r\n\r\n.. code-block:: output\r\n\r\n {\"id\": 1, \"age\": 4, \"name\": \"Cleo\"}\r\n {\"id\": 2, \"age\": 2, \"name\": \"Pancakes\"}\r\n```\r\nAnd this extra CSS:\r\n\r\n```css\r\n.highlight-output .highlight {\r\n border-left: 9px solid #30c94f;\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718607907, "label": "Make as many examples in the CLI docs as possible copy-and-pastable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556287599", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/551", "id": 1556287599, "node_id": "IC_kwDOCGYnMM5cwxBv", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T20:44:55Z", "updated_at": "2023-05-21T20:44:55Z", "author_association": "OWNER", "body": "Put this in a PR so I can preview it: \r\n- #553\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718607907, "label": "Make as many examples in the CLI docs as possible copy-and-pastable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556288270", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/551", "id": 1556288270, "node_id": "IC_kwDOCGYnMM5cwxMO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T20:47:51Z", "updated_at": "2023-05-21T20:47:51Z", "author_association": "OWNER", "body": "This page has all of the changes: https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718607907, "label": "Make as many examples in the CLI docs as possible copy-and-pastable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556291915", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/551", "id": 1556291915, "node_id": "IC_kwDOCGYnMM5cwyFL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T21:04:03Z", "updated_at": "2023-05-21T21:04:03Z", "author_association": "OWNER", "body": "Now live at https://sqlite-utils.datasette.io/en/latest/cli.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718607907, "label": "Make as many examples in the CLI docs as possible copy-and-pastable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556249984", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/550", "id": 1556249984, "node_id": "IC_kwDOCGYnMM5cwn2A", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T18:24:48Z", "updated_at": "2023-05-21T18:24:48Z", "author_association": "OWNER", "body": "This is blocking:\r\n- #549", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718595700, "label": "AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556250236", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/550", "id": 1556250236, "node_id": "IC_kwDOCGYnMM5cwn58", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T18:25:26Z", "updated_at": "2023-05-21T18:25:26Z", "author_association": "OWNER", "body": "Relevant issues:\r\n- https://github.com/python/importlib_metadata/issues/406\r\n- https://github.com/PyCQA/flake8/issues/1701\r\n\r\nIt looks to me like this is only a problem for `flake8` on Python 3.7 - 3.8 and higher work OK.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718595700, "label": "AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556255309", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/550", "id": 1556255309, "node_id": "IC_kwDOCGYnMM5cwpJN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T18:42:25Z", "updated_at": "2023-05-21T18:42:25Z", "author_association": "OWNER", "body": "Tests passed here: https://github.com/simonw/sqlite-utils/actions/runs/5039119716", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718595700, "label": "AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/55#issuecomment-522238063", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/55", "id": 522238063, "node_id": "MDEyOklzc3VlQ29tbWVudDUyMjIzODA2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-08-17T13:42:40Z", "updated_at": "2019-08-17T13:42:40Z", "author_association": "OWNER", "body": "I still need to implement the part of this where certain actions against views (like updating/inserting) are not allowed.\r\n\r\nI think the better way to do that is to introduce a new class - `View` - rather than reusing the existing `Table` class - since there's a lot of stuff on Table (like primary key / foreign key introspection) which simply doesn't make sense for views.\r\n\r\nThis will get a bit more complicated - maybe I need both Table and View to extend a common subclass here?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 481887482, "label": "Ability to introspect and run queries against views"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556241555", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/549", "id": 1556241555, "node_id": "IC_kwDOCGYnMM5cwlyT", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-05-21T17:57:24Z", "updated_at": "2023-05-21T18:28:44Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage: **`83.33`**% and project coverage change: **`+0.06`** :tada:\n> Comparison is base [(`b3b100d`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/b3b100d7f5b2a76ccd4bfe8b0301a29e321d0375?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.30% compared to head [(`948692a`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36%.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #549 +/- ##\n==========================================\n+ Coverage 96.30% 96.36% +0.06% \n==========================================\n Files 6 6 \n Lines 2707 2726 +19 \n==========================================\n+ Hits 2607 2627 +20 \n+ Misses 100 99 -1 \n```\n\n\n| [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <83.33%> (-0.03%)` | :arrow_down: |\n\n... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718586377, "label": "TUI powered by Trogon"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556241812", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/549", "id": 1556241812, "node_id": "IC_kwDOCGYnMM5cwl2U", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T17:58:25Z", "updated_at": "2023-05-21T17:58:25Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718586377, "label": "TUI powered by Trogon"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556242262", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/549", "id": 1556242262, "node_id": "IC_kwDOCGYnMM5cwl9W", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T18:00:05Z", "updated_at": "2023-05-21T18:00:05Z", "author_association": "OWNER", "body": "Failing `mypy` test: https://github.com/simonw/sqlite-utils/actions/runs/5038983349/jobs/9036828465\r\n```\r\nsqlite_utils/cli.py:37: error: Skipping analyzing \"trogon\": module is installed, but missing library stubs or py.typed marker [import]\r\nsqlite_utils/cli.py:37: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports\r\nFound 1 error in 1 file (checked 52 source files)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718586377, "label": "TUI powered by Trogon"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/548#issuecomment-1556231832", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/548", "id": 1556231832, "node_id": "IC_kwDOCGYnMM5cwjaY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T17:24:13Z", "updated_at": "2023-05-21T17:24:13Z", "author_association": "OWNER", "body": "Oh, I see why that is now:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/6027f3ea6939a399aeef2578fca17efec0e539df/sqlite_utils/cli.py#L2670-L2679\r\n\r\nThis is because of the following command:\r\n\r\n sqlite-utils analyze-tables table1 table2 --column x\r\n\r\nSince you can pass multiple tables AND multiple columns, the tool currently assumes that the column(s) you specify may be available on a subset of the provided tables.\r\n\r\nI'm going to change this so if the column is not on ANY of those tables you get an error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718576761, "label": "analyze-tables should validate provide --column names"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/547#issuecomment-1556228395", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/547", "id": 1556228395, "node_id": "IC_kwDOCGYnMM5cwikr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T17:11:15Z", "updated_at": "2023-05-21T17:11:15Z", "author_association": "OWNER", "body": "This will be a cosmetic change to the CLI output only - the options to save data to the database and the Python API function will continue to return `[(None, 158)]`.\r\n\r\nI can add an optimization though to avoid running the SQL count query if we know that it's all `null`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718572201, "label": "No need to show common values if everything is null"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213031", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/546", "id": 1556213031, "node_id": "IC_kwDOCGYnMM5cwe0n", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-05-21T15:56:05Z", "updated_at": "2023-05-21T16:18:03Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage: **`93.75`**% and no project coverage change.\n> Comparison is base [(`b3b100d`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/b3b100d7f5b2a76ccd4bfe8b0301a29e321d0375?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.30% compared to head [(`9f23e68`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.31%.\n\n> :exclamation: Current head 9f23e68 differs from pull request most recent head 2eca17d. Consider uploading reports for the commit 2eca17d to get more accurate results\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #546 +/- ##\n=======================================\n Coverage 96.30% 96.31% \n=======================================\n Files 6 6 \n Lines 2707 2712 +5 \n=======================================\n+ Hits 2607 2612 +5 \n Misses 100 100 \n```\n\n\n| [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.37% <90.90%> (+<0.01%)` | :arrow_up: |\n| [sqlite\\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.26% <100.00%> (+0.01%)` | :arrow_up: |\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718550688, "label": "Analyze tables options: --common-limit, --no-most, --no-least"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213396", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/546", "id": 1556213396, "node_id": "IC_kwDOCGYnMM5cwe6U", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T15:58:12Z", "updated_at": "2023-05-21T16:18:46Z", "author_association": "OWNER", "body": "Documentation preview:\r\n- https://sqlite-utils--546.org.readthedocs.build/en/546/cli.html#cli-analyze-tables\r\n- https://sqlite-utils--546.org.readthedocs.build/en/546/cli-reference.html#analyze-tables\r\n- https://sqlite-utils--546.org.readthedocs.build/en/546/python-api.html#analyzing-a-column\r\n- https://sqlite-utils--546.org.readthedocs.build/en/546/reference.html#sqlite_utils.db.Table.analyze_column", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718550688, "label": "Analyze tables options: --common-limit, --no-most, --no-least"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556189823", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/545", "id": 1556189823, "node_id": "IC_kwDOCGYnMM5cwZJ_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T14:09:59Z", "updated_at": "2023-05-21T14:09:59Z", "author_association": "OWNER", "body": "I don't want to add `trogon` as a default dependency because it's a little heavy - it pulls in all of Rich and Textual as well. People who use `sqlite-utils` just for its Python API won't benefit from this - it's a CLI feature only.\r\n\r\nBut I have a `sqlite-utils install ...` command for helping people to install packages into the same virtual environment as `sqlite-utils` no matter how they installed that tool: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install\r\n\r\nSo I can treat Trogon as an optional dependency and add the `sqlite-utils tui` command only if that package is also installed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718517882, "label": "Try out Trogon for a tui interface"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556190531", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/545", "id": 1556190531, "node_id": "IC_kwDOCGYnMM5cwZVD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T14:13:43Z", "updated_at": "2023-05-21T14:13:43Z", "author_association": "OWNER", "body": "OK, this works!\r\n\r\n![trogon](https://github.com/simonw/sqlite-utils/assets/9599/2ae194c5-ec82-471a-9d1b-b01b3f2632f3)\r\n\r\nTo try it out, install that branch from GitHub:\r\n\r\n pip install https://github.com/simonw/sqlite-utils/archive/refs/heads/trogon.zip\r\n\r\nThen run this:\r\n\r\n sqlite-utils install trogon\r\n\r\nAnd this:\r\n\r\n sqlite-utils tui\r\n", "reactions": "{\"total_count\": 5, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 3, \"eyes\": 0}", "issue": {"value": 1718517882, "label": "Try out Trogon for a tui interface"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556191894", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/545", "id": 1556191894, "node_id": "IC_kwDOCGYnMM5cwZqW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T14:20:14Z", "updated_at": "2023-05-21T14:20:14Z", "author_association": "OWNER", "body": "Opened a feature request for customizing the help and command name:\r\n\r\n- https://github.com/Textualize/trogon/issues/2", "reactions": "{\"total_count\": 2, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718517882, "label": "Try out Trogon for a tui interface"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556210844", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/545", "id": 1556210844, "node_id": "IC_kwDOCGYnMM5cweSc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T15:44:10Z", "updated_at": "2023-05-21T15:44:10Z", "author_association": "OWNER", "body": "It looks like `nargs=-1` on a positional argument isn't yet supported - opened an issue here:\r\n- https://github.com/Textualize/trogon/issues/4", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718517882, "label": "Try out Trogon for a tui interface"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556247818", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/545", "id": 1556247818, "node_id": "IC_kwDOCGYnMM5cwnUK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T18:17:46Z", "updated_at": "2023-05-21T18:17:46Z", "author_association": "OWNER", "body": "Draft documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718517882, "label": "Try out Trogon for a tui interface"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556269616", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/545", "id": 1556269616, "node_id": "IC_kwDOCGYnMM5cwsow", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T19:33:13Z", "updated_at": "2023-05-21T19:33:13Z", "author_association": "OWNER", "body": "Now released: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-32", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718517882, "label": "Try out Trogon for a tui interface"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556211643", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/544", "id": 1556211643, "node_id": "IC_kwDOCGYnMM5cwee7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T15:48:17Z", "updated_at": "2023-05-21T15:48:17Z", "author_association": "OWNER", "body": "I generated the commit message in https://github.com/simonw/sqlite-utils/commit/1c1991b447a1ddd3d61d9d4a8a1d6a9da47ced20 using `git diff | llm --system 'describe this change'`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718515590, "label": "New options for analyze-tables --common-limit --no-most and --no-least"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556225788", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/544", "id": 1556225788, "node_id": "IC_kwDOCGYnMM5cwh78", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T17:02:05Z", "updated_at": "2023-05-21T17:02:05Z", "author_association": "OWNER", "body": "New docs:\r\n\r\n- https://sqlite-utils.datasette.io/en/latest/cli.html#cli-analyze-tables\r\n- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#analyze-tables\r\n- https://sqlite-utils.datasette.io/en/latest/python-api.html#analyzing-a-column\r\n- https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.analyze_column\r\n\r\nNew help output:\r\n\r\n```\r\n % sqlite-utils analyze-tables --help\r\nUsage: sqlite-utils analyze-tables [OPTIONS] PATH [TABLES]...\r\n\r\n Analyze the columns in one or more tables\r\n\r\n Example:\r\n\r\n sqlite-utils analyze-tables data.db trees\r\n\r\nOptions:\r\n -c, --column TEXT Specific columns to analyze\r\n --save Save results to _analyze_tables table\r\n --common-limit INTEGER How many common values\r\n --no-most Skip most common values\r\n --no-least Skip least common values\r\n --load-extension TEXT Path to SQLite extension, with optional :entrypoint\r\n -h, --help Show this message and exit.\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718515590, "label": "New options for analyze-tables --common-limit --no-most and --no-least"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/542#issuecomment-1539052467", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/542", "id": 1539052467, "node_id": "IC_kwDOCGYnMM5bvBOz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T21:07:41Z", "updated_at": "2023-05-08T21:07:41Z", "author_association": "OWNER", "body": "Relevant commits (will mostly revert these):\r\n- https://github.com/simonw/sqlite-utils/commit/455c35b512895c19bf922c2b804d750d27cb8dbd\r\n- https://github.com/simonw/sqlite-utils/commit/e0ec4c345129996011951e400388fd74865f65a2", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1700936245, "label": "Remove `skip_false=True` and `--no-skip-false` in `sqlite-utils` 4.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/541#issuecomment-1538963959", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/541", "id": 1538963959, "node_id": "IC_kwDOCGYnMM5burn3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:59:34Z", "updated_at": "2023-05-08T19:59:34Z", "author_association": "OWNER", "body": "There are 8 failing tests left:\r\n\r\n```\r\n==== short test summary info ====\r\nFAILED tests/test_cli_memory.py::test_memory_csv[False-test] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>\r\nFAILED tests/test_cli_memory.py::test_memory_csv[False-t] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>\r\nFAILED tests/test_cli_memory.py::test_memory_csv[False-t1] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>\r\nFAILED tests/test_cli_memory.py::test_memory_tsv[False] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>\r\nFAILED tests/test_cli_memory.py::test_memory_dump[extra_args0] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>\r\nFAILED tests/test_cli_memory.py::test_memory_two_files_with_same_stem - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>\r\nFAILED tests/test_recipes.py::test_dateparse_errors[None-parsedate] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: .convert_value at 0x106bcaca0>\r\nFAILED tests/test_recipes.py::test_dateparse_errors[None-parsedatetime] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: .convert_value at 0x106bc9620>\r\nERROR tests/test_cli.py::test_upsert_analyze - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>\r\n==== 8 failed, 894 passed, 4 skipped, 1 error in 6.27s ====\r\n```\r\nFull traceback here: https://gist.github.com/simonw/b40b3e814729d6c02a0302a84ce54d9e", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1700840265, "label": "Get tests to pass with `pytest -Werror`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513653", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/540", "id": 1537513653, "node_id": "IC_kwDOCGYnMM5bpJi1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-07T18:37:59Z", "updated_at": "2023-05-07T18:38:19Z", "author_association": "OWNER", "body": "Useful comment here:\r\n\r\n- https://github.com/urllib3/urllib3/issues/2168#issuecomment-1537360928\r\n\r\n> I faced the same issue. I switched to a different Python kernel (3.11.2) and it worked.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1699184583, "label": "sphinx.builders.linkcheck build error"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513912", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/540", "id": 1537513912, "node_id": "IC_kwDOCGYnMM5bpJm4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-07T18:39:29Z", "updated_at": "2023-05-07T18:39:29Z", "author_association": "OWNER", "body": "https://readthedocs.org/projects/sqlite-utils/builds/20513034/ said:\r\n\r\n> Problem in your project's configuration. Invalid \"python.version\": expected one of (2, 2.7, 3, 3.5, 3.6, 3.7, 3.8, pypy3.5), got 3.11", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1699184583, "label": "sphinx.builders.linkcheck build error"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537514069", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/540", "id": 1537514069, "node_id": "IC_kwDOCGYnMM5bpJpV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-07T18:40:18Z", "updated_at": "2023-05-07T18:40:18Z", "author_association": "OWNER", "body": "https://docs.readthedocs.io/en/stable/config-file/v2.html suggests:\r\n\r\n```yaml\r\nbuild:\r\n os: ubuntu-22.04\r\n tools:\r\n python: \"3.11\"\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1699184583, "label": "sphinx.builders.linkcheck build error"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537744000", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/540", "id": 1537744000, "node_id": "IC_kwDOCGYnMM5bqByA", "user": {"value": 42327, "label": "pquentin"}, "created_at": "2023-05-08T04:56:12Z", "updated_at": "2023-05-08T04:56:12Z", "author_association": "NONE", "body": "Hey @simonw, urllib3 maintainer here :wave:\r\n\r\nSorry for breaking your CI. I understand you may prefer to pin the Python version, but note that specifying just `python: \"3\"` will get you the latest. We use that in urllib3: https://github.com/urllib3/urllib3/blob/main/.readthedocs.yml\r\n\r\nI can open PRs to sqlite-utils / datasette if you're interested", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1699184583, "label": "sphinx.builders.linkcheck build error"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/54#issuecomment-522236053", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/54", "id": 522236053, "node_id": "MDEyOklzc3VlQ29tbWVudDUyMjIzNjA1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-08-17T13:22:26Z", "updated_at": "2019-08-17T13:22:26Z", "author_association": "OWNER", "body": "Good catch: `create_view()` is an earlier utility method from before I started adding methods like `db[\"table\"].rows` and `.rows_where()`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 480961330, "label": "Ability to list views, and to access db[\"view_name\"].rows / rows_where / etc"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/54#issuecomment-524294383", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/54", "id": 524294383, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNDI5NDM4Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-08-23T12:20:15Z", "updated_at": "2019-08-23T12:20:15Z", "author_association": "OWNER", "body": "Closed by #55", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 480961330, "label": "Ability to list views, and to access db[\"view_name\"].rows / rows_where / etc"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/54#issuecomment-524294928", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/54", "id": 524294928, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNDI5NDkyOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-08-23T12:22:11Z", "updated_at": "2019-08-23T12:22:11Z", "author_association": "OWNER", "body": "Documentation is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#listing-views", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 480961330, "label": "Ability to list views, and to access db[\"view_name\"].rows / rows_where / etc"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/54#issuecomment-524297256", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/54", "id": 524297256, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNDI5NzI1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-08-23T12:30:20Z", "updated_at": "2019-08-23T12:30:20Z", "author_association": "OWNER", "body": "Released in 1.10: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v1-10", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 480961330, "label": "Ability to list views, and to access db[\"view_name\"].rows / rows_where / etc"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/54#issuecomment-524300388", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/54", "id": 524300388, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNDMwMDM4OA==", "user": {"value": 20264, "label": "ftrain"}, "created_at": "2019-08-23T12:41:09Z", "updated_at": "2019-08-23T12:41:09Z", "author_association": "NONE", "body": "Extremely cool and easy to understand. Thank you!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 480961330, "label": "Ability to list views, and to access db[\"view_name\"].rows / rows_where / etc"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507394", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/539", "id": 1537507394, "node_id": "IC_kwDOCGYnMM5bpIBC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-07T18:08:44Z", "updated_at": "2023-05-07T18:08:44Z", "author_association": "OWNER", "body": "Prototype:\r\n```diff\r\ndiff --git a/docs/cli-reference.rst b/docs/cli-reference.rst\r\nindex 153e5f9..c830518 100644\r\n--- a/docs/cli-reference.rst\r\n+++ b/docs/cli-reference.rst\r\n@@ -124,6 +124,7 @@ See :ref:`cli_query`.\r\n --json-cols Detect JSON cols and output them as JSON, not\r\n escaped strings\r\n -r, --raw Raw output, first column of first row\r\n+ --raw-lines Raw output, first column of each row\r\n -p, --param ... Named :parameters for SQL query\r\n --functions TEXT Python code defining one or more custom SQL\r\n functions\r\n@@ -192,6 +193,7 @@ See :ref:`cli_memory`.\r\n --json-cols Detect JSON cols and output them as JSON, not\r\n escaped strings\r\n -r, --raw Raw output, first column of first row\r\n+ --raw-lines Raw output, first column of each row\r\n -p, --param ... Named :parameters for SQL query\r\n --encoding TEXT Character encoding for CSV input, defaults to\r\n utf-8\r\ndiff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py\r\nindex d25b1df..da0e4b6 100644\r\n--- a/sqlite_utils/cli.py\r\n+++ b/sqlite_utils/cli.py\r\n@@ -1653,6 +1653,7 @@ def drop_view(path, view, ignore, load_extension):\r\n )\r\n @output_options\r\n @click.option(\"-r\", \"--raw\", is_flag=True, help=\"Raw output, first column of first row\")\r\n+@click.option(\"--raw-lines\", is_flag=True, help=\"Raw output, first column of each row\")\r\n @click.option(\r\n \"-p\",\r\n \"--param\",\r\n@@ -1677,6 +1678,7 @@ def query(\r\n fmt,\r\n json_cols,\r\n raw,\r\n+ raw_lines,\r\n param,\r\n load_extension,\r\n functions,\r\n@@ -1700,7 +1702,19 @@ def query(\r\n _register_functions(db, functions)\r\n \r\n _execute_query(\r\n- db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols\r\n+ db,\r\n+ sql,\r\n+ param,\r\n+ raw,\r\n+ raw_lines,\r\n+ table,\r\n+ csv,\r\n+ tsv,\r\n+ no_headers,\r\n+ fmt,\r\n+ nl,\r\n+ arrays,\r\n+ json_cols,\r\n )\r\n \r\n \r\n@@ -1728,6 +1742,7 @@ def query(\r\n )\r\n @output_options\r\n @click.option(\"-r\", \"--raw\", is_flag=True, help=\"Raw output, first column of first row\")\r\n+@click.option(\"--raw-lines\", is_flag=True, help=\"Raw output, first column of each row\")\r\n @click.option(\r\n \"-p\",\r\n \"--param\",\r\n@@ -1773,6 +1788,7 @@ def memory(\r\n fmt,\r\n json_cols,\r\n raw,\r\n+ raw_lines,\r\n param,\r\n encoding,\r\n no_detect_types,\r\n@@ -1879,12 +1895,36 @@ def memory(\r\n _register_functions(db, functions)\r\n \r\n _execute_query(\r\n- db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols\r\n+ db,\r\n+ sql,\r\n+ param,\r\n+ raw,\r\n+ raw_lines,\r\n+ table,\r\n+ csv,\r\n+ tsv,\r\n+ no_headers,\r\n+ fmt,\r\n+ nl,\r\n+ arrays,\r\n+ json_cols,\r\n )\r\n \r\n \r\n def _execute_query(\r\n- db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols\r\n+ db,\r\n+ sql,\r\n+ param,\r\n+ raw,\r\n+ raw_lines,\r\n+ table,\r\n+ csv,\r\n+ tsv,\r\n+ no_headers,\r\n+ fmt,\r\n+ nl,\r\n+ arrays,\r\n+ json_cols,\r\n ):\r\n with db.conn:\r\n try:\r\n@@ -1903,6 +1943,13 @@ def _execute_query(\r\n sys.stdout.buffer.write(data)\r\n else:\r\n sys.stdout.write(str(data))\r\n+ elif raw_lines:\r\n+ for row in cursor:\r\n+ data = row[0]\r\n+ if isinstance(data, bytes):\r\n+ sys.stdout.buffer.write(data + b\"\\n\")\r\n+ else:\r\n+ sys.stdout.write(str(data) + \"\\n\")\r\n elif fmt or table:\r\n print(\r\n tabulate.tabulate(\r\n```\r\nNeeds tests and more documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1699174055, "label": "`--raw-lines` option, like `--raw` for multiple lines"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507525", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/539", "id": 1537507525, "node_id": "IC_kwDOCGYnMM5bpIDF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-07T18:09:09Z", "updated_at": "2023-05-07T18:09:09Z", "author_association": "OWNER", "body": "I'm tempted to upgrade `--raw` to do this instead, but that would be a breaking change.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1699174055, "label": "`--raw-lines` option, like `--raw` for multiple lines"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507676", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/539", "id": 1537507676, "node_id": "IC_kwDOCGYnMM5bpIFc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-07T18:09:43Z", "updated_at": "2023-05-07T18:09:54Z", "author_association": "OWNER", "body": "This worked:\r\n\r\n```bash\r\nsqlite-utils memory /tmp/books3.json:nl \\\r\n 'select name from books3' --raw-lines > titles.txt\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1699174055, "label": "`--raw-lines` option, like `--raw` for multiple lines"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537514610", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/539", "id": 1537514610, "node_id": "IC_kwDOCGYnMM5bpJxy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-07T18:43:24Z", "updated_at": "2023-05-07T18:43:24Z", "author_association": "OWNER", "body": "Documentation:\r\n- https://sqlite-utils.datasette.io/en/latest/cli.html#returning-raw-data-such-as-binary-content\r\n- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query\r\n- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#memory", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1699174055, "label": "`--raw-lines` option, like `--raw` for multiple lines"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538793817", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538793817, "node_id": "IC_kwDOCGYnMM5buCFZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T17:55:10Z", "updated_at": "2023-05-08T17:55:10Z", "author_association": "OWNER", "body": "Confirmed - I added this test and it fails:\r\n```python\r\ndef test_upsert_all_not_null(fresh_db):\r\n # https://github.com/simonw/sqlite-utils/issues/538\r\n fresh_db[\"comments\"].upsert_all(\r\n [{\"id\": 1, \"name\": \"Cleo\"}],\r\n pk=\"id\",\r\n not_null=[\"name\"],\r\n )\r\n assert list(fresh_db[\"comments\"].rows) == [{\"id\": 1, \"name\": \"Cleo\"}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538801855", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538801855, "node_id": "IC_kwDOCGYnMM5buEC_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T18:00:17Z", "updated_at": "2023-05-08T18:00:17Z", "author_association": "OWNER", "body": "From time in the debugger, after creating the table it ends up doing this:\r\n```\r\n(Pdb) queries_and_params\r\n[\r\n ('INSERT OR IGNORE INTO [comments]([id]) VALUES(?);', [1]),\r\n ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1])\r\n]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538887361", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538887361, "node_id": "IC_kwDOCGYnMM5buY7B", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:01:20Z", "updated_at": "2023-05-08T19:01:20Z", "author_association": "OWNER", "body": "Here's the problem:\r\n```python\r\nimport sqlite3\r\ndb = sqlite3.connect(\":memory:\")\r\ndb.execute('create table foo (id integer primary key, name not null)')\r\ndb.execute('insert into foo (id) values (1)')\r\n```\r\nProduces:\r\n```\r\nIntegrityError: NOT NULL constraint failed: foo.name\r\n```\r\nBut this:\r\n```python\r\ndb.execute('insert or ignore into foo (id) values (1)')\r\n```\r\nCompletes without an exception.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538889482", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538889482, "node_id": "IC_kwDOCGYnMM5buZcK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:02:38Z", "updated_at": "2023-05-08T19:02:38Z", "author_association": "OWNER", "body": "Here's the code at fault:\r\nhttps://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2774-L2788", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538893329", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538893329, "node_id": "IC_kwDOCGYnMM5buaYR", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:04:47Z", "updated_at": "2023-05-08T19:04:47Z", "author_association": "OWNER", "body": "This feels like a fundamental flaw in the way upserts are implemented by `sqlite-utils`.\r\n\r\nOne fix would be to switch to using the `UPSERT` feature in SQLite: https://www.sqlite.org/lang_UPSERT.html\r\n\r\nBut...\r\n\r\n> UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04).\r\n\r\nI still want to support SQLite versions earlier than that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538903556", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538903556, "node_id": "IC_kwDOCGYnMM5buc4E", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:11:24Z", "updated_at": "2023-05-08T19:13:23Z", "author_association": "OWNER", "body": "I could detect if this happens using `cursor.rowcount` - not sure how I would recover from it though.\r\n\r\nThis would also require some major re-engineering, since currently it all works by generating a list of SQL queries in advance and applying them inside a loop in `.insert_chunk()`:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2839-L2878\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538910894", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538910894, "node_id": "IC_kwDOCGYnMM5buequ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:16:52Z", "updated_at": "2023-05-08T19:17:00Z", "author_association": "OWNER", "body": "How about if I had logic which checked that all not-null columns were provided in the call to `upsert_all()` - and if they were, modified the `INSERT OR IGNORE INTO` to include a placeholder value for those columns that would then be fixed by the later `UPDATE`?\r\n\r\nSomething like this:\r\n\r\n```python\r\n[\r\n ('INSERT OR IGNORE INTO [comments]([id], name) VALUES(?, ?);', [1, '']),\r\n ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1])\r\n]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538921774", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538921774, "node_id": "IC_kwDOCGYnMM5buhUu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:24:41Z", "updated_at": "2023-05-08T19:24:41Z", "author_association": "OWNER", "body": "That fix seems to work!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null}