{"id": 1067771698, "node_id": "I_kwDOCGYnMM4_pOcy", "number": 348, "title": "Command for creating an empty database", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 6, "created_at": "2021-11-30T23:24:27Z", "updated_at": "2022-01-13T07:06:59Z", "closed_at": "2022-01-09T20:33:20Z", "author_association": "OWNER", "pull_request": null, "body": "I sometimes find the need to create an empty SQLite database file - for example if I want to enable WAL on it before using it with another script. I currently do that like this:\r\n\r\n    sqlite3 my.db vacuum\r\n    sqlite-utils enable-wal my.db\r\n\r\nIt would be nice if `sqlite-utils` had a convenience command for doing this.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/348/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1095570074, "node_id": "I_kwDOCGYnMM5BTRKa", "number": 364, "title": "`--batch-size 1` doesn't seem to commit for every item", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 16, "created_at": "2022-01-06T18:18:50Z", "updated_at": "2022-01-10T19:27:17Z", "closed_at": "2022-01-10T05:36:19Z", "author_association": "OWNER", "pull_request": null, "body": "I'm trying this, but it doesn't seem to write anything to the database file until I hit `CTRL+C`:\r\n\r\n```\r\nheroku logs --app=simonwillisonblog --tail | grep 'measure#nginx.service' | \\\r\n  sqlite-utils insert /tmp/herokutail.db log - --import re --convert \"$(cat <<EOD\r\n    r = re.compile(r'([^\\s=]+)=(?:\"(.*?)\"|(\\S+))')\r\n    pairs = {}\r\n    for key, value1, value2 in r.findall(line):\r\n        pairs[key] = value1 or value2\r\n    return pairs\r\nEOD\r\n)\" --lines --batch-size 1\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/364/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1096558279, "node_id": "I_kwDOCGYnMM5BXCbH", "number": 365, "title": "create-index should run analyze after creating index", "user": {"value": 536941, "label": "fgregg"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 16, "created_at": "2022-01-07T18:21:25Z", "updated_at": "2022-01-11T02:43:34Z", "closed_at": "2022-01-11T01:36:48Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "sqlite's query planner depends upon analyze to make good use of indices. It would be nice if analyze was run as part of the create-index command.\r\n\r\nIf data is inserted later, things can get out date, but it would still probably be a net win. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/365/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1096563265, "node_id": "I_kwDOCGYnMM5BXDpB", "number": 366, "title": "Python library methods for calling ANALYZE", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 10, "created_at": "2022-01-07T18:28:01Z", "updated_at": "2022-01-11T01:09:33Z", "closed_at": "2022-01-11T01:09:33Z", "author_association": "OWNER", "pull_request": null, "body": "> Relevant documentation: https://www.sqlite.org/lang_analyze.html\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007633376_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/366/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1097041471, "node_id": "PR_kwDOCGYnMM4wsVM6", "number": 367, "title": "Initial prototype of .analyze() methods", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 2, "created_at": "2022-01-08T21:35:12Z", "updated_at": "2022-01-10T19:31:08Z", "closed_at": "2022-01-10T19:31:08Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/367", "body": "Refs #366", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/367/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 1, "state_reason": null}
{"id": 1097087280, "node_id": "I_kwDOCGYnMM5BZDkw", "number": 368, "title": "Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 3, "created_at": "2022-01-09T02:29:30Z", "updated_at": "2022-01-10T19:27:20Z", "closed_at": "2022-01-09T02:40:50Z", "author_association": "OWNER", "pull_request": null, "body": "> Add this to `sqlite_utils/cli.py`:\r\n>\r\n> ```python\r\n> if __name__ == \"__main__\":\r\n>     cli()\r\n> ```\r\n> Now the tool can be run using `python -m sqlite_utils.cli --help`\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/368/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1097128334, "node_id": "I_kwDOCGYnMM5BZNmO", "number": 371, "title": "Support mutating row in `--convert` without returning it", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 6, "created_at": "2022-01-09T07:38:44Z", "updated_at": "2022-01-10T19:27:30Z", "closed_at": "2022-01-09T20:06:15Z", "author_association": "OWNER", "pull_request": null, "body": "Currently you have to do this:\r\n```\r\n$ sqlite-utils insert dogs.db dogs dogs.json --convert '\r\nrow[\"is_good\"] = 1\r\nreturn row'\r\n```\r\nWould be neat if this worked too:\r\n```\r\n$ sqlite-utils insert dogs.db dogs dogs.json \\\r\n  --convert 'row[\"is_good\"] = 1'\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/371/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1097129710, "node_id": "I_kwDOCGYnMM5BZN7u", "number": 372, "title": "Idea: `suffix` and `stem` file columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 1, "created_at": "2022-01-09T07:48:53Z", "updated_at": "2022-01-10T19:27:34Z", "closed_at": "2022-01-09T20:17:00Z", "author_association": "OWNER", "pull_request": null, "body": "For https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files\r\n\r\nGiven a file called `dogs.jpg` stem would be `dogs` and ext would be `jpg`.\r\n\r\nNeed to decide what happens for `dogs.and.cats.jpg.gz`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/372/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1097135732, "node_id": "I_kwDOCGYnMM5BZPZ0", "number": 373, "title": "List `--fmt` options in the docs ", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 3, "created_at": "2022-01-09T08:22:11Z", "updated_at": "2022-01-10T19:27:24Z", "closed_at": "2022-01-09T17:49:00Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite-utils.datasette.io/en/stable/cli.html#table-formatted-output currently cheats and tells the user to run `--help` - can fix this using `cog`. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/373/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1097135860, "node_id": "I_kwDOCGYnMM5BZPb0", "number": 374, "title": "`--fmt` should imply `-t`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 4, "created_at": "2022-01-09T08:23:07Z", "updated_at": "2022-01-10T19:27:26Z", "closed_at": "2022-01-09T18:07:59Z", "author_association": "OWNER", "pull_request": null, "body": "Not sure why I didn't implement this.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/374/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1097251014, "node_id": "I_kwDOCGYnMM5BZrjG", "number": 375, "title": "`sqlite-utils bulk` command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 3, "created_at": "2022-01-09T17:12:38Z", "updated_at": "2022-01-11T02:12:58Z", "closed_at": "2022-01-11T02:10:55Z", "author_association": "OWNER", "pull_request": null, "body": "The `.executemany()` method is a very efficient way to execute the same SQL query against a huge list of parameters.\r\n\r\n`sqlite-utils insert` supports a bunch of ways of loading a list of dictionaries - from CSV, TSV, JSON, newline JSON and more thanks to:\r\n- #361\r\n\r\nWhat if you could load a list of dictionaries and provide a SQL query with `:named` parameters that correspond to keys in those dictionaries instead?\r\n\r\nThis would need to be a new command - I thought about adding a `--sql` option to `insert` but that doesn't make sense as that command already requires a table name.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/375/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1097436959, "node_id": "I_kwDOCGYnMM5BaY8f", "number": 376, "title": "`--nl` mode should ignore blank lines", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 0, "created_at": "2022-01-10T04:10:54Z", "updated_at": "2022-01-10T19:27:41Z", "closed_at": "2022-01-10T04:12:46Z", "author_association": "OWNER", "pull_request": null, "body": "Spotted this while manually testing #364 - there's no reason `--nl` should crash if you feed it an empty line in between JSON objects.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/376/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1097477582, "node_id": "PR_kwDOCGYnMM4wtl17", "number": 377, "title": "`sqlite-utils bulk` command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 3, "created_at": "2022-01-10T05:34:24Z", "updated_at": "2022-01-11T02:10:57Z", "closed_at": "2022-01-11T02:10:54Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/377", "body": "Refs #375\r\n\r\nStill needs:\r\n\r\n- [x] Refactor `@insert_upsert_options` so that it doesn't duplicate `@import_options`\r\n- [x] Tests\r\n- [x] Documentation\r\n- [x] Try it against a really big file", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/377/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1098309897, "node_id": "I_kwDOCGYnMM5BduEJ", "number": 378, "title": "analyze=True parameter for some methods", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 0, "created_at": "2022-01-10T19:54:52Z", "updated_at": "2022-01-11T01:08:11Z", "closed_at": "2022-01-11T01:08:09Z", "author_association": "OWNER", "pull_request": null, "body": "This would cause `ANALYZE` to be run against the relevant table at the end of executing the method.\r\n\r\n> Having browsed the API reference I think the methods that would benefit from an `analyze=True` parameter are:\r\n\r\n- [x] `table.create_index`\r\n- [x] `table.insert_all`\r\n- [x] `table.upsert_all`\r\n- [x] `table.delete_where`\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/378/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1098544628, "node_id": "I_kwDOCGYnMM5BenX0", "number": 379, "title": "CLI options for running ANALYZE", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 0, "created_at": "2022-01-11T01:09:16Z", "updated_at": "2022-01-11T01:38:01Z", "closed_at": "2022-01-11T01:36:48Z", "author_association": "OWNER", "pull_request": null, "body": "> The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865_\r\n\r\n- [x] `sqlite-utils analyze` command\r\n- [x] `sqlite-utils create-index --analyze` option (see #365)\r\n- [x] `sqlite-utils insert --analyze` option\r\n- [x] `sqlite-utils upsert --analyze` option\r\n\r\nIn #378 I also added `.delete_where(..., analyze=True)` but there isn't currently a `sqlite-utils delete-where` CLI command - deletions via CLI are expected to be handled using SQL queries.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/379/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1098574572, "node_id": "I_kwDOCGYnMM5Beurs", "number": 380, "title": "Release notes for 3.21", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 1, "created_at": "2022-01-11T02:12:30Z", "updated_at": "2022-01-11T02:34:26Z", "closed_at": "2022-01-11T02:34:26Z", "author_association": "OWNER", "pull_request": null, "body": "For these commits: https://github.com/simonw/sqlite-utils/compare/3.20...129141572f249ea290e2a075437e2ebaad215859", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/380/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}