html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394619,https://api.github.com/repos/simonw/sqlite-utils/issues/96,606394619,MDEyOklzc3VlQ29tbWVudDYwNjM5NDYxOQ==,9599,simonw,2020-03-31T04:38:17Z,2020-03-31T04:40:23Z,OWNER,I wonder if there are any other Pandas conversions we should be doing? https://pandas.pydata.org/pandas-docs/stable/getting_started/basics.html#dtypes ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",589801352,Add type conversion for Panda's Timestamp, https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394349,https://api.github.com/repos/simonw/sqlite-utils/issues/96,606394349,MDEyOklzc3VlQ29tbWVudDYwNjM5NDM0OQ==,9599,simonw,2020-03-31T04:37:16Z,2020-03-31T04:37:16Z,OWNER,Test failure was just a Black formatting issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",589801352,Add type conversion for Panda's Timestamp, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580584269,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580584269,MDEyOklzc3VlQ29tbWVudDU4MDU4NDI2OQ==,9599,simonw,2020-01-31T05:08:04Z,2020-01-31T05:08:04Z,OWNER,Ditching this since it won't actually solve my problem.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819,on_create mechanism for after table creation, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580569059,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580569059,MDEyOklzc3VlQ29tbWVudDU4MDU2OTA1OQ==,9599,simonw,2020-01-31T03:48:41Z,2020-01-31T03:48:41Z,OWNER,"This may not be the right feature after all, see https://github.com/simonw/geojson-to-sqlite/issues/6#issuecomment-580569002","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819,on_create mechanism for after table creation, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567886,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580567886,MDEyOklzc3VlQ29tbWVudDU4MDU2Nzg4Ng==,9599,simonw,2020-01-31T03:41:31Z,2020-01-31T03:41:31Z,OWNER,I think it does make sense to be able to pass it to the `.table()` constructor.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819,on_create mechanism for after table creation, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567604,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580567604,MDEyOklzc3VlQ29tbWVudDU4MDU2NzYwNA==,9599,simonw,2020-01-31T03:39:58Z,2020-01-31T03:39:58Z,OWNER,Perhaps this should be called `after_create` instead of `on_create`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819,on_create mechanism for after table creation, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567505,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580567505,MDEyOklzc3VlQ29tbWVudDU4MDU2NzUwNQ==,9599,simonw,2020-01-31T03:39:19Z,2020-01-31T03:39:19Z,OWNER,"Still needs documentation and tests. Also I'm not certain that this should be an argument you can pass to the `.table()` constructor, need to think that over.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819,on_create mechanism for after table creation, https://github.com/simonw/sqlite-utils/pull/75#issuecomment-580523995,https://api.github.com/repos/simonw/sqlite-utils/issues/75,580523995,MDEyOklzc3VlQ29tbWVudDU4MDUyMzk5NQ==,9599,simonw,2020-01-31T00:21:11Z,2020-01-31T00:21:11Z,OWNER,"This makes sense, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546078359,Explicitly include tests and docs in sdist, https://github.com/simonw/sqlite-utils/pull/67#issuecomment-569844320,https://api.github.com/repos/simonw/sqlite-utils/issues/67,569844320,MDEyOklzc3VlQ29tbWVudDU2OTg0NDMyMA==,9599,simonw,2019-12-31T01:29:43Z,2019-12-31T01:29:43Z,OWNER,I don't really care about 3.5 any more.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",529376481,Run tests against 3.5 too, https://github.com/simonw/sqlite-utils/pull/67#issuecomment-559108591,https://api.github.com/repos/simonw/sqlite-utils/issues/67,559108591,MDEyOklzc3VlQ29tbWVudDU1OTEwODU5MQ==,9599,simonw,2019-11-27T14:24:59Z,2019-11-27T14:24:59Z,OWNER,Failed due to black testing dependency: https://travis-ci.com/simonw/sqlite-utils/jobs/260995814,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",529376481,Run tests against 3.5 too, https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527266798,https://api.github.com/repos/simonw/sqlite-utils/issues/57,527266798,MDEyOklzc3VlQ29tbWVudDUyNzI2Njc5OA==,9599,simonw,2019-09-03T01:03:59Z,2019-09-03T01:03:59Z,OWNER,Released in 1.11 - thanks again! https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-11,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487987958,Add triggers while enabling FTS, https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527258212,https://api.github.com/repos/simonw/sqlite-utils/issues/57,527258212,MDEyOklzc3VlQ29tbWVudDUyNzI1ODIxMg==,9599,simonw,2019-09-02T23:40:56Z,2019-09-02T23:40:56Z,OWNER,"This is fantastic, thanks so much. I spotted a Datasette bug triggered by this: Datasette has its own version of the `detect_fts` function - at https://github.com/simonw/datasette/blob/d224ee2c98ac39c2c6e21a0ac0c62e5c3e1ccd11/datasette/utils/__init__.py#L466-L479 - which fails to pick up FTS tables created using the new escaping pattern. It's a bug in Datasette, not sqlite-utils - so I'll open an issue there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487987958,Add triggers while enabling FTS, https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527211047,https://api.github.com/repos/simonw/sqlite-utils/issues/57,527211047,MDEyOklzc3VlQ29tbWVudDUyNzIxMTA0Nw==,49260,amjith,2019-09-02T17:30:43Z,2019-09-02T17:30:43Z,CONTRIBUTOR,"I have merged the other PR (#56) into this one. I have incorporated your suggestions. Cheers!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487987958,Add triggers while enabling FTS, https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527091278,https://api.github.com/repos/simonw/sqlite-utils/issues/57,527091278,MDEyOklzc3VlQ29tbWVudDUyNzA5MTI3OA==,9599,simonw,2019-09-02T10:13:32Z,2019-09-02T10:13:32Z,OWNER,"This is brilliant! One tiny suggestion: I like `--create-triggers` and `create_triggers=True` better for this as they are shorter but still capture what it dies - especially since the underlying SQL uses `CREATE TRIGGER` statements.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487987958,Add triggers while enabling FTS, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606315321,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606315321,IC_kwDOCGYnMM5fvm05,9599,simonw,2023-06-25T23:18:33Z,2023-06-25T23:18:33Z,OWNER,Documentation preview: https://sqlite-utils--560.org.readthedocs.build/en/560/installation.html#alternatives-to-sqlite3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606310630,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606310630,IC_kwDOCGYnMM5fvlrm,9599,simonw,2023-06-25T23:06:07Z,2023-06-25T23:06:07Z,OWNER,"Filed an issue about the above with `pysqlite3` (which `sqlean.py` is based on) here: - https://github.com/coleifer/pysqlite3/issues/58","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606297356,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606297356,IC_kwDOCGYnMM5fvicM,9599,simonw,2023-06-25T22:42:41Z,2023-06-25T22:42:41Z,OWNER,"Yes that does seem to do the trick: ```pycon >>> import sqlean >>> db = sqlean.connect(""/tmp/4.db"") >>> db.execute('PRAGMA journal_mode;').fetchall() [('delete',)] >>> db.isolation_level '' >>> db.execute('PRAGMA journal_mode=wal;') Traceback (most recent call last): File """", line 1, in sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction >>> db.isolation_level = None >>> db.isolation_level >>> db.execute('PRAGMA journal_mode=wal;') ``` Weird how `isolation_level` of empty string causes the error, but setting that to `None` fixes the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606294627,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606294627,IC_kwDOCGYnMM5fvhxj,9599,simonw,2023-06-25T22:40:10Z,2023-06-25T22:40:10Z,OWNER,I suspect this has something to do with `autocommit` mode in `sqlite3` - which I may be able to turn off by setting `con.isolation_level = None`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606293382,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606293382,IC_kwDOCGYnMM5fvheG,9599,simonw,2023-06-25T22:34:47Z,2023-06-25T22:34:47Z,OWNER,"```pycon >>> import sqlite3 >>> db = sqlite3.connect(""/tmp/1.db"") >>> db.execute('PRAGMA journal_mode=wal;') >>> import sqlean >>> db2 = sqlean.connect(""/tmp/2.db"") >>> db2.execute('PRAGMA journal_mode=wal;') Traceback (most recent call last): File """", line 1, in sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606290917,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606290917,IC_kwDOCGYnMM5fvg3l,9599,simonw,2023-06-25T22:32:28Z,2023-06-25T22:32:28Z,OWNER,"I've fixed most of the test failures, but I still need to fix this one: > cannot change into wal mode from within a transaction","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606273005,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606273005,IC_kwDOCGYnMM5fvcft,9599,simonw,2023-06-25T21:47:47Z,2023-06-25T21:47:47Z,OWNER,I can use https://github.com/simonw/sqlite-dump as an optional dependency to handle the missing `.iterdump()` method.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270887,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606270887,IC_kwDOCGYnMM5fvb-n,9599,simonw,2023-06-25T21:37:12Z,2023-06-26T08:21:00Z,OWNER,"On my own laptop I got a crash running the tests - details here: - https://github.com/nalgeon/sqlean.py/issues/3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270055,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606270055,IC_kwDOCGYnMM5fvbxn,9599,simonw,2023-06-25T21:31:56Z,2023-06-25T21:31:56Z,OWNER,"Lots of failures now that I'm trying to run the tests against `sqlean.py` on macOS and Python 3.10: https://github.com/simonw/sqlite-utils/actions/runs/5371800108/jobs/9744802953 A bunch of these, because `pysqlite3` chooses not to implement `.iterdump()`: ``` @pytest.fixture def db_to_analyze_path(db_to_analyze, tmpdir): path = str(tmpdir / ""test.db"") db = sqlite3.connect(path) > db.executescript(""\n"".join(db_to_analyze.conn.iterdump())) E AttributeError: 'sqlean.dbapi2.Connection' object has no attribute 'iterdump' ``` Also some of these: ``` def test_analyze_whole_database(db): assert set(db.table_names()) == {""one_index"", ""two_indexes""} db.analyze() > assert set(db.table_names()) == {""one_index"", ""two_indexes"", ""sqlite_stat1""} E AssertionError: assert {'one_index',...'two_indexes'} == {'one_index',...'two_indexes'} E Extra items in the left set: E 'sqlite_stat4' E Full diff: E - {'two_indexes', 'sqlite_stat1', 'one_index'} E + {'two_indexes', 'sqlite_stat1', 'sqlite_stat4', 'one_index'} E ? ++++++++++++++++ ``` Apparently `sqlean.py` adds a `sqlite_stat4` table that the tests are not expecting. Plus some errors that look like this: ``` def test_enable_wal(): runner = CliRunner() dbs = [""test.db"", ""test2.db""] with runner.isolated_filesystem(): for dbname in dbs: db = Database(dbname) db[""t""].create({""pk"": int}, pk=""pk"") assert db.journal_mode == ""delete"" result = runner.invoke(cli.cli, [""enable-wal""] + dbs) > assert 0 == result.exit_code E AssertionError: assert 0 == 1 E + where 1 = .exit_code ``` Test summary: ``` ============ 13 failed, 909 passed, 16 skipped, 2 errors in 19.29s ============= ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606237836,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606237836,IC_kwDOCGYnMM5fvT6M,22429695,codecov[bot],2023-06-25T19:49:45Z,2023-06-26T08:20:59Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`80.55`**% and project coverage change: **`-0.15`** :warning: > Comparison is base [(`2747257`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/2747257a3334d55e890b40ec58fada57ae8cfbfd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`5e7d27e`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.22%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #560 +/- ## ========================================== - Coverage 96.36% 96.22% -0.15% ========================================== Files 6 6 Lines 2726 2752 +26 ========================================== + Hits 2627 2648 +21 - Misses 99 104 +5 ``` | [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/utils.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.56% <63.63%> (-0.62%)` | :arrow_down: | | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.33% <86.36%> (-0.20%)` | :arrow_down: | | [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <100.00%> (ø)` | |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/56#issuecomment-527209840,https://api.github.com/repos/simonw/sqlite-utils/issues/56,527209840,MDEyOklzc3VlQ29tbWVudDUyNzIwOTg0MA==,49260,amjith,2019-09-02T17:23:21Z,2019-09-02T17:23:21Z,CONTRIBUTOR,"I have updated the other PR with the changes from this one and added tests. I have also changed the escaping from double quotes to brackets. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487847945,Escape the table name in populate_fts and search., https://github.com/simonw/sqlite-utils/pull/56#issuecomment-527090411,https://api.github.com/repos/simonw/sqlite-utils/issues/56,527090411,MDEyOklzc3VlQ29tbWVudDUyNzA5MDQxMQ==,9599,simonw,2019-09-02T10:10:48Z,2019-09-02T10:10:48Z,OWNER,"Good spot, thanks! Would be useful to add a test as well, derived from your above example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487847945,Escape the table name in populate_fts and search., https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556288300,https://api.github.com/repos/simonw/sqlite-utils/issues/553,1556288300,IC_kwDOCGYnMM5cwxMs,9599,simonw,2023-05-21T20:48:01Z,2023-05-21T20:48:01Z,OWNER,If https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries looks good I can merge this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718635018,Reformatted CLI examples in docs, https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556287870,https://api.github.com/repos/simonw/sqlite-utils/issues/553,1556287870,IC_kwDOCGYnMM5cwxF-,22429695,codecov[bot],2023-05-21T20:45:58Z,2023-05-21T20:57:08Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`e240133`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/e240133b11588d31dc22c632f7a7ca636c72978d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`0b81794`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36%. > :exclamation: Current head 0b81794 differs from pull request most recent head 21036a5. Consider uploading reports for the commit 21036a5 to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #553 +/- ## ======================================= Coverage 96.36% 96.36% ======================================= Files 6 6 Lines 2726 2726 ======================================= Hits 2627 2627 Misses 99 99 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718635018,Reformatted CLI examples in docs, https://github.com/simonw/sqlite-utils/pull/55#issuecomment-522238063,https://api.github.com/repos/simonw/sqlite-utils/issues/55,522238063,MDEyOklzc3VlQ29tbWVudDUyMjIzODA2Mw==,9599,simonw,2019-08-17T13:42:40Z,2019-08-17T13:42:40Z,OWNER,"I still need to implement the part of this where certain actions against views (like updating/inserting) are not allowed. I think the better way to do that is to introduce a new class - `View` - rather than reusing the existing `Table` class - since there's a lot of stuff on Table (like primary key / foreign key introspection) which simply doesn't make sense for views. This will get a bit more complicated - maybe I need both Table and View to extend a common subclass here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",481887482,Ability to introspect and run queries against views, https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556242262,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556242262,IC_kwDOCGYnMM5cwl9W,9599,simonw,2023-05-21T18:00:05Z,2023-05-21T18:00:05Z,OWNER,"Failing `mypy` test: https://github.com/simonw/sqlite-utils/actions/runs/5038983349/jobs/9036828465 ``` sqlite_utils/cli.py:37: error: Skipping analyzing ""trogon"": module is installed, but missing library stubs or py.typed marker [import] sqlite_utils/cli.py:37: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 52 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon, https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556241812,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556241812,IC_kwDOCGYnMM5cwl2U,9599,simonw,2023-05-21T17:58:25Z,2023-05-21T17:58:25Z,OWNER,Documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon, https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556241555,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556241555,IC_kwDOCGYnMM5cwlyT,22429695,codecov[bot],2023-05-21T17:57:24Z,2023-05-21T18:28:44Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`83.33`**% and project coverage change: **`+0.06`** :tada: > Comparison is base [(`b3b100d`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/b3b100d7f5b2a76ccd4bfe8b0301a29e321d0375?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.30% compared to head [(`948692a`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #549 +/- ## ========================================== + Coverage 96.30% 96.36% +0.06% ========================================== Files 6 6 Lines 2707 2726 +19 ========================================== + Hits 2607 2627 +20 + Misses 100 99 -1 ``` | [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <83.33%> (-0.03%)` | :arrow_down: | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon, https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213396,https://api.github.com/repos/simonw/sqlite-utils/issues/546,1556213396,IC_kwDOCGYnMM5cwe6U,9599,simonw,2023-05-21T15:58:12Z,2023-05-21T16:18:46Z,OWNER,"Documentation preview: - https://sqlite-utils--546.org.readthedocs.build/en/546/cli.html#cli-analyze-tables - https://sqlite-utils--546.org.readthedocs.build/en/546/cli-reference.html#analyze-tables - https://sqlite-utils--546.org.readthedocs.build/en/546/python-api.html#analyzing-a-column - https://sqlite-utils--546.org.readthedocs.build/en/546/reference.html#sqlite_utils.db.Table.analyze_column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718550688,"Analyze tables options: --common-limit, --no-most, --no-least", https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213031,https://api.github.com/repos/simonw/sqlite-utils/issues/546,1556213031,IC_kwDOCGYnMM5cwe0n,22429695,codecov[bot],2023-05-21T15:56:05Z,2023-05-21T16:18:03Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`93.75`**% and no project coverage change. > Comparison is base [(`b3b100d`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/b3b100d7f5b2a76ccd4bfe8b0301a29e321d0375?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.30% compared to head [(`9f23e68`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.31%. > :exclamation: Current head 9f23e68 differs from pull request most recent head 2eca17d. Consider uploading reports for the commit 2eca17d to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #546 +/- ## ======================================= Coverage 96.30% 96.31% ======================================= Files 6 6 Lines 2707 2712 +5 ======================================= + Hits 2607 2612 +5 Misses 100 100 ``` | [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.37% <90.90%> (+<0.01%)` | :arrow_up: | | [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.26% <100.00%> (+0.01%)` | :arrow_up: |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718550688,"Analyze tables options: --common-limit, --no-most, --no-least", https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539157643,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1539157643,IC_kwDOCGYnMM5bva6L,9599,simonw,2023-05-08T22:45:09Z,2023-05-08T22:45:21Z,OWNER,"Here's an example from the new tests: https://github.com/simonw/sqlite-utils/blob/a75abeb61b91a28650d3b9933e7ec80ad0d92529/tests/test_create.py#L291-L307","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`, https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539055393,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1539055393,IC_kwDOCGYnMM5bvB8h,9599,simonw,2023-05-08T21:10:06Z,2023-05-08T21:10:06Z,OWNER,Thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`, https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1506200813,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1506200813,IC_kwDOCGYnMM5Zxszt,22429695,codecov[bot],2023-04-13T01:45:22Z,2023-04-13T01:45:22Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`100.00`**% and no project coverage change. > Comparison is base [(`c0251cc`)](https://codecov.io/gh/simonw/sqlite-utils/commit/c0251cc9271260de73b4227859a51fab9b4cb745?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25% compared to head [(`a75abeb`)](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #537 +/- ## ======================================= Coverage 96.25% 96.25% ======================================= Files 6 6 Lines 2671 2673 +2 ======================================= + Hits 2571 2573 +2 Misses 100 100 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.33% <100.00%> (+<0.01%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`, https://github.com/simonw/sqlite-utils/pull/536#issuecomment-1500893216,https://api.github.com/repos/simonw/sqlite-utils/issues/536,1500893216,IC_kwDOCGYnMM5ZddAg,22429695,codecov[bot],2023-04-08T13:35:42Z,2023-04-08T13:35:42Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`c0251cc`)](https://codecov.io/gh/simonw/sqlite-utils/commit/c0251cc9271260de73b4227859a51fab9b4cb745?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25% compared to head [(`cea05dc`)](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #536 +/- ## ======================================= Coverage 96.25% 96.25% ======================================= Files 6 6 Lines 2671 2671 ======================================= Hits 2571 2571 Misses 100 100 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `95.13% <ø> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1659525418,Add paths for homebrew on Apple silicon, https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1501017004,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1501017004,IC_kwDOCGYnMM5Zd7Os,25778,eyeseast,2023-04-09T01:49:43Z,2023-04-09T01:49:43Z,CONTRIBUTOR,I'm going to close this in favor of #536. Will try a cleaner approach to custom paths once that one is merge.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon, https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465315726,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1465315726,IC_kwDOCGYnMM5XVvGO,25778,eyeseast,2023-03-12T22:21:56Z,2023-03-12T22:21:56Z,CONTRIBUTOR,"Exactly, that's what I was running into. On my M2 MacBook, SpatiaLite ends up in what is -- for the moment -- a non-standard location, so even when I passed in the location with `--load-extension`, I still hit an error on `create-spatial-index`. What I learned doing this originally is that SQLite needs to load the extension for each connection, even if all the SpatiaLite stuff is already in the database. So that's why `init_spatialite()` gets called again. Here's the code where I hit the error: https://github.com/eyeseast/boston-parcels/blob/main/Makefile#L30 It works using this branch. I'm not attached to this solution if you can think of something better. And I'm not sure, TBH, my test would actually catch what I'm after here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon, https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465302343,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1465302343,IC_kwDOCGYnMM5XVr1H,9599,simonw,2023-03-12T21:19:13Z,2023-03-12T21:19:13Z,OWNER,"Aah, I think I see why you wrote it like that. The problem is that `init_spatialite()` does other stuff too: https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L1161-L1171 So it needs to be able to load the SpatiaLite extension from the correct place, and THEN run `select InitSpatialMetadata()` to configure the database, if needed. So the problem you're trying to solve here is to let people optionally pass in the path to SpatiaLite if it's not one of the ones that are searched by default. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon, https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465038901,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1465038901,IC_kwDOCGYnMM5XUrg1,22429695,codecov[bot],2023-03-11T22:29:19Z,2023-04-07T18:19:49Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`100.00`**% and project coverage change: **`+0.03`** :tada: > Comparison is base [(`c0251cc`)](https://codecov.io/gh/simonw/sqlite-utils/commit/c0251cc9271260de73b4227859a51fab9b4cb745?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25% compared to head [(`afdf618`)](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.29%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #531 +/- ## ========================================== + Coverage 96.25% 96.29% +0.03% ========================================== Files 6 6 Lines 2671 2671 ========================================== + Hits 2571 2572 +1 + Misses 100 99 -1 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `95.13% <ø> (ø)` | | | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.26% <100.00%> (+0.09%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon, https://github.com/simonw/sqlite-utils/pull/528#issuecomment-1539053230,https://api.github.com/repos/simonw/sqlite-utils/issues/528,1539053230,IC_kwDOCGYnMM5bvBau,9599,simonw,2023-05-08T21:08:23Z,2023-05-08T21:08:23Z,OWNER,"I fixed this in: - #527 Will fully remove this misfeature in: - #542","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578793661,Enable `Table.convert()` on falsey values, https://github.com/simonw/sqlite-utils/pull/519#issuecomment-1539058795,https://api.github.com/repos/simonw/sqlite-utils/issues/519,1539058795,IC_kwDOCGYnMM5bvCxr,9599,simonw,2023-05-08T21:12:52Z,2023-05-08T21:12:52Z,OWNER,"This is a really neat fix, thank you.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1505568103,Fixes breaking DEFAULT values, https://github.com/simonw/sqlite-utils/pull/515#issuecomment-1539077777,https://api.github.com/repos/simonw/sqlite-utils/issues/515,1539077777,IC_kwDOCGYnMM5bvHaR,9599,simonw,2023-05-08T21:27:10Z,2023-05-08T21:27:10Z,OWNER,I should have spotted this PR before I shipped my own fix! https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194930,"upsert new rows with constraints, fixes #514", https://github.com/simonw/sqlite-utils/pull/51#issuecomment-515756563,https://api.github.com/repos/simonw/sqlite-utils/issues/51,515756563,MDEyOklzc3VlQ29tbWVudDUxNTc1NjU2Mw==,9599,simonw,2019-07-28T11:56:10Z,2019-07-28T11:56:10Z,OWNER,"Interesting. The tests failed presumably because the version of SQLite I am running on Travis CI doesn't have that same 999 limit. I'm going to enforce the 999 limit within the library itself, to discourage people from writing code which will fail if it runs on a host that DOES have that limit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473733752,"Fix for too many SQL variables, closes #50", https://github.com/simonw/sqlite-utils/pull/508#issuecomment-1297788531,https://api.github.com/repos/simonw/sqlite-utils/issues/508,1297788531,IC_kwDOCGYnMM5NWq5z,7908073,chapmanjacobd,2022-10-31T22:54:33Z,2022-11-17T15:11:16Z,CONTRIBUTOR,"Maybe this is actually a problem in the python sqlite bindings. Given [SQLITE's stance on this](https://www.sqlite.org/invalidutf.html) they should probably use `encode('utf-8', 'surrogatepass')`. As far as I understand the error here won't actually be resolved by this PR as-is. We would need to modify the data with `surrogateescape`... :/ or modify the sqlite3 module to use `surrogatepass`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430563092,Allow surrogates in parameters, https://github.com/simonw/sqlite-utils/pull/508#issuecomment-1297754631,https://api.github.com/repos/simonw/sqlite-utils/issues/508,1297754631,IC_kwDOCGYnMM5NWioH,22429695,codecov[bot],2022-10-31T22:14:48Z,2022-10-31T22:53:59Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **96.25**% // Head: **96.09**% // Decreases project coverage by **`-0.15%`** :warning: > Coverage data is based on head [(`2d6a149`)](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`529110e`)](https://codecov.io/gh/simonw/sqlite-utils/commit/529110e7d8c4a6b1bbf5fb61f2e29d72aa95a611?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 63.63% of modified lines in pull request are covered. > :exclamation: Current head 2d6a149 differs from pull request most recent head 43a8c4c. Consider uploading reports for the commit 43a8c4c to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #508 +/- ## ========================================== - Coverage 96.25% 96.09% -0.16% ========================================== Files 4 4 Lines 2401 2407 +6 ========================================== + Hits 2311 2313 +2 - Misses 90 94 +4 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/508/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `96.79% <63.63%> (-0.30%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430563092,Allow surrogates in parameters, https://github.com/simonw/sqlite-utils/pull/499#issuecomment-1292401308,https://api.github.com/repos/simonw/sqlite-utils/issues/499,1292401308,IC_kwDOCGYnMM5NCHqc,7908073,chapmanjacobd,2022-10-26T17:54:26Z,2022-10-26T17:54:51Z,CONTRIBUTOR,"The problem with how it is currently is that the transformed fts table _will_ return incorrect results (unless the table was only 1 row or something), even if create_triggers was enabled previously. Maybe the simplest solution is to disable fts on a transformed table rather than try to recreate it? Thoughts?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1405196044,feat: recreate fts triggers after table transform, https://github.com/simonw/sqlite-utils/pull/499#issuecomment-1275248761,https://api.github.com/repos/simonw/sqlite-utils/issues/499,1275248761,IC_kwDOCGYnMM5MAsB5,22429695,codecov[bot],2022-10-11T20:38:13Z,2022-10-11T20:38:13Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/499?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **96.37**% // Head: **96.25**% // Decreases project coverage by **`-0.11%`** :warning: > Coverage data is based on head [(`a25e838`)](https://codecov.io/gh/simonw/sqlite-utils/pull/499?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`d792dad`)](https://codecov.io/gh/simonw/sqlite-utils/commit/d792dad1cf5f16525da81b1e162fb71d469995f3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 40.00% of modified lines in pull request are covered.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #499 +/- ## ========================================== - Coverage 96.37% 96.25% -0.12% ========================================== Files 4 4 Lines 2401 2406 +5 ========================================== + Hits 2314 2316 +2 - Misses 87 90 +3 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/499?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.09% <40.00%> (-0.23%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/499?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1405196044,feat: recreate fts triggers after table transform, https://github.com/simonw/sqlite-utils/pull/498#issuecomment-1274153135,https://api.github.com/repos/simonw/sqlite-utils/issues/498,1274153135,IC_kwDOCGYnMM5L8giv,7908073,chapmanjacobd,2022-10-11T06:34:31Z,2022-10-11T06:34:31Z,CONTRIBUTOR,nevermind it was because I was running `db[table].transform`. The fts tables would still be there but the triggers would be dropped,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1404013495,fix: enable-fts permanently save triggers, https://github.com/simonw/sqlite-utils/pull/498#issuecomment-1274097701,https://api.github.com/repos/simonw/sqlite-utils/issues/498,1274097701,IC_kwDOCGYnMM5L8TAl,22429695,codecov[bot],2022-10-11T05:13:37Z,2022-10-11T05:13:37Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/498?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **96.37**% // Head: **96.37**% // Increases project coverage by **`+0.00%`** :tada: > Coverage data is based on head [(`75577e5`)](https://codecov.io/gh/simonw/sqlite-utils/pull/498?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`d792dad`)](https://codecov.io/gh/simonw/sqlite-utils/commit/d792dad1cf5f16525da81b1e162fb71d469995f3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 100.00% of modified lines in pull request are covered.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #498 +/- ## ======================================= Coverage 96.37% 96.37% ======================================= Files 4 4 Lines 2401 2402 +1 ======================================= + Hits 2314 2315 +1 Misses 87 87 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/498?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/498/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.32% <100.00%> (+<0.01%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/498?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1404013495,fix: enable-fts permanently save triggers, https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248593835,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248593835,IC_kwDOCGYnMM5KbAer,9599,simonw,2022-09-15T20:37:14Z,2022-09-15T20:37:14Z,OWNER,"I'm going to land this anyway. The lack of a streaming JSON parser is a separate issue, I don't think it should block landing this improvement.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248591268,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248591268,IC_kwDOCGYnMM5Ka_2k,9599,simonw,2022-09-15T20:36:02Z,2022-09-15T20:40:03Z,OWNER,"I had a big CSV file lying around, I converted it to other formats like this: sqlite-utils insert /tmp/t.db t /tmp/en.openfoodfacts.org.products.csv --csv sqlite-utils rows /tmp/t.db t --nl > /tmp/big.nl sqlite-utils rows /tmp/t.db t > /tmp/big.json Then tested the progress bar like this: sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl Output: ``` sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl [------------------------------------] 0% [#######-----------------------------] 20% 00:00:20 ``` With `--silent` it is silent. And for regular JSON: ``` sqlite-utils insert /tmp/t3.db t /tmp/big.json [####################################] 100% ``` This is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248582147,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248582147,IC_kwDOCGYnMM5Ka9oD,9599,simonw,2022-09-15T20:29:17Z,2022-09-15T20:29:17Z,OWNER,This looks good to me. I need to run some manual tests before merging (it's a good sign that the automated tests pass though).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248568775,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248568775,IC_kwDOCGYnMM5Ka6XH,9599,simonw,2022-09-15T20:16:14Z,2022-09-15T20:16:14Z,OWNER,"https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#using-the-python-version-input says can set the full version: ``` - uses: actions/setup-python@v4 with: python-version: ""3.10.6"" ``` I'll try that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248567323,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248567323,IC_kwDOCGYnMM5Ka6Ab,9599,simonw,2022-09-15T20:14:45Z,2022-09-15T20:14:45Z,OWNER,"There's a fix for `mypy` that has landed but isn't out in a release yet: - https://github.com/python/mypy/issues/13385 For the moment looks like pinning to Python 3.10.6 could help. Need to figure out how to do that in GitHub Actions though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248565396,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248565396,IC_kwDOCGYnMM5Ka5iU,9599,simonw,2022-09-15T20:12:50Z,2022-09-15T20:12:50Z,OWNER,"Annoying `mypy` test failure: ``` /Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/site-packages/numpy/__init__.pyi:636: error: Positional-only parameters are only supported in Python 3.8 and greater ``` Looks like this: - https://github.com/python/mypy/issues/13627","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248527646,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248527646,IC_kwDOCGYnMM5KawUe,22429695,codecov[bot],2022-09-15T19:34:59Z,2022-09-15T20:23:12Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **96.47**% // Head: **96.52**% // Increases project coverage by **`+0.04%`** :tada: > Coverage data is based on head [(`0acbc68`)](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`d9b9e07`)](https://codecov.io/gh/simonw/sqlite-utils/commit/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 100.00% of modified lines in pull request are covered. > :exclamation: Current head 0acbc68 differs from pull request most recent head d5db749. Consider uploading reports for the commit d5db749 to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #486 +/- ## ========================================== + Coverage 96.47% 96.52% +0.04% ========================================== Files 6 6 Lines 2642 2646 +4 ========================================== + Hits 2549 2554 +5 + Misses 93 92 -1 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/486/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.86% <100.00%> (ø)` | | | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/486/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.98% <100.00%> (+0.47%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232419522,https://api.github.com/repos/simonw/sqlite-utils/issues/480,1232419522,IC_kwDOCGYnMM5JdTrC,9599,simonw,2022-08-31T03:33:27Z,2022-08-31T03:33:27Z,OWNER,"Tests look great, thank you!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1355433619,search_sql add include_rank option, https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232356302,https://api.github.com/repos/simonw/sqlite-utils/issues/480,1232356302,IC_kwDOCGYnMM5JdEPO,7908073,chapmanjacobd,2022-08-31T01:51:49Z,2022-08-31T01:51:49Z,CONTRIBUTOR,Thanks for pointing me to the right place,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1355433619,search_sql add include_rank option, https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232091957,https://api.github.com/repos/simonw/sqlite-utils/issues/480,1232091957,IC_kwDOCGYnMM5JcDs1,22429695,codecov[bot],2022-08-30T19:45:23Z,2022-08-31T03:34:37Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/480?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **96.47**% // Head: **96.47**% // Increases project coverage by **`+0.00%`** :tada: > Coverage data is based on head [(`3dbfca6`)](https://codecov.io/gh/simonw/sqlite-utils/pull/480?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`087753c`)](https://codecov.io/gh/simonw/sqlite-utils/commit/087753cd42c406f1e060c1822dcd9b5fda3d60f4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 100.00% of modified lines in pull request are covered.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #480 +/- ## ======================================= Coverage 96.47% 96.47% ======================================= Files 6 6 Lines 2640 2642 +2 ======================================= + Hits 2547 2549 +2 Misses 93 93 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/480?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/480/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.32% <100.00%> (+<0.01%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/480?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1355433619,search_sql add include_rank option, https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232089808,https://api.github.com/repos/simonw/sqlite-utils/issues/480,1232089808,IC_kwDOCGYnMM5JcDLQ,9599,simonw,2022-08-30T19:43:02Z,2022-08-30T19:43:02Z,OWNER,"Yeah this seems like a reasonable addition to me. Needs a test, which can go next to this one: https://github.com/simonw/sqlite-utils/blob/087753cd42c406f1e060c1822dcd9b5fda3d60f4/tests/test_fts.py#L561","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1355433619,search_sql add include_rank option, https://github.com/simonw/sqlite-utils/pull/473#issuecomment-1229130453,https://api.github.com/repos/simonw/sqlite-utils/issues/473,1229130453,IC_kwDOCGYnMM5JQwrV,22429695,codecov[bot],2022-08-27T05:55:12Z,2022-08-27T05:55:18Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/473?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#473](https://codecov.io/gh/simonw/sqlite-utils/pull/473?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (72db599) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a46a5e3a9e03dcdd8c84a92e4a5dbfa02ba461fa?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a46a5e3) will **decrease** coverage by `0.07%`. > The diff coverage is `50.00%`. ```diff @@ Coverage Diff @@ ## main #473 +/- ## ========================================== - Coverage 96.67% 96.60% -0.08% ========================================== Files 6 6 Lines 2587 2590 +3 ========================================== + Hits 2501 2502 +1 - Misses 86 88 +2 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/473?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/473/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.94% <50.00%> (-0.18%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352953535,Support entrypoints for `--load-extension`, https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229303691,https://api.github.com/repos/simonw/sqlite-utils/issues/468,1229303691,IC_kwDOCGYnMM5JRa-L,9599,simonw,2022-08-27T23:13:06Z,2022-08-27T23:13:06Z,OWNER,"Documentation: - https://sqlite-utils--468.org.readthedocs.build/en/468/python-api.html#explicitly-creating-a-table - https://sqlite-utils--468.org.readthedocs.build/en/468/cli.html#creating-tables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348294436,"db[table].create(..., transform=True) and create-table --transform", https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229284539,https://api.github.com/repos/simonw/sqlite-utils/issues/468,1229284539,IC_kwDOCGYnMM5JRWS7,9599,simonw,2022-08-27T22:54:07Z,2022-08-27T22:55:01Z,OWNER,"Just needs documentation now, which can go here: https://sqlite-utils--468.org.readthedocs.build/en/468/python-api.html#explicitly-creating-a-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348294436,"db[table].create(..., transform=True) and create-table --transform", https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229279539,https://api.github.com/repos/simonw/sqlite-utils/issues/468,1229279539,IC_kwDOCGYnMM5JRVEz,9599,simonw,2022-08-27T22:24:33Z,2022-08-27T22:24:33Z,OWNER,"Interesting challenge with `default_value`: I need to be able to tell if the default values passed to `.create()` differ from those in the database already. Introspecting that is a bit tricky: ```pycon >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db[""blah""].create({""id"": int, ""name"": str}, not_null=(""name"",), defaults={""name"": ""bob""}) >>> db[""blah""].columns [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value=""'bob'"", is_pk=0)] ``` Note how a default value of the Python string `bob` is represented in the results of `PRAGMA table_info()` as `default_value=""'bob'""` - it's got single quotes added to it! So comparing default values from introspecting the database needs me to first parse that syntax. This may require a new table introspection method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348294436,"db[table].create(..., transform=True) and create-table --transform", https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229276554,https://api.github.com/repos/simonw/sqlite-utils/issues/468,1229276554,IC_kwDOCGYnMM5JRUWK,9599,simonw,2022-08-27T22:05:40Z,2022-08-27T22:05:51Z,OWNER,For the moment I'm not going to pay attention to `foreign_keys` changes - I will note that these are not modified in the documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348294436,"db[table].create(..., transform=True) and create-table --transform", https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229207725,https://api.github.com/repos/simonw/sqlite-utils/issues/468,1229207725,IC_kwDOCGYnMM5JRDit,9599,simonw,2022-08-27T14:55:32Z,2022-08-27T22:03:50Z,OWNER,"The main challenge here is coming up with comprehensive tests. The cases I need to cover are from this block of code: https://github.com/simonw/sqlite-utils/blob/c5f8a2eb1a81a18b52825cc649112f71fe419b12/sqlite_utils/db.py#L1468-L1480","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348294436,"db[table].create(..., transform=True) and create-table --transform", https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1224412060,https://api.github.com/repos/simonw/sqlite-utils/issues/468,1224412060,IC_kwDOCGYnMM5I-wuc,22429695,codecov[bot],2022-08-23T17:30:24Z,2022-08-27T23:11:44Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/468?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#468](https://codecov.io/gh/simonw/sqlite-utils/pull/468?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fc38480) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/c5f8a2eb1a81a18b52825cc649112f71fe419b12?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c5f8a2e) will **decrease** coverage by `0.12%`. > The diff coverage is `88.88%`. > :exclamation: Current head fc38480 differs from pull request most recent head 2f6a64f. Consider uploading reports for the commit 2f6a64f to get more accurate results ```diff @@ Coverage Diff @@ ## main #468 +/- ## ========================================== - Coverage 96.60% 96.47% -0.13% ========================================== Files 6 6 Lines 2590 2640 +50 ========================================== + Hits 2502 2547 +45 - Misses 88 93 +5 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/468?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/468/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.86% <66.66%> (-0.09%)` | :arrow_down: | | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/468/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.31% <90.19%> (-0.23%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348294436,"db[table].create(..., transform=True) and create-table --transform", https://github.com/simonw/sqlite-utils/pull/465#issuecomment-1218663096,https://api.github.com/repos/simonw/sqlite-utils/issues/465,1218663096,IC_kwDOCGYnMM5Io1K4,9599,simonw,2022-08-17T23:37:51Z,2022-08-17T23:37:51Z,OWNER,Source links work on the preview: https://sqlite-utils--465.org.readthedocs.build/en/465/reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1342357149,beanbag-docutils>=2.0, https://github.com/simonw/sqlite-utils/pull/465#issuecomment-1218565147,https://api.github.com/repos/simonw/sqlite-utils/issues/465,1218565147,IC_kwDOCGYnMM5IodQb,22429695,codecov[bot],2022-08-17T22:44:40Z,2022-08-17T23:21:22Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/465?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#465](https://codecov.io/gh/simonw/sqlite-utils/pull/465?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (bb90d33) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/45e24deffea042b5db7ab84cd1eb63b3ed9bb9da?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (45e24de) will **not change** coverage. > The diff coverage is `n/a`. ```diff @@ Coverage Diff @@ ## main #465 +/- ## ======================================= Coverage 96.64% 96.64% ======================================= Files 6 6 Lines 2566 2566 ======================================= Hits 2480 2480 Misses 86 86 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1342357149,beanbag-docutils>=2.0, https://github.com/simonw/sqlite-utils/pull/463#issuecomment-1218610320,https://api.github.com/repos/simonw/sqlite-utils/issues/463,1218610320,IC_kwDOCGYnMM5IooSQ,9599,simonw,2022-08-17T23:11:07Z,2022-08-17T23:11:07Z,OWNER,Thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1334416486,Use Read the Docs action v1, https://github.com/simonw/sqlite-utils/pull/462#issuecomment-1203207940,https://api.github.com/repos/simonw/sqlite-utils/issues/462,1203207940,IC_kwDOCGYnMM5Ht38E,22429695,codecov[bot],2022-08-02T20:58:38Z,2022-08-02T21:05:58Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/462?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#462](https://codecov.io/gh/simonw/sqlite-utils/pull/462?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0a14a03) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/1856002e3c0fcc9f09f72ab7d97ad8c75f6de7df?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1856002) will **not change** coverage. > The diff coverage is `n/a`. ```diff @@ Coverage Diff @@ ## main #462 +/- ## ======================================= Coverage 96.64% 96.64% ======================================= Files 6 6 Lines 2566 2566 ======================================= Hits 2480 2480 Misses 86 86 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/462?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/462?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1856002...0a14a03](https://codecov.io/gh/simonw/sqlite-utils/pull/462?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1326391841,Discord badge, https://github.com/simonw/sqlite-utils/pull/462#issuecomment-1203207043,https://api.github.com/repos/simonw/sqlite-utils/issues/462,1203207043,IC_kwDOCGYnMM5Ht3uD,9599,simonw,2022-08-02T20:57:30Z,2022-08-02T20:57:30Z,OWNER,I created a new link for this - https://discord.gg/Ass7bCAMDw - which should drop people straight into the `#sqlite-utils` channel in the Datasette Discord.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1326391841,Discord badge, https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1220010289,https://api.github.com/repos/simonw/sqlite-utils/issues/460,1220010289,IC_kwDOCGYnMM5It-Ex,9599,simonw,2022-08-18T21:58:00Z,2022-08-18T21:58:00Z,OWNER,I'm going to merge this as-is and add more links separately.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1326087800,Cross-link CLI to Python docs, https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1203190312,https://api.github.com/repos/simonw/sqlite-utils/issues/460,1203190312,IC_kwDOCGYnMM5Htzoo,9599,simonw,2022-08-02T20:36:58Z,2022-08-02T20:36:58Z,OWNER,"> That preview link it added didn't work, maybe because I have a custom domain setup? Entirely my fault, fixed here :https://github.com/simonw/sqlite-utils/commit/98a28cbfe6cea67f6334b42b74f35b0ddd309561","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1326087800,Cross-link CLI to Python docs, https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1202947222,https://api.github.com/repos/simonw/sqlite-utils/issues/460,1202947222,IC_kwDOCGYnMM5Hs4SW,9599,simonw,2022-08-02T16:20:47Z,2022-08-02T16:20:47Z,OWNER,"That preview link it added didn't work, maybe because I have a custom domain setup? It linked to: https://readthedocs-preview--460.org.readthedocs.build/en/460/ - which 404s Here's the correct link: https://sqlite-utils--460.org.readthedocs.build/en/460/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1326087800,Cross-link CLI to Python docs, https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1202947160,https://api.github.com/repos/simonw/sqlite-utils/issues/460,1202947160,IC_kwDOCGYnMM5Hs4RY,22429695,codecov[bot],2022-08-02T16:20:42Z,2022-08-02T20:53:19Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/460?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#460](https://codecov.io/gh/simonw/sqlite-utils/pull/460?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f21f4cc) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/1856002e3c0fcc9f09f72ab7d97ad8c75f6de7df?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1856002) will **not change** coverage. > The diff coverage is `n/a`. ```diff @@ Coverage Diff @@ ## main #460 +/- ## ======================================= Coverage 96.64% 96.64% ======================================= Files 6 6 Lines 2566 2566 ======================================= Hits 2480 2480 Misses 86 86 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/460?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/460?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1856002...f21f4cc](https://codecov.io/gh/simonw/sqlite-utils/pull/460?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1326087800,Cross-link CLI to Python docs, https://github.com/simonw/sqlite-utils/pull/457#issuecomment-1197108865,https://api.github.com/repos/simonw/sqlite-utils/issues/457,1197108865,IC_kwDOCGYnMM5HWm6B,9599,simonw,2022-07-27T17:58:10Z,2022-07-27T17:58:10Z,OWNER,Here's the build from the PR: https://sqlite-utils--457.org.readthedocs.build/en/457/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1319881016,Link to installation instructions, https://github.com/simonw/sqlite-utils/pull/457#issuecomment-1197090890,https://api.github.com/repos/simonw/sqlite-utils/issues/457,1197090890,IC_kwDOCGYnMM5HWihK,22429695,codecov[bot],2022-07-27T17:40:14Z,2022-07-27T17:40:40Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/457?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#457](https://codecov.io/gh/simonw/sqlite-utils/pull/457?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (80b3907) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/9e6cceac1c0e086429e2d308b700e59cc53a1991?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9e6ccea) will **not change** coverage. > The diff coverage is `n/a`. ```diff @@ Coverage Diff @@ ## main #457 +/- ## ======================================= Coverage 96.64% 96.64% ======================================= Files 6 6 Lines 2566 2566 ======================================= Hits 2480 2480 Misses 86 86 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/457?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/457?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [9e6ccea...80b3907](https://codecov.io/gh/simonw/sqlite-utils/pull/457?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1319881016,Link to installation instructions, https://github.com/simonw/sqlite-utils/pull/455#issuecomment-1229205990,https://api.github.com/repos/simonw/sqlite-utils/issues/455,1229205990,IC_kwDOCGYnMM5JRDHm,9599,simonw,2022-08-27T14:44:45Z,2022-08-27T14:44:45Z,OWNER,"I did not know `is` worked against strings and integers and booleans! https://latest.datasette.io/_memory?sql=select+1+is+1%2C+2.3+is+2.3%2C+true+is+true%2C+true+is+false%2C+%27dog%27+is+%27dog%27%2C+%27dog%27+is+%27cat%27%2C+null+is+null%2C+null+is+not+null ```sql select 1 is 1, 2.3 is 2.3, true is true, true is false, 'dog' is 'dog', 'dog' is 'cat', null is null, null is not null ``` | 1 is 1 | 2.3 is 2.3 | true is true | true is false | 'dog' is 'dog' | 'dog' is 'cat' | null is null | null is not null | |----------|--------------|----------------|-----------------|------------------|------------------|----------------|--------------------| | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1309542173,"in extract code, check equality with IS instead of = for nulls", https://github.com/simonw/sqlite-utils/pull/455#issuecomment-1190686273,https://api.github.com/repos/simonw/sqlite-utils/issues/455,1190686273,IC_kwDOCGYnMM5G-G5B,9599,simonw,2022-07-20T19:46:15Z,2022-07-20T19:46:15Z,OWNER,Can you add a new test for this? Something derived from the example in #423 would work great.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1309542173,"in extract code, check equality with IS instead of = for nulls", https://github.com/simonw/sqlite-utils/pull/455#issuecomment-1189130490,https://api.github.com/repos/simonw/sqlite-utils/issues/455,1189130490,IC_kwDOCGYnMM5G4LD6,22429695,codecov[bot],2022-07-19T14:32:10Z,2022-07-19T14:32:10Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/455?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#455](https://codecov.io/gh/simonw/sqlite-utils/pull/455?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b35a92) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/855bce8c3823718def13e0b8928c58bf857e41b2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (855bce8) will **not change** coverage. > The diff coverage is `n/a`. ```diff @@ Coverage Diff @@ ## main #455 +/- ## ======================================= Coverage 96.64% 96.64% ======================================= Files 6 6 Lines 2566 2566 ======================================= Hits 2480 2480 Misses 86 86 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/455?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/455/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.54% <ø> (ø)` | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/455?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/455?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [855bce8...1b35a92](https://codecov.io/gh/simonw/sqlite-utils/pull/455?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1309542173,"in extract code, check equality with IS instead of = for nulls", https://github.com/simonw/sqlite-utils/pull/452#issuecomment-1185949850,https://api.github.com/repos/simonw/sqlite-utils/issues/452,1185949850,IC_kwDOCGYnMM5GsCia,22429695,codecov[bot],2022-07-15T21:15:45Z,2022-07-15T21:18:34Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/452?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#452](https://codecov.io/gh/simonw/sqlite-utils/pull/452?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (eef350f) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/42440d6345c242ee39778045e29143fb550bd2c2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (42440d6) will **increase** coverage by `0.00%`. > The diff coverage is `100.00%`. ```diff @@ Coverage Diff @@ ## main #452 +/- ## ======================================= Coverage 96.60% 96.61% ======================================= Files 6 6 Lines 2534 2540 +6 ======================================= + Hits 2448 2454 +6 Misses 86 86 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/452?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/452/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.53% <100.00%> (+0.01%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/452?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/452?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [42440d6...eef350f](https://codecov.io/gh/simonw/sqlite-utils/pull/452?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1299760627,Add duplicate table feature, https://github.com/simonw/sqlite-utils/pull/437#issuecomment-1141488533,https://api.github.com/repos/simonw/sqlite-utils/issues/437,1141488533,IC_kwDOCGYnMM5ECbuV,9599,simonw,2022-05-30T21:32:36Z,2022-05-30T21:32:36Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1244294227,docs to dogs, https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115197644,https://api.github.com/repos/simonw/sqlite-utils/issues/429,1115197644,IC_kwDOCGYnMM5CeJDM,9599,simonw,2022-05-02T18:04:28Z,2022-05-02T18:04:28Z,OWNER,I'm going to ship this straight away as `3.26.1`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223177069,Depend on click-default-group-wheel, https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115196863,https://api.github.com/repos/simonw/sqlite-utils/issues/429,1115196863,IC_kwDOCGYnMM5CeI2_,9599,simonw,2022-05-02T18:03:47Z,2022-05-02T18:52:42Z,OWNER,"I made a build of this branch and tested it like this: https://pyodide.org/en/stable/console.html ```pycon >>> import micropip >>> await micropip.install(""https://s3.amazonaws.com/simonwillison-cors-allowed-public/sqlite_utils-3.26-py3-none-any.whl"") >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> list(db.query(""select 32443 + 55"")) [{'32443 + 55': 32498}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223177069,Depend on click-default-group-wheel, https://github.com/simonw/sqlite-utils/pull/419#issuecomment-1076662556,https://api.github.com/repos/simonw/sqlite-utils/issues/419,1076662556,IC_kwDOCGYnMM5ALJEc,22429695,codecov[bot],2022-03-23T18:12:47Z,2022-03-23T18:12:47Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#419](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (228f736) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/93fa79d30b1531bea281d0eb6b925c4e61bc1aa6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (93fa79d) will **not change** coverage. > The diff coverage is `n/a`. ```diff @@ Coverage Diff @@ ## main #419 +/- ## ======================================= Coverage 96.55% 96.55% ======================================= Files 6 6 Lines 2498 2498 ======================================= Hits 2412 2412 Misses 86 86 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [93fa79d...228f736](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178484369,Ignore common generated files, https://github.com/simonw/sqlite-utils/pull/410#issuecomment-1055856441,https://api.github.com/repos/simonw/sqlite-utils/issues/410,1055856441,IC_kwDOCGYnMM4-7xc5,9599,simonw,2022-03-01T21:05:21Z,2022-03-01T21:05:21Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149729902,Correct spelling mistakes (found with codespell), https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040998433,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040998433,IC_kwDOCGYnMM4-DGAh,25778,eyeseast,2022-02-16T01:29:39Z,2022-02-16T01:29:39Z,CONTRIBUTOR,Happy to do it and have it in the library. Going to use it a bunch. This whole SpatiaLite toolchain become a huge part of my work in the past year.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040959312,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040959312,IC_kwDOCGYnMM4-C8dQ,9599,simonw,2022-02-16T00:58:32Z,2022-02-16T00:58:32Z,OWNER,This is honestly one of the most complete PRs I've ever seen for a feature of this size. Thanks so much for this!,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040598665,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040598665,IC_kwDOCGYnMM4-BkaJ,9599,simonw,2022-02-15T17:58:11Z,2022-02-15T17:58:11Z,OWNER,"Wow, just found out I can edit files in this PR branch by hitting `.` on my keyboard while looking at the PR, then making changes in the VS Code for web on `github.dev`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040596969,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040596969,IC_kwDOCGYnMM4-Bj_p,9599,simonw,2022-02-15T17:56:22Z,2022-02-15T17:56:35Z,OWNER,"We should add SpatiaLite to the action that calculates code coverage - that way we can calculate coverage across the new GIS tests as well: https://github.com/simonw/sqlite-utils/blob/main/.github/workflows/test-coverage.yml Should just be a case of adding this to that workflow - we can do this in the same PR. ``` - name: Install SpatiaLite run: sudo apt-get install libsqlite3-mod-spatialite ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040595572,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040595572,IC_kwDOCGYnMM4-Bjp0,9599,simonw,2022-02-15T17:54:58Z,2022-02-15T17:54:58Z,OWNER,This PR looks fantastic.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040580250,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040580250,IC_kwDOCGYnMM4-Bf6a,25778,eyeseast,2022-02-15T17:40:00Z,2022-02-15T17:40:00Z,CONTRIBUTOR,@simonw I think this is ready for a look.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040519196,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040519196,IC_kwDOCGYnMM4-BRAc,22429695,codecov[bot],2022-02-15T16:52:21Z,2022-02-15T18:12:03Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#407](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a974da5) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/e7f040106b5f5a892ebd984f19b21c605e87c142?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e7f0401) will **increase** coverage by `0.71%`. > The diff coverage is `85.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/407/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #407 +/- ## ========================================== + Coverage 95.91% 96.62% +0.71% ========================================== Files 6 6 Lines 2421 2460 +39 ========================================== + Hits 2322 2377 +55 + Misses 99 83 -16 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.76% <85.00%> (+0.06%)` | :arrow_up: | | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.59% <0.00%> (ø)` | | | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.72% <0.00%> (+1.43%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [e7f0401...a974da5](https://codecov.io/gh/simonw/sqlite-utils/pull/407?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/40#issuecomment-511266664,https://api.github.com/repos/simonw/sqlite-utils/issues/40,511266664,MDEyOklzc3VlQ29tbWVudDUxMTI2NjY2NA==,9599,simonw,2019-07-15T03:47:26Z,2019-07-15T03:47:26Z,OWNER,This will close #36 and #39 once it is finished.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",467928674,.get() method plus support for compound primary keys, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1030002502,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1030002502,IC_kwDOCGYnMM49ZJdG,25778,eyeseast,2022-02-04T13:50:19Z,2022-02-04T13:50:19Z,CONTRIBUTOR,Awesome. Thanks for your help getting it in. Will now look at adding CLI versions of this. It's going to be super helpful on a bunch of my projects.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029703216,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029703216,IC_kwDOCGYnMM49YAYw,9599,simonw,2022-02-04T06:45:43Z,2022-02-04T06:45:43Z,OWNER,Shipped this as `sqlite-utils` 3.23: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-23,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029682294,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029682294,IC_kwDOCGYnMM49X7R2,9599,simonw,2022-02-04T05:53:26Z,2022-02-04T05:53:26Z,OWNER,"This looks fantastic, thanks for all of the work you put into this!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029370537,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029370537,IC_kwDOCGYnMM49WvKp,25778,eyeseast,2022-02-03T20:25:58Z,2022-02-03T20:25:58Z,CONTRIBUTOR,"OK, I moved all the GIS helpers into `db.py` as methods on `Database` and `Table`, and I put `find_spatialite` back in `utils.py`. I deleted `gis.py`, since there's nothing left it. Docs and tests are updated and passing. I think this is better.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029338360,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029338360,IC_kwDOCGYnMM49WnT4,25778,eyeseast,2022-02-03T19:43:56Z,2022-02-03T19:43:56Z,CONTRIBUTOR,"Works for me. I was just looking at how the FTS extensions work and they're just methods, too. So this can be consistent with that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029335225,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029335225,IC_kwDOCGYnMM49Wmi5,9599,simonw,2022-02-03T19:39:40Z,2022-02-03T19:39:40Z,OWNER,"> I thought about adding these as methods on `Database` and `Table`, and I'm back and forth on it for the same reasons you are. It's certainly cleaner, and it's clearer what you're operating on. I could go either way. > > I do sort of like having all the Spatialite stuff in its own module, just because it's built around an extension you might not have or want, but I don't know if that's a good reason to have a different API. > > You could have `init_spatialite` add methods to `Database` and `Table`, so they're only there if you have Spatialite set up. Is that too clever? It feels too clever. Yeah that's too clever. You know what? I'm pretty confident we are both massively over-thinking this. We should put the methods on `Database` and `Table`! API simplicity and consistency matters more than vague concerns about purity.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029326568,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029326568,IC_kwDOCGYnMM49Wkbo,25778,eyeseast,2022-02-03T19:28:26Z,2022-02-03T19:28:26Z,CONTRIBUTOR,"> `from sqlite_utils.utils import find_spatialite` is part of the documented API already: > > https://sqlite-utils.datasette.io/en/3.22.1/python-api.html#finding-spatialite > > To avoid needing to bump the major version number to 4 to indicate a backwards incompatible change, we should keep a `from .gis import find_spatialite` line at the top of `utils.py` such that any existing code with that documented import continues to work. This is fixed now. I had to take out the type annotations for `Database` and `Table` to avoid a circular import, but that's fine and may be moot if these become class methods.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029306428,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029306428,IC_kwDOCGYnMM49Wfg8,25778,eyeseast,2022-02-03T19:03:43Z,2022-02-03T19:03:43Z,CONTRIBUTOR,"I thought about adding these as methods on `Database` and `Table`, and I'm back and forth on it for the same reasons you are. It's certainly cleaner, and it's clearer what you're operating on. I could go either way. I do sort of like having all the Spatialite stuff in its own module, just because it's built around an extension you might not have or want, but I don't know if that's a good reason to have a different API. You could have `init_spatialite` add methods to `Database` and `Table`, so they're only there if you have Spatialite set up. Is that too clever? It feels too clever. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029297971,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029297971,IC_kwDOCGYnMM49Wdcz,9599,simonw,2022-02-03T18:52:50Z,2022-02-03T18:52:50Z,OWNER,"I'm not sure I like `name=""geometry""` as the default argument to `add_geometry_column` - mainly because of this example here: ```python add_geometry_column(db[""locations""], ""POINT"") create_spatial_index(db[""locations""], ""geometry"") ``` I had to go and look at the code to figure out if `""POINT""` was the name of the column - and I don't like how inconsistent it looks next to the following `create_spatial_index()` call where you DO need to pass the column name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029296782,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029296782,IC_kwDOCGYnMM49WdKO,9599,simonw,2022-02-03T18:51:21Z,2022-02-03T18:51:21Z,OWNER,"What do you think about adding these as methods on the `Database` class instead? Then you could do: ```python # This is with an optional argument, which if omitted runs find_spatialite() for you: db.init_spatialite() # Instead of: init_spatialite(db, find_spatialite()) ``` Likewise, the `add_geometry_column` and `create_spatial_index` methods could live on `Table`: ```python # Instead of this: add_geometry_column(db[""locations""], ""POINT"", ""geometry"") create_spatial_index(db[""locations""], ""geometry"") # Could have this: db[""locations""].add_geometry_column(""POINT"") db[""locations""].create_spatial_index(""geometry"") ``` On the one hand, this is much more consistent with the existing `sqlite-utils` Python API. But on the other hand... this is mixing SpatiaLite functionality directly into the core classes. Is that a good idea, seeing as SpatiaLite is both an optional extension (which can be tricky to install) AND something that has a very different release cadence and quality-of-documentation from SQLite itself? There's a third option: the SpatiaLite could exist on subclasses of `Database` and `Table` - so the above examples would look something like this: ```python from sqlite_utils.gis import SpatiaLiteDatabase db = SpatiaLiteDatabase(""geo.db"") db.init_spatialite() db[""locations""].add_geometry_column(""POINT"") db[""locations""].create_spatial_index(""geometry"") ``` On the one hand, this would keep the SpatiaLite-specific stuff out of the core Database/Table classes. But it feels a bit untidy to me, especially since it raises the spectre of someone who was already subclassing Database for some reason now needing to instead subclass `SpatiaLiteDatabase` (not too keen on that capitalization) - or even (horror) trying to dabble with multiple inheritance, which can only lead to pain. So I don't have a strong opinion formed on this question yet!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029285985,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029285985,IC_kwDOCGYnMM49Wahh,9599,simonw,2022-02-03T18:37:48Z,2022-02-03T18:37:48Z,OWNER,"`from sqlite_utils.utils import find_spatialite` is part of the documented API already: https://sqlite-utils.datasette.io/en/3.22.1/python-api.html#finding-spatialite To avoid needing to bump the major version number to 4 to indicate a backwards incompatible change, we should keep a `from .gis import find_spatialite` line at the top of `utils.py` such that any existing code with that documented import continues to work.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029273853,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029273853,IC_kwDOCGYnMM49WXj9,9599,simonw,2022-02-03T18:23:30Z,2022-02-03T18:31:21Z,OWNER,"OK, this change makes a bunch of sense to me - and also raises some interesting questions about future additions to `sqlite-utils` with regards to SpatiaLite. Would `sqlite-utils add-geometry-column ...` be a good CLI enhancement. for example? I see you've already talked about that in #79 - moving this conversation there!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029180984,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029180984,IC_kwDOCGYnMM49WA44,25778,eyeseast,2022-02-03T16:42:04Z,2022-02-03T16:42:04Z,CONTRIBUTOR,Fixed my spelling. That's a useful thing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029177700,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029177700,IC_kwDOCGYnMM49WAFk,22429695,codecov[bot],2022-02-03T16:38:45Z,2022-02-04T05:52:39Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#385](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (af86b17) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/74586d3cb26fa3cc3412721985ecdc1864c2a31d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (74586d3) will **decrease** coverage by `0.61%`. > The diff coverage is `28.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/385/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #385 +/- ## ========================================== - Coverage 96.52% 95.91% -0.62% ========================================== Files 6 6 Lines 2389 2421 +32 ========================================== + Hits 2306 2322 +16 - Misses 83 99 +16 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/385/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.69% <ø> (+0.15%)` | :arrow_up: | | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/385/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `96.29% <15.00%> (-1.40%)` | :arrow_down: | | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/385/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.59% <80.00%> (ø)` | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [74586d3...af86b17](https://codecov.io/gh/simonw/sqlite-utils/pull/385?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029177015,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029177015,IC_kwDOCGYnMM49V_63,9599,simonw,2022-02-03T16:38:02Z,2022-02-03T16:38:02Z,OWNER,Sorry had missed this - tests should run now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029175907,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029175907,IC_kwDOCGYnMM49V_pj,25778,eyeseast,2022-02-03T16:36:54Z,2022-02-03T16:36:54Z,CONTRIBUTOR,"@simonw Not sure if you've seen this, but any chance you can run the tests?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/38#issuecomment-514339130,https://api.github.com/repos/simonw/sqlite-utils/issues/38,514339130,MDEyOklzc3VlQ29tbWVudDUxNDMzOTEzMA==,9599,simonw,2019-07-23T18:57:33Z,2019-07-23T18:57:33Z,OWNER,"This should switch to using `.get()`, introduced in https://github.com/simonw/sqlite-utils/commit/c65b67ca46f70e2da46a5b945f4ed358173262e9","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",467862459,table.update() method, https://github.com/simonw/sqlite-utils/pull/38#issuecomment-511219187,https://api.github.com/repos/simonw/sqlite-utils/issues/38,511219187,MDEyOklzc3VlQ29tbWVudDUxMTIxOTE4Nw==,9599,simonw,2019-07-14T17:05:26Z,2019-07-14T17:05:26Z,OWNER,I think I need to solve #36 (compound primary key support) first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",467862459,table.update() method, https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009534817,https://api.github.com/repos/simonw/sqlite-utils/issues/377,1009534817,IC_kwDOCGYnMM48LEdh,9599,simonw,2022-01-11T02:09:38Z,2022-01-11T02:09:38Z,OWNER,"I tested this like so: ``` % wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' % sqlite-utils create-database test.db % sqlite-utils create-table test.db power_plants url text owner text % sqlite-utils schema test.db CREATE TABLE [power_plants] ( [url] TEXT, [owner] TEXT ); % sqlite-utils bulk test.db 'insert into power_plants (url, owner) values (:url, :owner)' global_power_plant_database.csv --csv [------------------------------------] 0% [###################################-] 99% % sqlite-utils tables --counts test.db -t table count ------------ ------- power_plants 33643 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097477582,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009532125,https://api.github.com/repos/simonw/sqlite-utils/issues/377,1009532125,IC_kwDOCGYnMM48LDzd,9599,simonw,2022-01-11T02:03:35Z,2022-01-11T02:03:35Z,OWNER,Documentation: https://github.com/simonw/sqlite-utils/blob/f4ea0d32c0543373eefaa9b9f3911eb07549eecb/docs/cli.rst#executing-sql-in-bulk,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097477582,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009531863,https://api.github.com/repos/simonw/sqlite-utils/issues/377,1009531863,IC_kwDOCGYnMM48LDvX,22429695,codecov[bot],2022-01-11T02:03:00Z,2022-01-11T02:03:00Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#377](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f4ea0d3) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/b6dad08a8389736b7e960cfe9bc719cfc21a98f5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b6dad08) will **decrease** coverage by `0.01%`. > The diff coverage is `96.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/377/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #377 +/- ## ========================================== - Coverage 96.52% 96.50% -0.02% ========================================== Files 6 6 Lines 2330 2378 +48 ========================================== + Hits 2249 2295 +46 - Misses 81 83 +2 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/377/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.49% <94.28%> (-0.06%)` | :arrow_down: | | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/377/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.68% <100.00%> (+0.03%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b6dad08...f4ea0d3](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097477582,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1009272446,https://api.github.com/repos/simonw/sqlite-utils/issues/367,1009272446,IC_kwDOCGYnMM48KEZ-,9599,simonw,2022-01-10T19:31:08Z,2022-01-10T19:31:08Z,OWNER,I'm going to implement this in a separate commit from this PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097041471,Initial prototype of .analyze() methods, https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1008158799,https://api.github.com/repos/simonw/sqlite-utils/issues/367,1008158799,IC_kwDOCGYnMM48F0hP,22429695,codecov[bot],2022-01-08T21:36:55Z,2022-01-09T02:34:44Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#367](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9848eaa) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a8f9cc6f64f299830834428509940d448b82b4ed?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a8f9cc6) will **decrease** coverage by `0.20%`. > The diff coverage is `50.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/367/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #367 +/- ## ========================================== - Coverage 96.44% 96.24% -0.21% ========================================== Files 5 6 +1 Lines 2307 2317 +10 ========================================== + Hits 2225 2230 +5 - Misses 82 87 +5 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.15% <28.57%> (-0.42%)` | :arrow_down: | | [sqlite\_utils/\_\_main\_\_.py](https://codecov.io/gh/simonw/sqlite-utils/pull/367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL19fbWFpbl9fLnB5) | `100.00% <100.00%> (ø)` | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a8f9cc6...9848eaa](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097041471,Initial prototype of .analyze() methods, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006315145,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006315145,IC_kwDOCGYnMM47-yaJ,9599,simonw,2022-01-06T06:20:51Z,2022-01-06T06:20:51Z,OWNER,This is all documented. I'm going to rebase-merge it to keep the individual commits.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006311742,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006311742,IC_kwDOCGYnMM47-xk-,9599,simonw,2022-01-06T06:12:19Z,2022-01-06T06:12:19Z,OWNER,"Got that working: ``` % echo 'This is cool' | sqlite-utils insert words.db words - --text --convert '({""word"": w} for w in text.split())' % sqlite-utils dump words.db BEGIN TRANSACTION; CREATE TABLE [words] ( [word] TEXT ); INSERT INTO ""words"" VALUES('This'); INSERT INTO ""words"" VALUES('is'); INSERT INTO ""words"" VALUES('cool'); COMMIT; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006309834,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006309834,IC_kwDOCGYnMM47-xHK,9599,simonw,2022-01-06T06:08:01Z,2022-01-06T06:08:01Z,OWNER,"For `--text` the conversion function should be allowed to return an iterable instead of a dictionary, in which case it will be treated as the full list of records to be inserted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006301546,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006301546,IC_kwDOCGYnMM47-vFq,9599,simonw,2022-01-06T05:44:47Z,2022-01-06T05:44:47Z,OWNER,Just need documentation for `--convert` now against the various different types of input.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006300280,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006300280,IC_kwDOCGYnMM47-ux4,9599,simonw,2022-01-06T05:40:45Z,2022-01-06T05:40:45Z,OWNER,"I'm going to rename `--all` to `--text`: > - Use `--text` to write the entire input to a column called ""text"" To avoid that clash with Python's `all()` function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006299778,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006299778,IC_kwDOCGYnMM47-uqC,9599,simonw,2022-01-06T05:39:10Z,2022-01-06T05:39:10Z,OWNER,`all` is a bad variable name because it clashes with the Python `all()` built-in function.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006295276,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006295276,IC_kwDOCGYnMM47-tjs,9599,simonw,2022-01-06T05:26:11Z,2022-01-06T05:26:11Z,OWNER,"Here's the traceback if your `--convert` function doesn't return a dict right now: ``` % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils"", line 33, in sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1137, in __call__ return self.main(*args, **kwargs) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1062, in main rv = self.invoke(ctx) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 763, in invoke return __callback(*args, **kwargs) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 949, in insert insert_upsert_implementation( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 834, in insert_upsert_implementation db[table].insert_all( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2602, in insert_all first_record = next(records) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 3044, in fix_square_braces for record in records: File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 831, in docs = (decode_base64_values(doc) for doc in docs) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 86, in decode_base64_values to_fix = [ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 89, in if isinstance(doc[k], dict) TypeError: string indices must be integers ``` I can live with that for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006294777,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006294777,IC_kwDOCGYnMM47-tb5,9599,simonw,2022-01-06T05:24:54Z,2022-01-06T05:24:54Z,OWNER,"> I added a custom error message for if the user's `--convert` code doesn't return a dict. That turned out to be a bad idea because it meant exhausting the iterator early for the check - before we got to the `.insert_all()` code that breaks the iterator up into chunks. I tried fixing that with `itertools.tee()` to run the generator twice but that's grossly memory-inefficient for large imports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006288444,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006288444,IC_kwDOCGYnMM47-r48,9599,simonw,2022-01-06T05:07:10Z,2022-01-06T05:07:10Z,OWNER,"And here's a demo of `--convert` used with `--all` - I added a custom error message for if the user's `--convert` code doesn't return a dict. ``` % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all Error: Records returned by your --convert function must be dicts % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '{""all"": all.upper()}' --all % sqlite-utils dump /tmp/all.db BEGIN TRANSACTION; CREATE TABLE [blah] ( [all] TEXT ); INSERT INTO ""blah"" VALUES('INFO: 127.0.0.1:60581 - ""GET / HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FAVICON.ICO HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FOO/TIDDLYWIKI HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK INFO: 127.0.0.1:60584 - ""GET /FOO/-/STATIC/SQL-FORMATTER-2.3.3.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60586 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60585 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.CSS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60588 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0-SQL.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60587 - ""GET /FOO/-/STATIC/CM-RESIZE-1.0.1.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60586 - ""GET /FOO/TIDDLYWIKI/TIDDLERS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60586 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK INFO: 127.0.0.1:60584 - ""GET /FOO/-/STATIC/TABLE.JS HTTP/1.1"" 200 OK '); COMMIT; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006284673,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006284673,IC_kwDOCGYnMM47-q-B,9599,simonw,2022-01-06T04:55:52Z,2022-01-06T04:55:52Z,OWNER,"Test code that just worked for me: ``` sqlite-utils insert /tmp/blah.db blah /tmp/log.log --convert ' bits = line.split() return dict([(""b_{}"".format(i), bit) for i, bit in enumerate(bits)])' --lines ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006232013,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006232013,IC_kwDOCGYnMM47-eHN,9599,simonw,2022-01-06T02:21:35Z,2022-01-06T02:21:35Z,OWNER,"I'm having second thoughts about this bit: > Your Python code will be passed a ""row"" variable representing the imported row, and can return a modified row. > > If you are using `--lines` your code will be passed a ""line"" variable, and for `--all` an ""all"" variable. The code in question is this: https://github.com/simonw/sqlite-utils/blob/500a35ad4d91c8a6232134ce9406efec11bedff8/sqlite_utils/utils.py#L296-L303 Do I really want to add the complexity of supporting different variable names there? I think always using `value` might be better. Except... `value` made sense for the existing `sqlite-utils convert` command where you are running a conversion function against the value for the column in the current row - is it confusing if applied to lines or documents or `all`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006230411,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006230411,IC_kwDOCGYnMM47-duL,9599,simonw,2022-01-06T02:17:35Z,2022-01-06T02:17:35Z,OWNER,"Documentation: https://github.com/simonw/sqlite-utils/blob/33223856ff7fe746b7b77750fbe5b218531d0545/docs/cli.rst#inserting-unstructured-data-with---lines-and---all - I went with a single section titled ""Inserting unstructured data with --lines and --all""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006220129,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006220129,IC_kwDOCGYnMM47-bNh,9599,simonw,2022-01-06T01:52:26Z,2022-01-06T01:52:26Z,OWNER,I'm going to refactor all of the tests for `sqlite-utils insert` into a new `test_cli_insert.py` module.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006219956,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006219956,IC_kwDOCGYnMM47-bK0,22429695,codecov[bot],2022-01-06T01:51:54Z,2022-01-06T06:22:25Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#361](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b7f0b88) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/f3fd8613113d21d44238a6ec54b375f5aa72c4e0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f3fd861) will **decrease** coverage by `0.05%`. > The diff coverage is `92.85%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/361/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #361 +/- ## ========================================== - Coverage 96.49% 96.44% -0.06% ========================================== Files 5 5 Lines 2283 2306 +23 ========================================== + Hits 2203 2224 +21 - Misses 80 82 +2 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/361/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.49% <92.00%> (-0.11%)` | :arrow_down: | | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/361/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.23% <100.00%> (ø)` | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [f3fd861...b7f0b88](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006219848,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006219848,IC_kwDOCGYnMM47-bJI,9599,simonw,2022-01-06T01:51:36Z,2022-01-06T01:51:36Z,OWNER,"So far I've just implemented the new help: ``` % sqlite-utils insert --help Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE Insert records from FILE into a table, creating the table if it does not already exist. By default the input is expected to be a JSON array of objects. Or: - Use --nl for newline-delimited JSON objects - Use --csv or --tsv for comma-separated or tab-separated input - Use --lines to write each incoming line to a column called ""line"" - Use --all to write the entire input to a column called ""all"" You can also use --convert to pass a fragment of Python code that will be used to convert each input. Your Python code will be passed a ""row"" variable representing the imported row, and can return a modified row. If you are using --lines your code will be passed a ""line"" variable, and for --all an ""all"" variable. Options: --pk TEXT Columns to use as the primary key, e.g. id --flatten Flatten nested JSON objects, so {""a"": {""b"": 1}} becomes {""a_b"": 1} --nl Expect newline-delimited JSON -c, --csv Expect CSV input --tsv Expect TSV input --lines Treat each line as a single value called 'line' --all Treat input as a single value called 'all' --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-991397907,https://api.github.com/repos/simonw/sqlite-utils/issues/347,991397907,IC_kwDOCGYnMM47F4gT,9599,simonw,2021-12-11T01:01:40Z,2021-12-11T01:01:40Z,OWNER,The change I made to that test in #354 might help with this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982137888,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982137888,IC_kwDOCGYnMM46ijwg,9599,simonw,2021-11-29T23:50:54Z,2021-11-29T23:50:54Z,OWNER,If I'm going to `skipIf()` those tests I need a way to check if `pysqlite3` is being used.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982137293,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982137293,IC_kwDOCGYnMM46ijnN,9599,simonw,2021-11-29T23:49:29Z,2021-11-29T23:49:29Z,OWNER,"A short term fix would be to skip those tests against `pysqlite3` - but longer term it would be good to address the underlying issue, particularly for the WAL ones (the FTS ones aren't too worrying since if you deliberately try and break the FTS table it's not hugely problematic if you corrupt your database).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982136747,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982136747,IC_kwDOCGYnMM46ijer,9599,simonw,2021-11-29T23:48:05Z,2021-11-29T23:48:05Z,OWNER,"Some interesting test failures in the version that runs with `pysqlite3`: ``` =========================== short test summary info ============================ FAILED tests/test_cli.py::test_enable_wal - assert 0 == 1 FAILED tests/test_cli.py::test_disable_wal - pysqlite3.dbapi2.OperationalErro... FAILED tests/test_fts.py::test_rebuild_fts[searchable] - pysqlite3.dbapi2.Dat... FAILED tests/test_fts.py::test_rebuild_fts[searchable_fts] - pysqlite3.dbapi2... FAILED tests/test_wal.py::test_enable_disable_wal - pysqlite3.dbapi2.Operatio... ================== 5 failed, 750 passed, 3 skipped in 15.20s =================== ``` https://github.com/simonw/sqlite-utils/runs/4360759085 The WAL errors look like this: ``` E pysqlite3.dbapi2.OperationalError: cannot change into wal mode from within a transaction ``` Triggered by a call to `db.enable_wal()` The FTS errors are caused by tests that try to deliberately corrupt the FTS index by running `fresh_db[""searchable_fts_data""].delete_where()` - and then rebuilding it using `rebuild_fts()`: ``` @pytest.mark.parametrize(""table_to_fix"", [""searchable"", ""searchable_fts""]) def test_rebuild_fts(fresh_db, table_to_fix): table = fresh_db[""searchable""] table.insert(search_records[0]) table.enable_fts([""text"", ""country""]) # Run a search rows = list(table.search(""tanuki"")) assert len(rows) == 1 assert { ""rowid"": 1, ""text"": ""tanuki are running tricksters"", ""country"": ""Japan"", ""not_searchable"": ""foo"", }.items() <= rows[0].items() # Delete from searchable_fts_data fresh_db[""searchable_fts_data""].delete_where() # This should have broken the index with pytest.raises(sqlite3.DatabaseError): list(table.search(""tanuki"")) # Running rebuild_fts() should fix it > fresh_db[table_to_fix].rebuild_fts() tests/test_fts.py:277: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sqlite_utils/db.py:1947: in rebuild_fts self.db.execute( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = > sql = ""INSERT INTO [searchable_fts]([searchable_fts]) VALUES('rebuild');"" parameters = None def execute( self, sql: str, parameters: Optional[Union[Iterable, dict]] = None ) -> sqlite3.Cursor: ""Execute SQL query and return a ``sqlite3.Cursor``."" if self._tracer: self._tracer(sql, parameters) if parameters is not None: return self.conn.execute(sql, parameters) else: > return self.conn.execute(sql) E pysqlite3.dbapi2.DatabaseError: database disk image is malformed ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982133970,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982133970,IC_kwDOCGYnMM46iizS,9599,simonw,2021-11-29T23:41:17Z,2021-11-29T23:41:17Z,OWNER,"Took a bit of experimenting to get both `mypy` AND `flake8` to ignore the same line. The incantation that worked was this one: https://github.com/simonw/sqlite-utils/blob/f990e134aa8219b687ff6c261330f36824b5df36/sqlite_utils/utils.py#L8 Order here matters - this did NOT work for both tools: ```python from sqlite3.dump import _iterdump as iterdump # noqa: F401 # type: ignore ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982129727,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982129727,IC_kwDOCGYnMM46ihw_,9599,simonw,2021-11-29T23:31:58Z,2021-11-29T23:31:58Z,OWNER,"It failed on other Python versions with `mypy`: ``` sqlite_utils/utils.py:8: error: Cannot find implementation or library stub for module named ""sqlite3.dump"" sqlite_utils/utils.py:8: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982129218,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982129218,IC_kwDOCGYnMM46ihpC,9599,simonw,2021-11-29T23:31:02Z,2021-11-29T23:31:02Z,OWNER,Here's the test run that's installing `pysqlite3` and that version of SQLite: https://github.com/simonw/sqlite-utils/runs/4360663292,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982126665,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982126665,IC_kwDOCGYnMM46ihBJ,9599,simonw,2021-11-29T23:26:01Z,2021-11-29T23:33:48Z,OWNER,"https://github.com/simonw/sqlite-utils/blob/93b059dd230eae9eaae472b7fbabd4a66feeb79d/.github/workflows/test.yml#L11-L20 This configuration means that the numpy=0, Python=3.10, os=Ubuntu build will additionally use `pysqlite3` with the SQLite 3.37.0. It's failing right now: https://github.com/simonw/sqlite-utils/runs/4360593156 - because `pysqlite3` doesn't provide `.iterdump()`. I can use the workaround from this comment: https://github.com/coleifer/pysqlite3/issues/24#issuecomment-982081267","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982123183,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982123183,IC_kwDOCGYnMM46igKv,22429695,codecov[bot],2021-11-29T23:20:35Z,2021-12-11T01:02:19Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#347](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (71b6c38) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/213a0ff177f23a35f3b235386366ff132eb879f1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (213a0ff) will **increase** coverage by `0.00%`. > The diff coverage is `100.00%`. > :exclamation: Current head 71b6c38 differs from pull request most recent head 1a7ef2f. Consider uploading reports for the commit 1a7ef2f to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/347/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #347 +/- ## ======================================= Coverage 96.51% 96.52% ======================================= Files 5 5 Lines 2270 2271 +1 ======================================= + Hits 2191 2192 +1 Misses 79 79 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/347/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.73% <100.00%> (ø)` | | | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/347/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `93.68% <100.00%> (+0.03%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [213a0ff...1a7ef2f](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/337#issuecomment-962259527,https://api.github.com/repos/simonw/sqlite-utils/issues/337,962259527,IC_kwDOCGYnMM45WupH,771193,urbas,2021-11-05T22:33:02Z,2021-11-05T22:33:02Z,NONE,"Smokes, it looks like there was a bug in click 8.0.2 (fixed in 8.0.3: https://github.com/pallets/click/issues/2089). Meaning this PR is not needed. Closing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1046271107,Default values for `--attach` and `--param` options, https://github.com/simonw/sqlite-utils/pull/333#issuecomment-979442854,https://api.github.com/repos/simonw/sqlite-utils/issues/333,979442854,IC_kwDOCGYnMM46YRym,9599,simonw,2021-11-25T19:47:26Z,2021-11-25T19:47:26Z,OWNER,"I just remembered that there's one other place that this could fit: as a Datasette ""insert"" plugin. This is vaporware at the moment, but the idea is that Datasette itself could grow a mechanism for importing data, that's driven by plugins. Out of the box Datasette would be able to import CSV and CSV files, similar to `sqlite-utils insert ... --csv` - but plugins would then be able to add support for additional format such as GeoJSON or - in this case - Parquet. The neat thing about having it as a Datasette plugin is that one plugin would enable three different ways of importing data: 1. Via a new `datasette insert ...` CLI option (similar to `sqlite-utils`) 2. Via a web form upload interface, where authenticated Datasette users would be able to upload files 3. Via an API interface, where files could be programatically submitted to a running Datasette server I started fleshing out this idea quite a while ago but didn't make much concrete progress, maybe I should revisit it: - https://github.com/simonw/datasette/issues/1160","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1039037439,Add functionality to read Parquet files., https://github.com/simonw/sqlite-utils/pull/333#issuecomment-979345527,https://api.github.com/repos/simonw/sqlite-utils/issues/333,979345527,IC_kwDOCGYnMM46X6B3,2118708,Florents-Tselai,2021-11-25T16:31:47Z,2021-11-25T16:31:47Z,NONE,"Thanks for your reply @simonw . Tbh, my first attempt was actually the `parquet-to-sqlite` package but I already had Makefiles that relied on `SQLite-utils` and it was less intrusive to my workflow. Maybe I'll revisit that decision. FYI: there's a `[sqlite-parquet-vtable](https://github.com/cldellow/sqlite-parquet-vtable)` I don't think plugins make much sense either. Probably defeats the purpose of simplicity: simple database along with a pip-able package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1039037439,Add functionality to read Parquet files., https://github.com/simonw/sqlite-utils/pull/333#issuecomment-974754412,https://api.github.com/repos/simonw/sqlite-utils/issues/333,974754412,IC_kwDOCGYnMM46GZJs,9599,simonw,2021-11-21T04:35:32Z,2021-11-21T04:35:32Z,OWNER,"Some other recent projects (like trying to get this library to work in JupyterLite) have made me much more cautious about adding new dependencies, especially dependencies like `pyarrow` which require custom C/Rust extensions. There are a few ways this could work though: - Have this as an optional dependency feature - so it only works if the user installs `pyarrow` as well - Implement this as a separate tool, `parquet-to-sqlite` - which could itself depend on `sqlite-utils` - Add a concept of ""plugins"" to `sqlite-utils`, similar to how those work in Datasette: https://docs.datasette.io/en/stable/plugins.html My favourite option is `parquet-to-sqlite` because that can be built without any additional changes to `sqlite-utils` at all! I find the concept of plugins for `sqlite-utils` interesting. I've so far not had quite enough potential use-cases to convince me this is worthwhile (especially since it should be very easy to build out separate tools entirely), but I'm ready to be convinced that a plugin mechanism would be worthwhile.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1039037439,Add functionality to read Parquet files., https://github.com/simonw/sqlite-utils/pull/330#issuecomment-942752844,https://api.github.com/repos/simonw/sqlite-utils/issues/330,942752844,IC_kwDOCGYnMM44MURM,22429695,codecov[bot],2021-10-13T22:00:31Z,2021-10-13T22:11:30Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#330](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fc3de90) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/718a8f61bcaed39c04d5d223104056213f8c8516?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (718a8f6) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/330/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #330 +/- ## ======================================= Coverage 96.59% 96.59% ======================================= Files 5 5 Lines 2230 2230 ======================================= Hits 2154 2154 Misses 76 76 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [718a8f6...fc3de90](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025726600,Test against Python 3.10, https://github.com/simonw/sqlite-utils/pull/326#issuecomment-916119657,https://api.github.com/repos/simonw/sqlite-utils/issues/326,916119657,IC_kwDOCGYnMM42muBp,191622,meatcar,2021-09-09T13:54:10Z,2021-09-09T13:54:10Z,CONTRIBUTOR,dupe of #293?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991237645,Test against 3.10-dev, https://github.com/simonw/sqlite-utils/pull/326#issuecomment-915321467,https://api.github.com/repos/simonw/sqlite-utils/issues/326,915321467,IC_kwDOCGYnMM42jrJ7,22429695,codecov[bot],2021-09-08T15:05:53Z,2021-09-08T15:05:53Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#326](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (078a087) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/77c240df56068341561e95e4a412cbfa24dc5bc7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (77c240d) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/326/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #326 +/- ## ======================================= Coverage 96.58% 96.58% ======================================= Files 5 5 Lines 2223 2223 ======================================= Hits 2147 2147 Misses 76 76 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [77c240d...078a087](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991237645,Test against 3.10-dev, https://github.com/simonw/sqlite-utils/pull/324#issuecomment-968384988,https://api.github.com/repos/simonw/sqlite-utils/issues/324,968384988,IC_kwDOCGYnMM45uGHc,9599,simonw,2021-11-14T23:25:16Z,2021-11-14T23:25:16Z,OWNER,"Yes this was absolutely the intention! Thanks, I wonder how often I've made that mistake in other projects?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988013247,Use python-dateutil package instead of dateutils, https://github.com/simonw/sqlite-utils/pull/322#issuecomment-968401459,https://api.github.com/repos/simonw/sqlite-utils/issues/322,968401459,IC_kwDOCGYnMM45uKIz,9599,simonw,2021-11-15T00:26:42Z,2021-11-15T00:26:42Z,OWNER,"This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947 So the `COLUMN_TYPE_MAPPING` should include those too - right now it looks like this: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L165-L188","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",979612115,Add dict type to be mapped as TEXT in sqllite, https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040902,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905040902,IC_kwDOCGYnMM418dQG,9599,simonw,2021-08-24T23:25:03Z,2021-08-24T23:25:03Z,OWNER,I'm going to skip this test on windows.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040307,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905040307,IC_kwDOCGYnMM418dGz,9599,simonw,2021-08-24T23:23:36Z,2021-08-24T23:23:36Z,OWNER,"https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819 says: > Currently, `TextIOWrapper` uses `locale.getpreferredencoding(False)` (hereinafter called “locale encoding”) when encoding is not specified. > ... > Package authors using macOS or Linux may forget that the default encoding is not always UTF-8.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905039576,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905039576,IC_kwDOCGYnMM418c7Y,9599,simonw,2021-08-24T23:21:29Z,2021-08-24T23:21:29Z,OWNER,"Hah, the error here is actually: ``` > assert result.exit_code == 1, result.output E AssertionError: E E assert 0 == 1 E + where 0 = .exit_code ``` So I was expecting an error, but instead the command worked. I suspect this is because on Windows the default character set may not be UTF-8?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905037323,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905037323,IC_kwDOCGYnMM418cYL,9599,simonw,2021-08-24T23:15:29Z,2021-08-24T23:15:29Z,OWNER,"Huh, tests are failing but only on Windows!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905022931,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905022931,IC_kwDOCGYnMM418Y3T,22429695,codecov[bot],2021-08-24T22:38:38Z,2021-08-24T23:27:26Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#321](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (db2dd6d) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/9258f4bd8450c951900de998a7bf81ca9b45a014?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9258f4b) will **increase** coverage by `0.16%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/321/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #321 +/- ## ========================================== + Coverage 96.41% 96.58% +0.16% ========================================== Files 5 5 Lines 2206 2223 +17 ========================================== + Hits 2127 2147 +20 + Misses 79 76 -3 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/321/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.69% <100.00%> (+0.42%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [9258f4b...db2dd6d](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/pull/32#issuecomment-506784912,https://api.github.com/repos/simonw/sqlite-utils/issues/32,506784912,MDEyOklzc3VlQ29tbWVudDUwNjc4NDkxMg==,9599,simonw,2019-06-28T15:55:13Z,2019-06-28T15:55:13Z,OWNER,"Documentation, to be added as a subsection of https://sqlite-utils.readthedocs.io/en/latest/python-api.html#adding-foreign-key-constraints ### Adding multiple foreign key constraints at once The final step in adding a new foreign key to a SQLite database is to run VACUUM, to ensure the new foreign key is available in future introspection queries. VACUUM against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using table.add_foreign_key(...) these can quickly add up. Instead, you can use db.add_foreign_keys(...) to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a table, column, other_table and other_column. Here's an example adding two foreign keys at once: db.add_foreign_keys([ (""dogs"", ""breed_id"", ""breeds"", ""id""), (""dogs"", ""home_town_id"", ""towns"", ""id"") ]) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",462094937,db.add_foreign_keys() method, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896284722,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896284722,IC_kwDOCGYnMM41bDgy,9599,simonw,2021-08-10T20:08:03Z,2021-08-10T20:08:21Z,OWNER,"Spotted a rogue backtick: ![A0147E27-7506-49B0-BEFB-20D99BBFEBAD](https://user-images.githubusercontent.com/9599/128927930-b3333dee-a385-409b-a945-f108e6ea40df.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896200682,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896200682,IC_kwDOCGYnMM41au_q,9599,simonw,2021-08-10T18:03:40Z,2021-08-10T18:03:40Z,OWNER,"Adding type signatures to `create_table()` and `.create_table_sql()` is a bit too involved, I'll do that in a separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896186025,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896186025,IC_kwDOCGYnMM41arap,9599,simonw,2021-08-10T17:42:51Z,2021-08-10T17:42:51Z,OWNER,That worked! https://sqlite-utils.datasette.io/en/autodoc/reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896182934,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896182934,IC_kwDOCGYnMM41aqqW,9599,simonw,2021-08-10T17:38:44Z,2021-08-10T17:38:44Z,OWNER,"From https://docs.readthedocs.io/en/stable/config-file/v2.html#packages it looks like I can tell Read The Docs to run `pip install -e .` using a `.readthedocs.yaml` configuration: ```yaml version: 2 sphinx: configuration: docs/conf.py python: version: ""3.9"" install: - method: pip path: . extra_requirements: - docs ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896180956,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896180956,IC_kwDOCGYnMM41aqLc,9599,simonw,2021-08-10T17:35:51Z,2021-08-10T17:35:51Z,OWNER,Reading the rest of https://sphinx-rtd-tutorial.readthedocs.io/en/latest/sphinx-config.html#autodoc-configuration it suggests using a `requirements.txt` file to install dependencies - but I use `setup.py` for that so I need to figure out a different pattern here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896175438,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896175438,IC_kwDOCGYnMM41ao1O,9599,simonw,2021-08-10T17:28:19Z,2021-08-10T17:28:19Z,OWNER,"https://sphinx-rtd-tutorial.readthedocs.io/en/latest/sphinx-config.html#autodoc-configuration says do something like this at the top of `conf.py`: ```python import os import sys sys.path.insert(0, os.path.abspath('../../simpleble/')) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896174456,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896174456,IC_kwDOCGYnMM41aol4,9599,simonw,2021-08-10T17:27:01Z,2021-08-10T17:27:01Z,OWNER,"Docs are now building at https://sqlite-utils.datasette.io/en/autodoc/reference.html But there's a problem! The page is semi-blank: I need to teach Read The Docs how to ensure `sqlite_utils` is available for introspection.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896162082,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896162082,IC_kwDOCGYnMM41alki,22429695,codecov[bot],2021-08-10T17:10:39Z,2021-08-10T23:07:35Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#312](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (43bc064) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/ee469e3122d6f5973ec2584c1580d930daca2e7c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ee469e3) will **decrease** coverage by `0.02%`. > The diff coverage is `96.84%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/312/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #312 +/- ## ========================================== - Coverage 96.30% 96.28% -0.03% ========================================== Files 5 5 Lines 2168 2179 +11 ========================================== + Hits 2088 2098 +10 - Misses 80 81 +1 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.91% <96.84%> (-0.08%)` | :arrow_down: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [ee469e3...43bc064](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896156971,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896156971,IC_kwDOCGYnMM41akUr,9599,simonw,2021-08-10T17:04:22Z,2021-08-10T17:05:59Z,OWNER,"I'm going to get Read The Docs to build the docs for this branch too - on https://readthedocs.org/projects/sqlite-utils/versions/ I am clicking this button: I then set it to ""active"" (so pushes to the branch will build it) and ""hidden"" (so it wouldn't show up in search or in the navigation menu). https://docs.readthedocs.io/en/stable/versions.html#version-states ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896154028,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896154028,IC_kwDOCGYnMM41ajms,9599,simonw,2021-08-10T17:01:06Z,2021-08-10T17:01:06Z,OWNER,"On Python 3.6: ``` sqlite_utils/db.py:366: in Database def tables(self) -> List[Table]: E NameError: name 'Table' is not defined ``` Python 3.7 can fix this with `from __future__ import annotations` but since we still support 3.6 I'll have to use a string instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/303#issuecomment-890553014,https://api.github.com/repos/simonw/sqlite-utils/issues/303,890553014,IC_kwDOCGYnMM41FMK2,22429695,codecov[bot],2021-08-01T16:53:35Z,2021-08-02T04:45:19Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#303](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4c3bf97) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/c7e8d72be9fe8fe0811f685a18eebc637662d41b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c7e8d72) will **increase** coverage by `0.24%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/303/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #303 +/- ## ========================================== + Coverage 96.04% 96.28% +0.24% ========================================== Files 4 5 +1 Lines 1998 2129 +131 ========================================== + Hits 1919 2050 +131 Misses 79 79 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/303/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `94.91% <100.00%> (+0.36%)` | :arrow_up: | | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/303/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.97% <100.00%> (+0.10%)` | :arrow_up: | | [sqlite\_utils/recipes.py](https://codecov.io/gh/simonw/sqlite-utils/pull/303/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3JlY2lwZXMucHk=) | `100.00% <100.00%> (ø)` | | | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/303/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `92.39% <100.00%> (+0.48%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c7e8d72...4c3bf97](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957536983,sqlite-utils convert command and db[table].convert(...) method, https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868728092,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868728092,MDEyOklzc3VlQ29tbWVudDg2ODcyODA5Mg==,9599,simonw,2021-06-25T17:39:35Z,2021-06-25T17:39:35Z,OWNER,Here's more about this problem: https://github.com/numpy/numpy/issues/15947,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",929748885,Test against Python 3.10-dev, https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868134040,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868134040,MDEyOklzc3VlQ29tbWVudDg2ODEzNDA0MA==,9599,simonw,2021-06-25T01:49:44Z,2021-06-25T01:50:33Z,OWNER,"Test failed on 3.10 with `numpy` on macOS: ``` sqlite_utils/__init__.py:1: in 11 from .db import Database 12 sqlite_utils/db.py:48: in 13 import numpy as np # type: ignore 14 ../../../hostedtoolcache/Python/3.10.0-beta.3/x64/lib/python3.10/site-packages/numpy/__init__.py:391: in 15 raise RuntimeError(msg) 16 E RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, more information is available at https://numpy.org/doc/stable/user/building.html#accelerated-blas-lapack-libraries Otherwise report this to the vendor that provided NumPy. 17 E RankWarning: Polyfit may be poorly conditioned 18 Error: Process completed with exit code 4. ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",929748885,Test against Python 3.10-dev, https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868125750,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868125750,MDEyOklzc3VlQ29tbWVudDg2ODEyNTc1MA==,22429695,codecov[bot],2021-06-25T01:42:43Z,2021-06-25T01:42:43Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#293](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ae0f46a) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/747be6057d09a4e5d9d726e29d5cf99b10c59dea?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (747be60) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/293/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #293 +/- ## ======================================= Coverage 96.03% 96.03% ======================================= Files 4 4 Lines 1994 1994 ======================================= Hits 1915 1915 Misses 79 79 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [747be60...ae0f46a](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",929748885,Test against Python 3.10-dev, https://github.com/simonw/sqlite-utils/pull/277#issuecomment-864092515,https://api.github.com/repos/simonw/sqlite-utils/issues/277,864092515,MDEyOklzc3VlQ29tbWVudDg2NDA5MjUxNQ==,9599,simonw,2021-06-18T14:47:57Z,2021-06-18T14:47:57Z,OWNER,This is a neat improvement.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",923612361,add -h support closes #276, https://github.com/simonw/sqlite-utils/pull/277#issuecomment-863205049,https://api.github.com/repos/simonw/sqlite-utils/issues/277,863205049,MDEyOklzc3VlQ29tbWVudDg2MzIwNTA0OQ==,22429695,codecov[bot],2021-06-17T12:40:49Z,2021-06-17T12:40:49Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#277](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (abbd324) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a19ce1a4d0048d389411cfe11a5dbe4c503720e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a19ce1a) will **increase** coverage by `0.00%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/277/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #277 +/- ## ======================================= Coverage 96.06% 96.06% ======================================= Files 4 4 Lines 1828 1829 +1 ======================================= + Hits 1756 1757 +1 Misses 72 72 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/277/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `94.03% <100.00%> (+<0.01%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a19ce1a...abbd324](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923612361,add -h support closes #276, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-864099764,https://api.github.com/repos/simonw/sqlite-utils/issues/273,864099764,MDEyOklzc3VlQ29tbWVudDg2NDA5OTc2NA==,9599,simonw,2021-06-18T14:59:27Z,2021-06-18T14:59:27Z,OWNER,I'm going to merge this as-is and work on the JSON/TSV support in a separate issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862817185,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862817185,MDEyOklzc3VlQ29tbWVudDg2MjgxNzE4NQ==,22429695,codecov[bot],2021-06-17T00:15:34Z,2021-06-17T00:15:34Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > :exclamation: No coverage uploaded for pull request base (`main@78aebb6`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#section-missing-base-commit). > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/273/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #273 +/- ## ======================================= Coverage ? 96.10% ======================================= Files ? 4 Lines ? 1873 Branches ? 0 ======================================= Hits ? 1800 Misses ? 73 Partials ? 0 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [78aebb6...df7a37b](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862605436,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862605436,MDEyOklzc3VlQ29tbWVudDg2MjYwNTQzNg==,9599,simonw,2021-06-16T18:19:05Z,2021-06-16T18:19:05Z,OWNER,`--attach` documentation: https://github.com/simonw/sqlite-utils/blob/192dc2c5b73bd836ab8e2e5fed4b36c6ea02f250/docs/cli.rst#joining-in-memory-data-against-existing-databases-using-attach,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862046009,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862046009,MDEyOklzc3VlQ29tbWVudDg2MjA0NjAwOQ==,9599,simonw,2021-06-16T05:15:38Z,2021-06-16T05:15:38Z,OWNER,"I'm going to add a `--encoding` option - it will affect ALL CSV input files, so if you have CSV files with different encodings you'll need to sort that mess out yourself (likely by importing each CSV file separately into a database using `sqlite-utils insert` with different `--encoding` values).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045639,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862045639,MDEyOklzc3VlQ29tbWVudDg2MjA0NTYzOQ==,9599,simonw,2021-06-16T05:14:38Z,2021-06-16T05:14:38Z,OWNER,"Can't share much code though since a bunch of that `insert` stuff is specific to that command - showing progress bars, returning errors on illegal option combinations etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045438,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862045438,MDEyOklzc3VlQ29tbWVudDg2MjA0NTQzOA==,9599,simonw,2021-06-16T05:14:00Z,2021-06-16T05:14:00Z,OWNER,I should probably refactor the CSV/JSON/loading stuff into a function in `utils.py` in order to share some of the implementation with the existing `sqlite-utils insert` code: https://github.com/simonw/sqlite-utils/blob/287cdcae8908916687f2ecccc87c38549d004ac6/sqlite_utils/cli.py#L691-L734,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862043974,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862043974,MDEyOklzc3VlQ29tbWVudDg2MjA0Mzk3NA==,9599,simonw,2021-06-16T05:10:12Z,2021-06-16T05:10:12Z,OWNER,"I can stop promoting `:memory:` here and promote `memory` instead: https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#L83-L86","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862042110,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862042110,MDEyOklzc3VlQ29tbWVudDg2MjA0MjExMA==,9599,simonw,2021-06-16T05:05:51Z,2021-06-16T05:06:11Z,OWNER,"Initial documentation is here: https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#running-queries-directly-against-csv-data It only talks about CSV at the moment - needs to be updated to mention JSON too once that is implemented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data, https://github.com/simonw/sqlite-utils/pull/258#issuecomment-843702392,https://api.github.com/repos/simonw/sqlite-utils/issues/258,843702392,MDEyOklzc3VlQ29tbWVudDg0MzcwMjM5Mg==,9599,simonw,2021-05-19T02:47:37Z,2021-05-19T02:47:37Z,OWNER,I'm going to merge this and add a test - thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",868191959,Fixing insert from JSON containing strings with non-ascii characters …, https://github.com/simonw/sqlite-utils/pull/254#issuecomment-843705533,https://api.github.com/repos/simonw/sqlite-utils/issues/254,843705533,MDEyOklzc3VlQ29tbWVudDg0MzcwNTUzMw==,9599,simonw,2021-05-19T02:57:22Z,2021-05-19T02:57:22Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",857280617,Fix incorrect create-table cli description, https://github.com/simonw/sqlite-utils/pull/247#issuecomment-901344634,https://api.github.com/repos/simonw/sqlite-utils/issues/247,901344634,IC_kwDOCGYnMM41uW16,22429695,codecov[bot],2021-08-18T18:42:54Z,2021-08-18T18:42:54Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#247](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (af989af) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/1fe73c898b44695052f1a9ca832818d50cecf662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1fe73c8) will **decrease** coverage by `0.03%`. > The diff coverage is `85.71%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/247/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #247 +/- ## ========================================== - Coverage 96.28% 96.24% -0.04% ========================================== Files 5 5 Lines 2179 2186 +7 ========================================== + Hits 2098 2104 +6 - Misses 81 82 +1 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.84% <85.71%> (-0.08%)` | :arrow_down: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1fe73c8...af989af](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832687563,FTS quote functionality from datasette, https://github.com/simonw/sqlite-utils/pull/247#issuecomment-901338988,https://api.github.com/repos/simonw/sqlite-utils/issues/247,901338988,IC_kwDOCGYnMM41uVds,9599,simonw,2021-08-18T18:33:39Z,2021-08-18T18:33:39Z,OWNER,This was also requested in #296.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832687563,FTS quote functionality from datasette, https://github.com/simonw/sqlite-utils/pull/245#issuecomment-843705721,https://api.github.com/repos/simonw/sqlite-utils/issues/245,843705721,MDEyOklzc3VlQ29tbWVudDg0MzcwNTcyMQ==,9599,simonw,2021-05-19T02:58:01Z,2021-05-19T02:58:01Z,OWNER,Thanks very much.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830803173,Correct some typos, https://github.com/simonw/sqlite-utils/pull/244#issuecomment-843705806,https://api.github.com/repos/simonw/sqlite-utils/issues/244,843705806,MDEyOklzc3VlQ29tbWVudDg0MzcwNTgwNg==,9599,simonw,2021-05-19T02:58:18Z,2021-05-19T02:58:18Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",820468864,Typo in upsert example, https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778841547,https://api.github.com/repos/simonw/sqlite-utils/issues/225,778841547,MDEyOklzc3VlQ29tbWVudDc3ODg0MTU0Nw==,9599,simonw,2021-02-14T21:04:13Z,2021-02-14T21:04:13Z,OWNER,I added a test and fixed this in #234 - thanks for the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797159961,fix for problem in Table.insert_all on search for columns per chunk of rows, https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778834504,https://api.github.com/repos/simonw/sqlite-utils/issues/225,778834504,MDEyOklzc3VlQ29tbWVudDc3ODgzNDUwNA==,9599,simonw,2021-02-14T20:09:30Z,2021-02-14T20:09:30Z,OWNER,Thanks for this. I'm going to try and get the test suite to run in Windows on GitHub Actions.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797159961,fix for problem in Table.insert_all on search for columns per chunk of rows, https://github.com/simonw/sqlite-utils/pull/224#issuecomment-778828495,https://api.github.com/repos/simonw/sqlite-utils/issues/224,778828495,MDEyOklzc3VlQ29tbWVudDc3ODgyODQ5NQ==,9599,simonw,2021-02-14T19:31:06Z,2021-02-14T19:31:06Z,OWNER,I'm going to add a `offset=` parameter to support this case. Thanks for the suggestion!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792297010,Add fts offset docs., https://github.com/simonw/sqlite-utils/pull/224#issuecomment-765678057,https://api.github.com/repos/simonw/sqlite-utils/issues/224,765678057,MDEyOklzc3VlQ29tbWVudDc2NTY3ODA1Nw==,37962604,polyrand,2021-01-22T20:53:06Z,2021-01-23T20:13:27Z,NONE,"I'm using the FTS methods in sqlite-utils for this website: [drwn.io](https://drwn.io/). I wanted to get pagination to have some kind of infinite scrolling in the landing page, and I ended up using that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792297010,Add fts offset docs., https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743956666,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743956666,MDEyOklzc3VlQ29tbWVudDc0Mzk1NjY2Ng==,9599,simonw,2020-12-13T05:44:49Z,2020-12-13T05:44:49Z,OWNER,"Example output: ``` % sqlite-utils analyze-tables github.db tags tags.repo: (1/3) Total rows: 261 Null rows: 0 Blank rows: 0 Distinct values: 14 Most common: 88: 107914493 75: 140912432 27: 206156866 21: 207052882 17: 197431109 8: 197882382 5: 256834907 5: 205429375 4: 248903544 3: 206202864 Least common: 1: 209590345 2: 206649770 2: 303218369 3: 206202864 3: 213286752 4: 248903544 5: 205429375 5: 256834907 8: 197882382 17: 197431109 tags.name: (2/3) Total rows: 261 Null rows: 0 Blank rows: 0 Distinct values: 175 Most common: 10: 0.2 9: 0.1 7: 0.3 6: 0.4 5: 0.7 5: 0.5 5: 0.1a 4: 0.9 4: 0.8 4: 0.6 Least common: 1: 0.1.1 1: 0.11.1 1: 0.1a2 1: 0.20.1 1: 0.21.1 1: 0.21.2 1: 0.21.3 1: 0.22 1: 0.22.1 1: 0.23 tags.sha: (3/3) Total rows: 261 Null rows: 0 Blank rows: 0 Distinct values: 261 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method, https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708524,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743708524,MDEyOklzc3VlQ29tbWVudDc0MzcwODUyNA==,9599,simonw,2020-12-12T05:48:20Z,2020-12-12T05:48:32Z,OWNER,"``` % sqlite-utils analyze-tables ../datasette/fixtures.db facetable --column pk 1/1: ColumnDetails(table='facetable', column='pk', total_rows=15, num_null=0, num_blank=0, num_distinct=15, most_common=None, least_common=None) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method, https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708325,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743708325,MDEyOklzc3VlQ29tbWVudDc0MzcwODMyNQ==,9599,simonw,2020-12-12T05:46:27Z,2020-12-12T05:46:27Z,OWNER,"It would be neat if you could optionally specify a subset of columns to analyze, using `-c` or `--column`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method, https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708169,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743708169,MDEyOklzc3VlQ29tbWVudDc0MzcwODE2OQ==,9599,simonw,2020-12-12T05:44:46Z,2020-12-12T05:44:46Z,OWNER,"If there are less than ten values is it worth outputting them twice, once in `most_common` and then in reverse in `least_common`? Feels redundant - I think I should leave `least_common` empty in that case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method, https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708080,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743708080,MDEyOklzc3VlQ29tbWVudDc0MzcwODA4MA==,9599,simonw,2020-12-12T05:43:45Z,2020-12-12T05:43:45Z,OWNER,"CLI output looks like this at the moment, which is bad: ``` % sqlite-utils analyze-tables ../datasette/fixtures.db facetable 1/10: ColumnDetails(table='facetable', column='pk', total_rows=15, num_null=0, num_blank=0, num_distinct=15, most_common=None, least_common=None) 2/10: ColumnDetails(table='facetable', column='created', total_rows=15, num_null=0, num_blank=0, num_distinct=4, most_common=[('2019-01-17 08:00:00', 4), ('2019-01-15 08:00:00', 4), ('2019-01-14 08:00:00', 4), ('2019-01-16 08:00:00', 3)], least_common=[('2019-01-16 08:00:00', 3), ('2019-01-14 08:00:00', 4), ('2019-01-15 08:00:00', 4), ('2019-01-17 08:00:00', 4)]) 3/10: ColumnDetails(table='facetable', column='planet_int', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[(1, 14), (2, 1)], least_common=[(2, 1), (1, 14)]) 4/10: ColumnDetails(table='facetable', column='on_earth', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[(1, 14), (0, 1)], least_common=[(0, 1), (1, 14)]) 5/10: ColumnDetails(table='facetable', column='state', total_rows=15, num_null=0, num_blank=0, num_distinct=3, most_common=[('CA', 10), ('MI', 4), ('MC', 1)], least_common=[('MC', 1), ('MI', 4), ('CA', 10)]) 6/10: ColumnDetails(table='facetable', column='city_id', total_rows=15, num_null=0, num_blank=0, num_distinct=4, most_common=[(1, 6), (3, 4), (2, 4), (4, 1)], least_common=[(4, 1), (2, 4), (3, 4), (1, 6)]) 7/10: ColumnDetails(table='facetable', column='neighborhood', total_rows=15, num_null=0, num_blank=0, num_distinct=14, most_common=[('Downtown', 2), ('Tenderloin', 1), ('SOMA', 1), ('Mission', 1), ('Mexicantown', 1), ('Los Feliz', 1), ('Koreatown', 1), ('Hollywood', 1), ('Hayes Valley', 1), ('Greektown', 1)], least_common=[('Arcadia Planitia', 1), ('Bernal Heights', 1), ('Corktown', 1), ('Dogpatch', 1), ('Greektown', 1), ('Hayes Valley', 1), ('Hollywood', 1), ('Koreatown', 1), ('Los Feliz', 1), ('Mexicantown', 1)]) 8/10: ColumnDetails(table='facetable', column='tags', total_rows=15, num_null=0, num_blank=0, num_distinct=3, most_common=[('[]', 13), ('[""tag1"", ""tag3""]', 1), ('[""tag1"", ""tag2""]', 1)], least_common=[('[""tag1"", ""tag2""]', 1), ('[""tag1"", ""tag3""]', 1), ('[]', 13)]) 9/10: ColumnDetails(table='facetable', column='complex_array', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[('[]', 14), ('[{""foo"": ""bar""}]', 1)], least_common=[('[{""foo"": ""bar""}]', 1), ('[]', 14)]) 10/10: ColumnDetails(table='facetable', column='distinct_some_null', total_rows=15, num_null=13, num_blank=0, num_distinct=2, most_common=[(None, 13), ('two', 1), ('one', 1)], least_common=[('one', 1), ('two', 1), (None, 13)]) (sqlite-utils) sqlite-utils % ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method, https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743707969,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743707969,MDEyOklzc3VlQ29tbWVudDc0MzcwNzk2OQ==,9599,simonw,2020-12-12T05:42:26Z,2020-12-12T05:43:06Z,OWNER,"Should truncate values in the least/most common JSON array to a sensible length, otherwise you end up with stuff like this: ```json [ [ ""b'\\x00\\x05barry\\x03\\x01\\x02\\x00\\x00\\x03cat\\x03\\x01\\x03\\x00\\x00\\x03dog\\x08\\x01\\x01\\x01\\x03\\x00\\x01\\x03\\x00\\x00\\x07panther\\x05\\x01\\x01\\x02\\x02\\x00\\x01\\x03uma\\x05\\x02\\x01\\x02\\x02\\x00\\x00\\x04sara\\x05\\x02\\x01\\x01\\x02\\x00\\x00\\x05terry\\x08\\x01\\x01\\x01\\x02\\x00\\x01\\x02\\x00\\x00\\x06weasel\\x05\\x02\\x01\\x01\\x03\\x00'"", 1 ] ] ``` This example also shows that binary values (like those in `_fts` tables) look a bit weird, but I think I'm OK with that since binary data can't be represented neatly in JSON anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method, https://github.com/simonw/sqlite-utils/pull/204#issuecomment-740796067,https://api.github.com/repos/simonw/sqlite-utils/issues/204,740796067,MDEyOklzc3VlQ29tbWVudDc0MDc5NjA2Nw==,9599,simonw,2020-12-08T17:49:22Z,2020-12-08T17:49:22Z,OWNER,"Great catch, thank you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752888228,use jsonify_if_need for sql updates, https://github.com/simonw/sqlite-utils/pull/203#issuecomment-774217792,https://api.github.com/repos/simonw/sqlite-utils/issues/203,774217792,MDEyOklzc3VlQ29tbWVudDc3NDIxNzc5Mg==,1049910,drkane,2021-02-05T18:44:13Z,2021-02-05T18:44:13Z,NONE,"Thanks for looking at this - home schooling kids has prevented me from replying. I'd struggled with how to adapt the API for the foreign keys too - I definitely tried the String/Tuple approach. I hadn't considered the breaking changes that would introduce though. I can take a look at this and try and make the change - see which of your options works best. I've got a workaround for the use-case I was looking at this for, so it wouldn't be a problem for me if it was put on the back burner until a hypothetical v4.0 anyway. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys, https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932,https://api.github.com/repos/simonw/sqlite-utils/issues/203,753567932,MDEyOklzc3VlQ29tbWVudDc1MzU2NzkzMg==,9599,simonw,2021-01-03T04:54:43Z,2021-01-03T04:54:43Z,OWNER,"Another option: expand the `ForeignKey` object to have `.columns` and `.other_columns` properties in addition to the existing `.column` and `.other_column` properties. These new plural properties would always return a tuple, which would be a one-item tuple for a non-compound-foreign-key. The question then is what should `.column` and `.other_column` return for compound foreign keys? I'd be inclined to say they should return `None` - which would trigger errors in code that encounters a compound foreign key for the first time, but those errors would at least be a strong indicator as to what had gone wrong. We can label `.column` and `.other_column` as deprecated and then remove them in `sqlite-utils 4.0`. Since this would still be a breaking change in some minor edge-cases I'm thinking maybe 4.0 needs to happen in order to land this feature. I'm not opposed to doing that, I was just hoping it might be avoidable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys, https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567744,https://api.github.com/repos/simonw/sqlite-utils/issues/203,753567744,MDEyOklzc3VlQ29tbWVudDc1MzU2Nzc0NA==,9599,simonw,2021-01-03T04:51:44Z,2021-01-03T04:51:44Z,OWNER,"One way that this could avoid a breaking change would be to have `fk.column` and `fk.other_column` remain as strings for non-compound-foreign-keys, but turn into tuples for a compound foreign key. This is a bit of an ugly API design, and it could still break existing code that encounters a compound foreign key for the first time - but it would leave code working for the more common case of a non-compound-foreign-key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys, https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567508,https://api.github.com/repos/simonw/sqlite-utils/issues/203,753567508,MDEyOklzc3VlQ29tbWVudDc1MzU2NzUwOA==,9599,simonw,2021-01-03T04:48:17Z,2021-01-03T04:48:17Z,OWNER,"Sorry for taking so long to review this! This approach looks great to me - being able to optionally pass a tuple anywhere the API currently expects a column is smart, and it's consistent with how the `pk=` parameter works elsewhere. There's just one problem I can see with this: the way it changes the `ForeignKey(...)` interface to always return a tuple for `.column` and `.other_column`, even if that tuple only contains a single item. This represents a breaking change to the existing API - any code that expects `ForeignKey.column` to be a single string (which is any code that has been written against that) will break. As such, I'd have to bump the major version of `sqlite-utils` to `4.0` in order to ship this. Ideally I'd like to make this change in a way that doesn't represent an API compatibility break. I need to think a bit harder about how that might be achieved.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys, https://github.com/simonw/sqlite-utils/pull/203#issuecomment-743966289,https://api.github.com/repos/simonw/sqlite-utils/issues/203,743966289,MDEyOklzc3VlQ29tbWVudDc0Mzk2NjI4OQ==,9599,simonw,2020-12-13T07:20:51Z,2020-12-13T07:20:51Z,OWNER,Sorry for not reviewing this yet! I'll try to carve out time to look at it in the next few days.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys, https://github.com/simonw/sqlite-utils/pull/203#issuecomment-1404070841,https://api.github.com/repos/simonw/sqlite-utils/issues/203,1404070841,IC_kwDOCGYnMM5TsGu5,536941,fgregg,2023-01-25T18:47:18Z,2023-01-25T18:47:18Z,CONTRIBUTOR,i'll adopt this PR to make the changes @simonw suggested https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys, https://github.com/simonw/sqlite-utils/pull/203#issuecomment-1033641009,https://api.github.com/repos/simonw/sqlite-utils/issues/203,1033641009,IC_kwDOCGYnMM49nBwx,82988,psychemedia,2022-02-09T11:06:18Z,2022-02-09T11:06:18Z,NONE,"Is there any progress elsewhere on the handling of compound / composite foreign keys, or is this PR still effectively open?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys, https://github.com/simonw/sqlite-utils/pull/195#issuecomment-723148906,https://api.github.com/repos/simonw/sqlite-utils/issues/195,723148906,MDEyOklzc3VlQ29tbWVudDcyMzE0ODkwNg==,9599,simonw,2020-11-06T15:43:51Z,2020-11-06T15:43:51Z,OWNER,Thanks to #198 (introducing a `rank_bm25()` custom function for FTS4) this feature will be able to offer relevance search for both FTS5 AND FTS4 tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735663855,table.search() improvements plus sqlite-utils search command, https://github.com/simonw/sqlite-utils/pull/195#issuecomment-722542895,https://api.github.com/repos/simonw/sqlite-utils/issues/195,722542895,MDEyOklzc3VlQ29tbWVudDcyMjU0Mjg5NQ==,9599,simonw,2020-11-05T18:01:33Z,2020-11-05T18:01:33Z,OWNER,"Latest test failure: ``` 114 -> assert [(""racoons are biting trash pandas"", ""USA"", ""bar"")] == table.search( 115 ""bite"", order=""rowid"" 116 ) 117 118 119 def test_optimize_fts(fresh_db): (Pdb) table.search(""bite"") [(2, 'racoons are biting trash pandas', 'USA', 'bar', -9.641434262948206e-07)] ``` The problem here is that the `table.search()` method now behaves differently for FTS4 v.s. FTS5 tables. With FTS4 you get back just the table columns. With FTS5 you also get back the `rowid` as the first column and the `rank` score as the last column. This is weird. It also makes me question whether having `.search()` return a list of tuples is the right API design.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735663855,table.search() improvements plus sqlite-utils search command, https://github.com/simonw/sqlite-utils/pull/195#issuecomment-721397665,https://api.github.com/repos/simonw/sqlite-utils/issues/195,721397665,MDEyOklzc3VlQ29tbWVudDcyMTM5NzY2NQ==,9599,simonw,2020-11-03T22:02:57Z,2020-11-03T22:02:57Z,OWNER,Documentation so far: https://github.com/simonw/sqlite-utils/blob/6cadc6103ff1ba58c6409ce7fba74259e72965d9/docs/cli.rst#executing-searches,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735663855,table.search() improvements plus sqlite-utils search command, https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717361487,https://api.github.com/repos/simonw/sqlite-utils/issues/189,717361487,MDEyOklzc3VlQ29tbWVudDcxNzM2MTQ4Nw==,9599,simonw,2020-10-27T16:24:04Z,2020-10-27T16:24:04Z,OWNER,"This is great, thank you very much.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records, https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717359145,https://api.github.com/repos/simonw/sqlite-utils/issues/189,717359145,MDEyOklzc3VlQ29tbWVudDcxNzM1OTE0NQ==,35681,adamwolf,2020-10-27T16:20:32Z,2020-10-27T16:20:32Z,CONTRIBUTOR,"No problem. I added a test. Let me know if it looks sufficient or if you want me to to tweak something! If you don't mind, would you tag this PR as ""hacktoberfest-accepted""? If you do mind, no problem and I'm sorry for asking :) My kiddos like the shirts.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records, https://github.com/simonw/sqlite-utils/pull/189#issuecomment-716756103,https://api.github.com/repos/simonw/sqlite-utils/issues/189,716756103,MDEyOklzc3VlQ29tbWVudDcxNjc1NjEwMw==,9599,simonw,2020-10-26T18:56:19Z,2020-10-26T18:56:19Z,OWNER,"This is a great fix, thanks! If you add a unit test somewhere in here I'll merge the PR: https://github.com/simonw/sqlite-utils/blob/main/tests/test_m2m.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records, https://github.com/simonw/sqlite-utils/pull/178#issuecomment-701627158,https://api.github.com/repos/simonw/sqlite-utils/issues/178,701627158,MDEyOklzc3VlQ29tbWVudDcwMTYyNzE1OA==,9599,simonw,2020-09-30T20:29:11Z,2020-09-30T20:29:11Z,OWNER,Thanks for the fix!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709043182,Update README.md, https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698400790,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698400790,MDEyOklzc3VlQ29tbWVudDY5ODQwMDc5MA==,9599,simonw,2020-09-24T14:59:50Z,2020-09-24T14:59:50Z,OWNER,For reusing the lookup table: I'm going to raise an error if a lookup table exists but without the correct columns. The caller can then add those columns and try again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698184166,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698184166,MDEyOklzc3VlQ29tbWVudDY5ODE4NDE2Ng==,9599,simonw,2020-09-24T08:01:07Z,2020-09-24T08:01:07Z,OWNER,I may revert the now unnecessary undocumented tweaks to the `.update()` method made in 66d506587eba9f0715267d6560b97c1fa44cc781 as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698182656,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698182656,MDEyOklzc3VlQ29tbWVudDY5ODE4MjY1Ng==,9599,simonw,2020-09-24T07:58:08Z,2020-09-24T07:58:08Z,OWNER,"The way the lookup table works here differs from the previous implementation. In the previous implementation the usage of `.lookup()` meant that an existing table would be modified to fit the new purpose. That no longer happens in this version. Need to make a design decision about how this should work. It should definitely be possible to use an existing lookup table - imagine a database where several tables have a ""Departments"" column and we want to extract all of those values out to a single shared ""Departments"" table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698182037,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698182037,MDEyOklzc3VlQ29tbWVudDY5ODE4MjAzNw==,9599,simonw,2020-09-24T07:56:50Z,2020-09-24T07:56:50Z,OWNER,I could also be a bit smarter about transaction handling. I think it may be possible to run this entire operation in a single transaction now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698181478,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698181478,MDEyOklzc3VlQ29tbWVudDY5ODE4MTQ3OA==,9599,simonw,2020-09-24T07:55:45Z,2020-09-24T07:55:45Z,OWNER,`import functools` is no longer needed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698180705,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698180705,MDEyOklzc3VlQ29tbWVudDY5ODE4MDcwNQ==,9599,simonw,2020-09-24T07:54:10Z,2020-09-24T07:54:10Z,OWNER,"After running through the steps in https://simonwillison.net/2020/Sep/23/sqlite-utils-extract/ I get a table that looks like this: The foreign key columns are all at the end of the table. It would be nicer if they were arranged in the same order as the columns they replaced.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698180113,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698180113,MDEyOklzc3VlQ29tbWVudDY5ODE4MDExMw==,9599,simonw,2020-09-24T07:53:03Z,2020-09-24T07:53:03Z,OWNER,This could do with a little bit more testing - I'm worried there may be column or table name edge cases that are not covered yet. I also need to remove the progress bar code since that no longer makes sense for this implementation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696494070,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696494070,MDEyOklzc3VlQ29tbWVudDY5NjQ5NDA3MA==,9599,simonw,2020-09-22T03:48:58Z,2020-09-22T03:48:58Z,OWNER,"One last thing. https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_change says that the first step should be: > If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. And the last steps should be: > If foreign key constraints were originally enabled then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints. > > Commit the transaction started in step 2. > > If foreign keys constraints were originally enabled, reenable them now. I need to implement that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696490851,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696490851,MDEyOklzc3VlQ29tbWVudDY5NjQ5MDg1MQ==,9599,simonw,2020-09-22T03:33:54Z,2020-09-22T03:33:54Z,OWNER,It would be neat if `.transform(pk=None)` converted a primary key table to a rowid table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696488201,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696488201,MDEyOklzc3VlQ29tbWVudDY5NjQ4ODIwMQ==,9599,simonw,2020-09-22T03:21:16Z,2020-09-22T03:21:16Z,OWNER,Just needs documentation now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696485791,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696485791,MDEyOklzc3VlQ29tbWVudDY5NjQ4NTc5MQ==,9599,simonw,2020-09-22T03:10:15Z,2020-09-22T03:10:15Z,OWNER,"Design decision needed on foreign keys: what does the syntax look like for removing an existing foreign key? Since I already have a good implementation of `add_foreign_key()` I'm tempted to only support dropping them. Maybe like this: ```python table.transform(drop_foreign_keys=[(""author_id"", ""author"", ""id"")]) ``` It's a bit crufty but it's such a rare use-case that I think this will be good enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696480925,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696480925,MDEyOklzc3VlQ29tbWVudDY5NjQ4MDkyNQ==,9599,simonw,2020-09-22T02:45:47Z,2020-09-22T02:45:47Z,OWNER,"I'm not going to do `conversions=` because it would be inconsistent with how they work elsewhere. The SQL generated by this function looks like this: INSERT INTO dogs_new_tmp VALUES (a, b) SELECT a, b from dogs; So passing `conversions={""name"": ""upper(?)""})` wouldn't make sense, since we're not using arguments hence there is no-where for that `?` to go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696446658,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696446658,MDEyOklzc3VlQ29tbWVudDY5NjQ0NjY1OA==,9599,simonw,2020-09-22T00:13:55Z,2020-09-22T00:14:21Z,OWNER,"Idea: allow a `conversions=` parameter, as seen on `.insert_all()` and friends, which lets you apply a SQL transformation function as part of the operation. E.g.: ```python table.transform({""age"": int}, conversions={""name"": ""upper(?)""}) ``` https://sqlite-utils.readthedocs.io/en/stable/python-api.html#converting-column-values-using-sql-functions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696445766,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696445766,MDEyOklzc3VlQ29tbWVudDY5NjQ0NTc2Ng==,9599,simonw,2020-09-22T00:10:50Z,2020-09-22T00:11:12Z,OWNER,"A less horrible interface might be the following: ```python # Ensure the 'age' column is not null: table.transform(not_null={""age""}) # The 'age' column is not null but I don't want it to be: table.transform(not_null={""age"": False}) ``` So if the argument is a set it means ""make sure these are all not null"" - if the argument is a dictionary it means ""set these to be null or not null depending on if their dictionary value is true or false"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444842,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696444842,MDEyOklzc3VlQ29tbWVudDY5NjQ0NDg0Mg==,9599,simonw,2020-09-22T00:07:43Z,2020-09-22T00:09:05Z,OWNER,"Syntax challenge: I could use `.transform(defaults={""age"": None})` to indicate that the `age` column should have its default removed, but how would I tell `.transform()` that the `age` column, currently `not null`, should have the `not null` removed from it? I could do this: `.transform(not_not_null={""age""})` - it's a bit gross but it's also kind of funny. I actually like it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444353,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696444353,MDEyOklzc3VlQ29tbWVudDY5NjQ0NDM1Mw==,9599,simonw,2020-09-22T00:06:12Z,2020-09-22T00:06:12Z,OWNER,I should support `not_null=` and `default=` arguments to the `.transform()` method because it looks like you can't use `ALTER TABLE` to change those.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443845,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443845,MDEyOklzc3VlQ29tbWVudDY5NjQ0Mzg0NQ==,9599,simonw,2020-09-22T00:04:31Z,2020-09-22T00:04:44Z,OWNER,"Good news: the `.columns` introspection does tell me those things: ``` >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db.create_table(""foo"", {""id"": int, ""name"": str, ""age"": int}, defaults={""age"": 1}, not_null={""name"", ""age""})
>>> db[""foo""]
>>> print(db[""foo""].schema) CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT NOT NULL, [age] INTEGER NOT NULL DEFAULT 1 ) >>> db[""foo""].columns [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value=None, is_pk=0), Column(cid=2, name='age', type='INTEGER', notnull=1, default_value='1', is_pk=0)] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443190,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443190,MDEyOklzc3VlQ29tbWVudDY5NjQ0MzE5MA==,9599,simonw,2020-09-22T00:02:22Z,2020-09-22T00:02:22Z,OWNER,How would I detect which columns are `not_null` and what their defaults are? I don`t think my introspection logic handles that yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443042,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443042,MDEyOklzc3VlQ29tbWVudDY5NjQ0MzA0Mg==,9599,simonw,2020-09-22T00:01:50Z,2020-09-22T00:01:50Z,OWNER,"When you transform a table, it should keep its primary key, foreign keys, not_null and defaults. I don't think it needs to care about `hash_id` or `extracts=` since those don't affect the structure of the table as it is being created - well, `hash_id` does but if we are transforming an existing table we will get the `hash_id` column for free.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696442621,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696442621,MDEyOklzc3VlQ29tbWVudDY5NjQ0MjYyMQ==,9599,simonw,2020-09-22T00:00:23Z,2020-09-22T00:00:23Z,OWNER,I still need to figure out what to do about these various other table properties: https://github.com/simonw/sqlite-utils/blob/b34c9b40c206d7a9d7ee57a8c1f198ff1f522735/sqlite_utils/db.py#L775-L787,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/158#issuecomment-693199392,https://api.github.com/repos/simonw/sqlite-utils/issues/158,693199392,MDEyOklzc3VlQ29tbWVudDY5MzE5OTM5Mg==,9599,simonw,2020-09-16T06:21:29Z,2020-09-16T06:21:29Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",697203800,Fix accidental mega long line in docs, https://github.com/simonw/sqlite-utils/pull/156#issuecomment-689735140,https://api.github.com/repos/simonw/sqlite-utils/issues/156,689735140,MDEyOklzc3VlQ29tbWVudDY4OTczNTE0MA==,9599,simonw,2020-09-09T18:21:06Z,2020-09-09T18:21:06Z,OWNER,"Good spot, thanks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",697030843,Typos in tests, https://github.com/simonw/sqlite-utils/pull/146#issuecomment-689185393,https://api.github.com/repos/simonw/sqlite-utils/issues/146,689185393,MDEyOklzc3VlQ29tbWVudDY4OTE4NTM5Mw==,9599,simonw,2020-09-08T23:17:42Z,2020-09-08T23:17:42Z,OWNER,"That seems like a reasonable approach to me, especially since this is going to be a pretty rare edge-case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688668680,Handle case where subsequent records (after first batch) include extra columns, https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688573964,https://api.github.com/repos/simonw/sqlite-utils/issues/146,688573964,MDEyOklzc3VlQ29tbWVudDY4ODU3Mzk2NA==,96218,simonwiles,2020-09-08T01:55:07Z,2020-09-08T01:55:07Z,CONTRIBUTOR,"Okay, I've rewritten this PR to preserve the batching behaviour but still fix #145, and rebased the branch to account for the `db.execute()` api change. It's not terribly sophisticated -- if it attempts to insert a batch which has too many variables, the exception is caught, the batch is split in two and each half is inserted separately, and then it carries on as before with the same `batch_size`. In the edge case where this gets triggered, subsequent batches will all be inserted in two groups too if they continue to have the same number of columns (which is presumably reasonably likely). Do you reckon this is acceptable when set against the awkwardness of recalculating the `batch_size` on the fly?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688668680,Handle case where subsequent records (after first batch) include extra columns, https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688508510,https://api.github.com/repos/simonw/sqlite-utils/issues/146,688508510,MDEyOklzc3VlQ29tbWVudDY4ODUwODUxMA==,9599,simonw,2020-09-07T20:56:03Z,2020-09-07T20:56:24Z,OWNER,"The problem with this approach is that it requires us to consume the entire iterator before we can start inserting rows into the table - here on line 1052: https://github.com/simonw/sqlite-utils/blob/bb131793feac16bc7181ab997568f941b0220ef2/sqlite_utils/db.py#L1047-L1054 I designed the `.insert_all()` to avoid doing this, because I want to be able to pass it an iterator (or more likely a generator) that could produce potentially millions of records. Doing things one batch of 100 records at a time means that the Python process doesn't need to pull millions of records into memory at once. `db-to-sqlite` is one example of a tool that uses that characteristic, in https://github.com/simonw/db-to-sqlite/blob/63e4ee972f292de13bb11767c0fb64b35339d954/db_to_sqlite/cli.py#L94-L106 So we need to solve this issue without consuming the entire iterator with a `records = list(records)` call. I think one way to do this is to execute each chunk one at a time and watch out for an exception that indicates that we sent too many parameters - then adjust the chunk size down and try again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688668680,Handle case where subsequent records (after first batch) include extra columns, https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688481317,https://api.github.com/repos/simonw/sqlite-utils/issues/146,688481317,MDEyOklzc3VlQ29tbWVudDY4ODQ4MTMxNw==,96218,simonwiles,2020-09-07T19:18:55Z,2020-09-07T19:18:55Z,CONTRIBUTOR,"Just force-pushed to update d042f9c with more formatting changes to satisfy `black==20.8b1` and pass the GitHub Actions ""Test"" workflow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688668680,Handle case where subsequent records (after first batch) include extra columns, https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688479163,https://api.github.com/repos/simonw/sqlite-utils/issues/146,688479163,MDEyOklzc3VlQ29tbWVudDY4ODQ3OTE2Mw==,96218,simonwiles,2020-09-07T19:10:33Z,2020-09-07T19:11:57Z,CONTRIBUTOR,"@simonw -- I've gone ahead updated the documentation to reflect the changes introduced in this PR. IMO it's ready to merge now. In writing the documentation changes, I begin to wonder about the value and role of `batch_size` at all, tbh. May I assume it was originally intended to prevent using the entire row set to determine columns and column types, and that this was a performance consideration? If so, this PR entirely undermines its purpose. I've been passing in excess of 500,000 rows at a time to `insert_all()` with these changes and although I'm sure the performance difference is measurable it's not really noticeable; given #145, I don't know that any performance advantages outweigh the problems doing it this way removes. What do you think about just dropping the argument and defaulting to the maximum `batch_size` permissible given `SQLITE_MAX_VARS`? Are there other reasons one might want to restrict `batch_size` that I've overlooked? I could open a new issue to discuss/implement this. Of course the documentation will need to change again too if/when something is done about #147.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688668680,Handle case where subsequent records (after first batch) include extra columns, https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683173375,https://api.github.com/repos/simonw/sqlite-utils/issues/142,683173375,MDEyOklzc3VlQ29tbWVudDY4MzE3MzM3NQ==,9599,simonw,2020-08-28T22:29:02Z,2020-08-28T22:29:02Z,OWNER,Yeah I think that failure is actually because there's a brand new release of Black out and it subtly changes some of the formatting rules. I'll merge this and then run Black against the entire codebase.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688386219,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records", https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172829,https://api.github.com/repos/simonw/sqlite-utils/issues/142,683172829,MDEyOklzc3VlQ29tbWVudDY4MzE3MjgyOQ==,9599,simonw,2020-08-28T22:27:05Z,2020-08-28T22:27:05Z,OWNER,"Looks like it failed the ""black"" formatting test - possibly because there's a new release if black out. I'm going to merge despite that failure.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688386219,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records", https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172082,https://api.github.com/repos/simonw/sqlite-utils/issues/142,683172082,MDEyOklzc3VlQ29tbWVudDY4MzE3MjA4Mg==,9599,simonw,2020-08-28T22:24:25Z,2020-08-28T22:24:25Z,OWNER,Thanks very much!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688386219,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records", https://github.com/simonw/sqlite-utils/pull/120#issuecomment-655289686,https://api.github.com/repos/simonw/sqlite-utils/issues/120,655289686,MDEyOklzc3VlQ29tbWVudDY1NTI4OTY4Ng==,9599,simonw,2020-07-08T05:13:11Z,2020-07-08T05:13:11Z,OWNER,"This is an excellent fix, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652816158,Fix query command's support for DML, https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655653292,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655653292,MDEyOklzc3VlQ29tbWVudDY1NTY1MzI5Mg==,9599,simonw,2020-07-08T17:26:02Z,2020-07-08T17:26:02Z,OWNER,"Awesome, thank you very much.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option, https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655643078,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655643078,MDEyOklzc3VlQ29tbWVudDY1NTY0MzA3OA==,79913,tsibley,2020-07-08T17:05:59Z,2020-07-08T17:05:59Z,CONTRIBUTOR,"> The only thing missing from this PR is updates to the documentation. Ah, yes, thanks for this reminder! I've repushed with doc bits added.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option, https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655286864,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655286864,MDEyOklzc3VlQ29tbWVudDY1NTI4Njg2NA==,9599,simonw,2020-07-08T05:05:27Z,2020-07-08T05:05:36Z,OWNER,"The only thing missing from this PR is updates to the documentation. Those need to go in two places: - In the Python API docs. I suggest adding a note to this section about bulk inserts: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/python-api.rst#bulk-inserts - In the CLI docs, in this section: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/cli.rst#inserting-json-data Here's an example of a previous commit that includes updates to both CLI and API documentation: https://github.com/simonw/sqlite-utils/commit/f9473ace14878212c1fa968b7bd2f51e4f064dba#diff-e3e2a9bfd88566b05001b02a3f51d286","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option, https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284168,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655284168,MDEyOklzc3VlQ29tbWVudDY1NTI4NDE2OA==,9599,simonw,2020-07-08T04:58:00Z,2020-07-08T04:58:00Z,OWNER,"Oops didn't mean to click ""close"" there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option, https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284054,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655284054,MDEyOklzc3VlQ29tbWVudDY1NTI4NDA1NA==,9599,simonw,2020-07-08T04:57:38Z,2020-07-08T04:57:38Z,OWNER,Thoughts on transactions would be much appreciated in #121 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option, https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655283393,MDEyOklzc3VlQ29tbWVudDY1NTI4MzM5Mw==,9599,simonw,2020-07-08T04:55:18Z,2020-07-08T04:55:18Z,OWNER,"This is a really good idea - and thank you for the detailed discussion in the pull request. I'm keen to discuss how transactions can work better. I tend to use this pattern in my own code: with db.conn: db[""table""].insert(...) But it's not documented and I've not though very hard about it! I like having inserts that handle 10,000+ rows commit on every chunk so I can watch their progress from another process, but the library should absolutely support people who want to commit all of the rows in a single transaction - or combine changes with DML. Lots to discuss here. I'll start a new issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option, https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655239728,MDEyOklzc3VlQ29tbWVudDY1NTIzOTcyOA==,79913,tsibley,2020-07-08T02:16:42Z,2020-07-08T02:16:42Z,CONTRIBUTOR,"I fixed my original oops by moving the `DELETE FROM $table` out of the chunking loop and repushed. I think this change can be considered in isolation from issues around transactions, which I discuss next. I wanted to make the DELETE + INSERT happen all in the same transaction so it was robust, but that was more complicated than I expected. The transaction handling in the Database/Table classes isn't systematic, and this poses big hurdles to making `Table.insert_all` (or other operations) consistent and robust in the face of errors. For example, I wanted to do this (whitespace ignored in diff, so indentation change not highlighted): ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index d6b9ecf..4107ceb 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -1028,6 +1028,11 @@ class Table(Queryable): batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns)) self.last_rowid = None self.last_pk = None + with self.db.conn: + # Explicit BEGIN is necessary because Python's sqlite3 doesn't + # issue implicit BEGINs for DDL, only DML. We mix DDL and DML + # below and might execute DDL first, e.g. for table creation. + self.db.conn.execute(""BEGIN"") if truncate and self.exists(): self.db.conn.execute(""DELETE FROM [{}];"".format(self.name)) for chunk in chunks(itertools.chain([first_record], records), batch_size): @@ -1038,7 +1043,11 @@ class Table(Queryable): # Use the first batch to derive the table names column_types = suggest_column_types(chunk) column_types.update(columns or {}) - self.create( + # Not self.create() because that is wrapped in its own + # transaction and Python's sqlite3 doesn't support + # nested transactions. + self.db.create_table( + self.name, column_types, pk, foreign_keys, @@ -1139,7 +1148,6 @@ class Table(Queryable): flat_values = list(itertools.chain(*values)) queries_and_params = [(sql, flat_values)] - with self.db.conn: for query, params in queries_and_params: try: result = self.db.conn.execute(query, params) ``` but that fails in tests because other methods call `insert/upsert/insert_all/upsert_all` in the middle of their transactions, so the BEGIN statement throws an error (no nested transactions allowed). Stepping back, it would be nice to make the transaction handling systematic and predictable. One way to do this is to make the `sqlite_utils/db.py` code generally not begin or commit any transactions, and require the caller to do that instead. This lets the caller mix and match the Python API calls into transactions as appropriate (which is impossible for the API methods themselves to fully determine). Then, make `sqlite_utils/cli.py` begin and commit a transaction in each `@cli.command` function, making each command robust and consistent in the face of errors. The big change here, and why I didn't just submit a patch, is that it dramatically changes the Python API to _require_ callers to begin a transaction rather than just immediately calling methods. There is also the caveat that for each transaction, an explicit `BEGIN` is also necessary so that DDL as well as DML (as well as `SELECT`s) are consistent and rolled back on error. There are several bugs.python.org discussions around this particular problem of DDL and some plans to make it better and consistent with DBAPI2, eventually. In the meantime, the sqlite-utils Database class could be a context manager which supports the incantations necessary to do proper transactions. This would still be a Python API change for callers but wouldn't expose them to the weirdness of the sqlite3's default transaction handling.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option, https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655052451,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655052451,MDEyOklzc3VlQ29tbWVudDY1NTA1MjQ1MQ==,79913,tsibley,2020-07-07T18:45:23Z,2020-07-07T18:45:23Z,CONTRIBUTOR,"Ah, I see the problem. The truncate is inside a loop I didn't realize was there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option, https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655018966,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655018966,MDEyOklzc3VlQ29tbWVudDY1NTAxODk2Ng==,79913,tsibley,2020-07-07T17:41:06Z,2020-07-07T17:41:06Z,CONTRIBUTOR,"Hmm, while tests pass, this may not work as intended on larger datasets. Looking into it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option, https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727814,https://api.github.com/repos/simonw/sqlite-utils/issues/99,612727814,MDEyOklzc3VlQ29tbWVudDYxMjcyNzgxNA==,9599,simonw,2020-04-13T03:05:04Z,2020-04-13T03:05:04Z,OWNER,"Bit trick from an implementation point of view this, since we want to be able to handle input that is a generator - so we can't scan through the input to validate that every dictionary has the same exact keys without consuming the entire iterator. The alternative would be to raise an error the first time we spot a dictionary with keys that differ... but that's weird because we commit changes in batches, so we may end up only applying half of the changes before exiting with the error. On that basis, I'm going to leave this as-is and mark this as wontfix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",598640234,.upsert_all() should maybe error if dictionaries passed to it do not have the same keys, https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727400,https://api.github.com/repos/simonw/sqlite-utils/issues/99,612727400,MDEyOklzc3VlQ29tbWVudDYxMjcyNzQwMA==,9599,simonw,2020-04-13T03:03:09Z,2020-04-13T03:03:09Z,OWNER,I think I'm going to leave this as intended behaviour. Or maybe passing multiple dictionaries to `.upsert_all()` with different numbers of keys should raise an error?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",598640234,.upsert_all() should maybe error if dictionaries passed to it do not have the same keys, https://github.com/simonw/sqlite-utils/issues/98#issuecomment-928790381,https://api.github.com/repos/simonw/sqlite-utils/issues/98,928790381,IC_kwDOCGYnMM43XDdt,36834097,patricktrainer,2021-09-28T04:38:44Z,2021-09-28T04:38:44Z,NONE,"Hi @simonw - wondering if you might be able to shed some light here. I've seemed to reproduce this issue. Here's the stacktrace: ``` ... db[""potholes""].insert(pothole, pk='id', alter=True, replace=True) ... Traceback (most recent call last): File """", line 3, in File ""/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2481, in insert return self.insert_all( File ""/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2596, in insert_all self.insert_chunk( File ""/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2424, in insert_chunk row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0] IndexError: list index out of range ``` Interesting enough, I found that omitting the `pk` param does not throw the error. Let me know how I can help out! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612728047,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612728047,MDEyOklzc3VlQ29tbWVudDYxMjcyODA0Nw==,9599,simonw,2020-04-13T03:06:10Z,2020-04-13T03:06:10Z,OWNER,Implementation plan: `.insert_all()` and `.upsert_all()` should only set `.last_rowid` and `last_pk` if they were called with a single item.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612708274,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612708274,MDEyOklzc3VlQ29tbWVudDYxMjcwODI3NA==,9599,simonw,2020-04-13T01:25:59Z,2020-04-13T01:26:11Z,OWNER,"In mucking around with `sqlite3` it looks like `result.lastrowid` is indeed populated for `UPDATE` - in this case with the last inserted rowid in the table. This differs from the documented behaviour I linked to above. ``` In [1]: import sqlite3 In [2]: c = sqlite3.connect("":memory:"") In [3]: c Out[3]: In [4]: c.execute('create table foo (bar integer);') Out[4]: In [5]: c.execute('insert into foo (bar) values (1)') Out[5]: In [6]: c.execute('select * from foo').fetchall() Out[6]: [(1,)] In [7]: c.execute('insert into foo (bar) values (1)') Out[7]: In [8]: c.execute('select * from foo').fetchall() Out[8]: [(1,), (1,)] In [9]: c.execute('insert into foo (bar) values (1)').lastrowid Out[9]: 3 In [10]: c.execute('select * from foo').fetchall() Out[10]: [(1,), (1,), (1,)] In [11]: c.execute('select rowid, bar from foo').fetchall() Out[11]: [(1, 1), (2, 1), (3, 1)] In [12]: c.execute('insert into foo (bar) values (1)').lastrowid Out[12]: 4 In [13]: c.execute('select rowid, bar from foo').fetchall() Out[13]: [(1, 1), (2, 1), (3, 1), (4, 1)] In [14]: r = c.execute('update foo set bar =2 where rowid = 1') In [15]: r.lastrowid Out[15]: 4 In [16]: c.execute('select rowid, bar from foo').fetchall() Out[16]: [(1, 2), (2, 1), (3, 1), (4, 1)] In [17]: r = c.execute('select rowid, bar from foo') In [18]: r.fetchall() Out[18]: [(1, 2), (2, 1), (3, 1), (4, 1)] In [19]: r.lastrowid Out[19]: 4 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707828,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612707828,MDEyOklzc3VlQ29tbWVudDYxMjcwNzgyOA==,9599,simonw,2020-04-13T01:24:05Z,2020-04-13T01:24:16Z,OWNER,"Why do I even care about `lastrowid` here? I'm trying to ensure that after you insert or upsert a row you can use `table.last_pk` to start doing things like building additional foreign key relationships. So maybe it doesn't make sense to make `.last_pk` available _at all_ for cases where you called `.upsert_all()` or `.insert_all()` - it should just be populated for `.upsert()` and `.insert()`. The documentation doesn't say it should work for `.upsert_all()` - it's only documented for the single actions. https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1113-L1124","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707293,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612707293,MDEyOklzc3VlQ29tbWVudDYxMjcwNzI5Mw==,9599,simonw,2020-04-13T01:21:22Z,2020-04-13T01:21:22Z,OWNER,"I have a hunch that the root of the problem here is that accessing `result.lastrowid` during my version of an `.upsert()` doesn't actually make sense: https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1102-L1113 In the bug I'm seeing (which I still haven't reduced to a reproducible test) the debugger shows me this at that point: ``` (Pdb) query 'UPDATE [files] SET [createdAt] = ?, [ext] = ?, [updatedAt] = ?, [uri] = ?, [uriType] = ? WHERE [project] = ? AND [name] = ?' (Pdb) params ['2020-03-04T04:04:40.152000+00:00', 'csv', '2020-03-04T04:04:40.152000+00:00', 'https://storage.googleapis.com/bln_prod/...', 'download', 'UHJvamVjdDo4MTgyMjU2Ny01ZjI0LTQxM2ItYWZmNi05NTlmNGY3MjExMjI=', 'loans_to_documentation.csv'] (Pdb) result.lastrowid 100 ``` But here's the weird thing... there's no row in the table with a rowid of 100! ``` (Pdb) [r['rowid'] for r in self.db.execute_returning_dicts('select rowid, * from files')] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99] ``` So what the heck is going on? The last SQL statement I executed here was an `UPDATE`. The `lastrowid` docs say: https://kite.com/python/docs/sqlite3.Cursor.lastrowid > This read-only attribute provides the rowid of the last modified row. It is only set if you issued a INSERT statement using the execute() method. For operations other than INSERT or when executemany() is called, lastrowid is set to None. So where did that `100` come from? It should be `None`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612258687,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612258687,MDEyOklzc3VlQ29tbWVudDYxMjI1ODY4Nw==,9599,simonw,2020-04-10T23:08:48Z,2020-04-10T23:08:48Z,OWNER,I need a test that reproduces this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612173156,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612173156,MDEyOklzc3VlQ29tbWVudDYxMjE3MzE1Ng==,9599,simonw,2020-04-10T19:03:32Z,2020-04-10T23:08:28Z,OWNER,"Investigate this traceback: ``` Traceback (most recent call last): File ""fetch_projects.py"", line 60, in fetch_projects(db, token) File ""fetch_projects.py"", line 41, in fetch_projects db[""projects""].upsert(project, pk=""id"") File ""/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1139, in upsert conversions=conversions, File ""/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1168, in upsert_all upsert=True, File ""/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1107, in insert_all row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0] IndexError: list index out of range ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/97#issuecomment-614073859,https://api.github.com/repos/simonw/sqlite-utils/issues/97,614073859,MDEyOklzc3VlQ29tbWVudDYxNDA3Mzg1OQ==,1448859,betatim,2020-04-15T14:29:30Z,2020-04-15T14:29:30Z,NONE,"Woah! Thanks a lot. Next time I will add a more obvious/explicit ""if you like this idea let me know I'd love to work on it to get my feet wet here"" :D","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",593751293,"Adding a ""recreate"" flag to the `Database` constructor", https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612738311,https://api.github.com/repos/simonw/sqlite-utils/issues/97,612738311,MDEyOklzc3VlQ29tbWVudDYxMjczODMxMQ==,9599,simonw,2020-04-13T03:55:11Z,2020-04-13T03:55:11Z,OWNER,Shipped in 2.5 - documentation is here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#connecting-to-or-creating-a-database,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",593751293,"Adding a ""recreate"" flag to the `Database` constructor", https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732453,https://api.github.com/repos/simonw/sqlite-utils/issues/97,612732453,MDEyOklzc3VlQ29tbWVudDYxMjczMjQ1Mw==,9599,simonw,2020-04-13T03:26:46Z,2020-04-13T03:26:46Z,OWNER,"I wonder if it should delete an recreate the file or if it would be safer to drop every table instead? Dropping tables gets messy: then you need to drop triggers and views, and you need to run `vacuum` to clean up the space. My worry with deleting and recreating the file is that it could trigger errors in other processes that are currently attached to that database file. But... if you know that's going to be likely, maybe you shouldn't use the `recreate=True` feature?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",593751293,"Adding a ""recreate"" flag to the `Database` constructor", https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732129,https://api.github.com/repos/simonw/sqlite-utils/issues/97,612732129,MDEyOklzc3VlQ29tbWVudDYxMjczMjEyOQ==,9599,simonw,2020-04-13T03:25:29Z,2020-04-13T03:25:29Z,OWNER,"Interesting thought. I've run into this myself a lot - many of my scripts intend to create the database from scratch, so I end up running `!rm /tmp/blah.db` in Jupyter and occasionally getting errors if the file doesn't exist. I think adding `recreate=True` could make sense. It could throw an error if you attempt to use it after passing in something other than a path to a file on disk.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",593751293,"Adding a ""recreate"" flag to the `Database` constructor", https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599247833,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599247833,MDEyOklzc3VlQ29tbWVudDU5OTI0NzgzMw==,9599,simonw,2020-03-15T18:37:28Z,2020-03-15T18:37:43Z,OWNER,Released in 2.4.2.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599128891,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599128891,MDEyOklzc3VlQ29tbWVudDU5OTEyODg5MQ==,9599,simonw,2020-03-14T20:03:45Z,2020-03-14T20:03:45Z,OWNER,I'm going to keep treating them as `str`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599127453,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599127453,MDEyOklzc3VlQ29tbWVudDU5OTEyNzQ1Mw==,9599,simonw,2020-03-14T19:50:08Z,2020-03-14T19:50:08Z,OWNER,"> If the declared type for a column contains the string ""BLOB"" or if no type is specified then the column has affinity BLOB I currently treat those as `str` - it sounds like I should treat them as `bytes`: https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L68-L69 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599127197,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599127197,MDEyOklzc3VlQ29tbWVudDU5OTEyNzE5Nw==,9599,simonw,2020-03-14T19:48:06Z,2020-03-14T19:48:06Z,OWNER,"Actually it looks like I should implement the exact rules described in https://www.sqlite.org/datatype3.html#determination_of_column_affinity > The affinity of a column is determined by the declared type of the column, according to the following rules in the order shown: > > 1. If the declared type contains the string ""INT"" then it is assigned INTEGER affinity. > 2. If the declared type of the column contains any of the strings ""CHAR"", ""CLOB"", or ""TEXT"" then that column has TEXT affinity. Notice that the type VARCHAR contains the string ""CHAR"" and is thus assigned TEXT affinity. > 3. If the declared type for a column contains the string ""BLOB"" or if no type is specified then the column has affinity BLOB. > 4. If the declared type for a column contains any of the strings ""REAL"", ""FLOA"", or ""DOUB"" then the column has REAL affinity. > 5. Otherwise, the affinity is NUMERIC. > > Note that the order of the rules for determining column affinity is important. A column whose declared type is ""CHARINT"" will match both rules 1 and 2 but the first rule takes precedence and so the column affinity will be INTEGER.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599126831,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599126831,MDEyOklzc3VlQ29tbWVudDU5OTEyNjgzMQ==,9599,simonw,2020-03-14T19:45:28Z,2020-03-14T19:45:28Z,OWNER,"Turns out there are a TON of valid column definitions that aren't being considered yet - https://www.sqlite.org/datatype3.html#affinity_name_examples - stuff like `VARYING CHARACTER(255)` and `DECIMAL(10,5)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599125557,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599125557,MDEyOklzc3VlQ29tbWVudDU5OTEyNTU1Nw==,9599,simonw,2020-03-14T19:35:29Z,2020-03-14T19:35:29Z,OWNER,"Fixing that would technically constitute a breaking change for library consumers, so it should be a major version release. I'm not inclined to release `3.0` just for this one issue, so I'm going to hold back on fixing that and address the smaller issue in this bug as a dot release instead for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599125455,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599125455,MDEyOklzc3VlQ29tbWVudDU5OTEyNTQ1NQ==,9599,simonw,2020-03-14T19:34:35Z,2020-03-14T19:34:35Z,OWNER,"From https://www.sqlite.org/datatype3.html it looks like `FLOAT` is a supported keyword for creating tables but `REAL` is the correct keyword. So actually `sqlite-utils` gets this wrong, because when we create a table we turn Python `float` values into a `FLOAT` column. Looks like the correct behaviour would be to turn them into a `REAL` column. https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L28-L48","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/91#issuecomment-723350956,https://api.github.com/repos/simonw/sqlite-utils/issues/91,723350956,MDEyOklzc3VlQ29tbWVudDcyMzM1MDk1Ng==,9599,simonw,2020-11-06T23:53:25Z,2020-11-06T23:53:25Z,OWNER,"This is now possible, for both FTS4 and FTS5 - see #197.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",577302229,Enable ordering FTS results by rank, https://github.com/simonw/sqlite-utils/issues/89#issuecomment-710460242,https://api.github.com/repos/simonw/sqlite-utils/issues/89,710460242,MDEyOklzc3VlQ29tbWVudDcxMDQ2MDI0Mg==,9599,simonw,2020-10-16T19:17:27Z,2020-10-16T19:17:50Z,OWNER,"I came up with potential syntax for that here: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550 - based on how `table.extract(...)` works: ```python fresh_db.table(""tree"", extracts=[Extract( columns=(""CommonName"", ""LatinName""), table=""Species"", fk_column=""species_id"", rename={""CommonName"": ""name"", ""LatinName"": ""latin""} )]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548,Ability to customize columns used by extracts= feature, https://github.com/simonw/sqlite-utils/issues/89#issuecomment-615515867,https://api.github.com/repos/simonw/sqlite-utils/issues/89,615515867,MDEyOklzc3VlQ29tbWVudDYxNTUxNTg2Nw==,9599,simonw,2020-04-18T00:00:41Z,2020-04-18T00:00:41Z,OWNER,"Yes pleas, I'd love to see that pull request!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548,Ability to customize columns used by extracts= feature, https://github.com/simonw/sqlite-utils/issues/89#issuecomment-593122605,https://api.github.com/repos/simonw/sqlite-utils/issues/89,593122605,MDEyOklzc3VlQ29tbWVudDU5MzEyMjYwNQ==,35075,chrishas35,2020-03-01T17:33:11Z,2020-03-01T17:33:11Z,NONE,"If you're happy with the proposed implementation, I have code & tests written that I'll get ready for a PR.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548,Ability to customize columns used by extracts= feature, https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769759,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591769759,MDEyOklzc3VlQ29tbWVudDU5MTc2OTc1OQ==,9599,simonw,2020-02-27T04:08:29Z,2020-02-27T04:08:29Z,OWNER,"I think the method should be called `table.disable_fts()` - the opposite of `table.enable_fts(...)`. There should be a `sqlite-utils disable-fts database.db tablename` command to match it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300,"table.disable_fts() method and ""sqlite-utils disable-fts ..."" command", https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769373,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591769373,MDEyOklzc3VlQ29tbWVudDU5MTc2OTM3Mw==,9599,simonw,2020-02-27T04:06:47Z,2020-02-27T04:06:47Z,OWNER,Looks like safest option is to loop through those trigger names and run `DROP TRIGGER IF EXISTS foo` on each one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300,"table.disable_fts() method and ""sqlite-utils disable-fts ..."" command", https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769171,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591769171,MDEyOklzc3VlQ29tbWVudDU5MTc2OTE3MQ==,9599,simonw,2020-02-27T04:05:58Z,2020-02-27T04:26:31Z,OWNER,"Strange - https://www.sqlite.org/lang_droptrigger.html says ""Note that triggers are automatically dropped when the associated table is dropped"" but that doesn't seem to be true in my experimenting. UPDATE: no that makes sense - the triggers are on `resources` which still exists, it was `resources_fts` that was dropped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300,"table.disable_fts() method and ""sqlite-utils disable-fts ..."" command", https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769046,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591769046,MDEyOklzc3VlQ29tbWVudDU5MTc2OTA0Ng==,9599,simonw,2020-02-27T04:05:15Z,2020-02-27T04:05:15Z,OWNER,I can reliably get the list of triggers to delete from `select name from sqlite_master where type = 'trigger' and tbl_name = 'resources';`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300,"table.disable_fts() method and ""sqlite-utils disable-fts ..."" command", https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591768604,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591768604,MDEyOklzc3VlQ29tbWVudDU5MTc2ODYwNA==,9599,simonw,2020-02-27T04:03:03Z,2020-02-27T04:03:03Z,OWNER,"`drop table resources_fts` drops the FTS table and the other ones that it created (resources_fts_data, resources_fts_idx, resources_fts_docsize, resources_fts_config) - but keeps the triggers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300,"table.disable_fts() method and ""sqlite-utils disable-fts ..."" command", https://github.com/simonw/sqlite-utils/issues/87#issuecomment-586661276,https://api.github.com/repos/simonw/sqlite-utils/issues/87,586661276,MDEyOklzc3VlQ29tbWVudDU4NjY2MTI3Ng==,9599,simonw,2020-02-16T02:20:14Z,2020-02-16T02:20:14Z,OWNER,"`OrderedDict` is actually a subclass of `dict` - so a smart fix would be for this logic to check and see if the type `t` is a subclass of one of `list`, `tuple` or `dict`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",565837965,Should detect collections.OrderedDict as a regular dictionary, https://github.com/simonw/sqlite-utils/issues/87#issuecomment-586661250,https://api.github.com/repos/simonw/sqlite-utils/issues/87,586661250,MDEyOklzc3VlQ29tbWVudDU4NjY2MTI1MA==,9599,simonw,2020-02-16T02:19:33Z,2020-02-16T02:19:33Z,OWNER,"Here's the code: https://github.com/simonw/sqlite-utils/blob/5e0000609f9be6efafea1b96f610988eb18d6d89/sqlite_utils/utils.py#L18-L24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",565837965,Should detect collections.OrderedDict as a regular dictionary, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-591770623,https://api.github.com/repos/simonw/sqlite-utils/issues/86,591770623,MDEyOklzc3VlQ29tbWVudDU5MTc3MDYyMw==,9599,simonw,2020-02-27T04:12:39Z,2020-02-27T04:12:39Z,OWNER,"I pushed a branch with my experiment in it, but I'm going to fix this by throwing an error on `[` or `]` in a column name instead - I won't implement the changes from that branch.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586729798,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586729798,MDEyOklzc3VlQ29tbWVudDU4NjcyOTc5OA==,9599,simonw,2020-02-16T17:11:02Z,2020-02-16T17:11:02Z,OWNER,I filed a bug in the Python issue tracker here: https://bugs.python.org/issue39652,"{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586683572,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586683572,MDEyOklzc3VlQ29tbWVudDU4NjY4MzU3Mg==,8149512,foscoj,2020-02-16T09:03:54Z,2020-02-16T09:03:54Z,NONE,"Probably the best option to just throw the error. Is there any active dev chan where we could post the issue to python sqlite3?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586676856,MDEyOklzc3VlQ29tbWVudDU4NjY3Njg1Ng==,9599,simonw,2020-02-16T07:20:34Z,2020-02-16T07:20:34Z,OWNER,"I'm not sure what to do about this one. I can't fix it: this bug in Python's `sqlite3` module means that even if I write a database out with column names that include `[]` I won't be able to read them back again. So... I could do one of the following: - Throw an error if a column name includes those characters. That's my preferred option I think. - Automatically replace `[` in column names with `(` and `]` with `)` - Do the automatic replacement but show a user-visible warning when I do it - Throw an error, but give the user an option to run with e.g. `--fix-column-names` which applies that automatic fix. Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676640,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586676640,MDEyOklzc3VlQ29tbWVudDU4NjY3NjY0MA==,9599,simonw,2020-02-16T07:16:31Z,2020-02-16T07:16:31Z,OWNER,"There's something weird about this. I created a test database file like so: ``` sqlite3 /tmp/demo.db < .schema CREATE TABLE IF NOT EXISTS ""data"" ( ""MTU (CET)"" TEXT, ""Day-ahead Price [EUR/MWh]"" TEXT ); sqlite> .headers on sqlite> select * from data; MTU (CET)|Day-ahead Price [EUR/MWh] 01.01.2016 00:00 - 01.01.2016 01:00|23.86 sqlite> ``` BUT... if I open the same database in Python, something weird happens: ``` In [1]: import sqlite3 In [2]: conn = sqlite3.connect(""/tmp/demo.db"") In [3]: cursor = conn.cursor() In [4]: cursor.execute(""select * from data"") Out[4]: In [5]: cursor.fetchall() Out[5]: [('01.01.2016 00:00 - 01.01.2016 01:00', '23.86')] In [6]: cursor.description Out[6]: (('MTU (CET)', None, None, None, None, None, None), ('Day-ahead Price', None, None, None, None, None, None)) In [7]: conn.row_factory = sqlite3.Row In [8]: cursor = conn.cursor() In [9]: cursor.execute(""select * from data"") Out[9]: In [10]: row = cursor.fetchall() In [12]: row Out[12]: In [15]: row.keys() Out[15]: ['MTU (CET)', 'Day-ahead Price'] ``` Note that in `cursor.description` AND in `row.keys()` above the second column is displayed as `'Day-ahead Price'` - when we would expect it to be displayed as `Day-ahead Price [EUR/MWh]` So.... it looks like there may be a bug in Python's `sqlite3` module where columns with square braces in them have that portion of the name stripped out!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586662404,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586662404,MDEyOklzc3VlQ29tbWVudDU4NjY2MjQwNA==,9599,simonw,2020-02-16T02:43:12Z,2020-02-16T02:43:12Z,OWNER,"https://stackoverflow.com/a/22694438 looks like the answer: > When using square brackets, it is not possible to have these characters in the identifier. > > When using double quotes, you can escape them in the name by doubling them: > > `CREATE TABLE ""hello """"world""""""(key INTEGER PRIMARY KEY);`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586661934,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586661934,MDEyOklzc3VlQ29tbWVudDU4NjY2MTkzNA==,9599,simonw,2020-02-16T02:33:07Z,2020-02-16T02:33:07Z,OWNER,"Thanks for the example file - looks like it can be trimmed down to just these two lines to replicate the bug: ```csv ""MTU (CET)"",""Day-ahead Price [EUR/MWh]"" ""01.01.2016 00:00 - 01.01.2016 01:00"",""23.86"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/85#issuecomment-584426938,https://api.github.com/repos/simonw/sqlite-utils/issues/85,584426938,MDEyOklzc3VlQ29tbWVudDU4NDQyNjkzOA==,9599,simonw,2020-02-11T00:35:09Z,2020-02-11T00:35:09Z,OWNER,"Here's why: https://github.com/simonw/sqlite-utils/blob/0c2451e0690c5f4e6463a2f339b0a280e30ed806/sqlite_utils/db.py#L627-L636","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",562911863,Create index doesn't work for columns containing spaces, https://github.com/simonw/sqlite-utils/issues/83#issuecomment-583789015,https://api.github.com/repos/simonw/sqlite-utils/issues/83,583789015,MDEyOklzc3VlQ29tbWVudDU4Mzc4OTAxNQ==,9599,simonw,2020-02-08T23:58:35Z,2020-02-08T23:58:35Z,OWNER,Shipped as 2.3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",559374410,"Make db[""table""].exists a documented API", https://github.com/simonw/sqlite-utils/issues/82#issuecomment-591771532,https://api.github.com/repos/simonw/sqlite-utils/issues/82,591771532,MDEyOklzc3VlQ29tbWVudDU5MTc3MTUzMg==,9599,simonw,2020-02-27T04:16:30Z,2020-02-27T04:16:30Z,OWNER,Closing as can't reproduce.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",559197745,Tutorial command no longer works, https://github.com/simonw/sqlite-utils/issues/82#issuecomment-581652388,https://api.github.com/repos/simonw/sqlite-utils/issues/82,581652388,MDEyOklzc3VlQ29tbWVudDU4MTY1MjM4OA==,9599,simonw,2020-02-03T22:35:44Z,2020-02-03T22:35:44Z,OWNER,"I can't replicate this problem: ``` /tmp $ sqlite-utils --version sqlite-utils, version 2.2 /tmp $ curl ""https://data.nasa.gov/resource/y77d-th95.json"" | sqlite-utils insert meteorites.db meteorites - --pk=id % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 240k 0 240k 0 0 185k 0 --:--:-- 0:00:01 --:--:-- 185k ``` Could you run `sqlite-utils --version` and tell me what you get?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",559197745,Tutorial command no longer works, https://github.com/simonw/sqlite-utils/issues/82#issuecomment-581651409,https://api.github.com/repos/simonw/sqlite-utils/issues/82,581651409,MDEyOklzc3VlQ29tbWVudDU4MTY1MTQwOQ==,9599,simonw,2020-02-03T22:32:41Z,2020-02-03T22:32:41Z,OWNER,This should work - the data should be chunked automatically. It looks like this is a bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",559197745,Tutorial command no longer works, https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071434,https://api.github.com/repos/simonw/sqlite-utils/issues/81,581071434,MDEyOklzc3VlQ29tbWVudDU4MTA3MTQzNA==,9599,simonw,2020-02-01T21:32:34Z,2020-02-01T21:32:34Z,OWNER,While I'm at it I think I'll rename it to `suggest_column_types` - it's not really detecting them since the input is just a list of dictionaries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",558600274,"Remove .detect_column_types() from table, make it a documented API", https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071235,https://api.github.com/repos/simonw/sqlite-utils/issues/81,581071235,MDEyOklzc3VlQ29tbWVudDU4MTA3MTIzNQ==,9599,simonw,2020-02-01T21:30:09Z,2020-02-01T21:30:09Z,OWNER,Actually I'll put it in the `utils.py` module.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",558600274,"Remove .detect_column_types() from table, make it a documented API", https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071116,https://api.github.com/repos/simonw/sqlite-utils/issues/81,581071116,MDEyOklzc3VlQ29tbWVudDU4MTA3MTExNg==,9599,simonw,2020-02-01T21:28:35Z,2020-02-01T21:28:53Z,OWNER,"Should I keep `table.detect_column_types()` working so as not to break existing code? If it was part of the documented API then I wouldn't break that without bumping to 3.x. Since it's undocumented I'm going to make it as a breaking change instead (and bump the `geojson-to-sqlite` dependency version).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",558600274,"Remove .detect_column_types() from table, make it a documented API", https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071010,https://api.github.com/repos/simonw/sqlite-utils/issues/81,581071010,MDEyOklzc3VlQ29tbWVudDU4MTA3MTAxMA==,9599,simonw,2020-02-01T21:27:00Z,2020-02-01T21:27:00Z,OWNER,"Here's the current method: https://github.com/simonw/sqlite-utils/blob/f7289174e66ae4d91d57de94bbd9d09fabf7aff4/sqlite_utils/db.py#L823-L845 If I make it a utility function instead of a class method I could ensure it is directly importable like so: ```python from sqlite_utils import detect_column_types ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",558600274,"Remove .detect_column_types() from table, make it a documented API", https://github.com/simonw/sqlite-utils/issues/8#issuecomment-482994231,https://api.github.com/repos/simonw/sqlite-utils/issues/8,482994231,MDEyOklzc3VlQ29tbWVudDQ4Mjk5NDIzMQ==,82988,psychemedia,2019-04-14T15:04:07Z,2019-04-14T15:29:33Z,NONE," PLEASE IGNORE THE BELOW... I did a package update and rebuilt the kernel I was working in... may just have been an old version of sqlite_utils, seems to be working now. (Too many containers / too many environments!) Has an issue been reintroduced here with FTS? eg I'm getting an error thrown by spaces in column names here: ``` /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) def enable_fts(self, columns, fts_version=""FTS5""): --> 329 ""Enables FTS on the specified columns"" 330 sql = """""" 331 CREATE VIRTUAL TABLE ""{table}_fts"" USING {fts_version} ( ``` when trying an `insert_all`. Also, if a col has a `.` in it, I seem to get: ``` /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) 327 jsonify_if_needed(record.get(key, None)) for key in all_columns 328 ) --> 329 result = self.db.conn.execute(sql, values) 330 self.db.conn.commit() 331 self.last_id = result.lastrowid OperationalError: near ""."": syntax error ``` (Can't post a worked minimal example right now; racing trying to build something against a live timing screen that will stop until next weekend in an hour or two...) PS Hmmm I did a test and they seem to work; I must be messing up s/where else... ``` import sqlite3 from sqlite_utils import Database dbname='testingDB_sqlite_utils.db' #!rm $dbname conn = sqlite3.connect(dbname, timeout=10) #Setup database tables c = conn.cursor() setup=''' CREATE TABLE IF NOT EXISTS ""test1"" ( ""NO"" INTEGER, ""NAME"" TEXT ); CREATE TABLE IF NOT EXISTS ""test2"" ( ""NO"" INTEGER, `TIME OF DAY` TEXT ); CREATE TABLE IF NOT EXISTS ""test3"" ( ""NO"" INTEGER, `AVG. SPEED (MPH)` FLOAT ); ''' c.executescript(setup) DB = Database(conn) import pandas as pd df1 = pd.DataFrame({'NO':[1,2],'NAME':['a','b']}) DB['test1'].insert_all(df1.to_dict(orient='records')) df2 = pd.DataFrame({'NO':[1,2],'TIME OF DAY':['early on','late']}) DB['test2'].insert_all(df2.to_dict(orient='records')) df3 = pd.DataFrame({'NO':[1,2],'AVG. SPEED (MPH)':['123.3','123.4']}) DB['test3'].insert_all(df3.to_dict(orient='records')) ``` all seem to work ok. I'm still getting errors in my set up though, which is not too different to the text cases?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403922644,Problems handling column names containing spaces or - , https://github.com/simonw/sqlite-utils/issues/8#issuecomment-466695500,https://api.github.com/repos/simonw/sqlite-utils/issues/8,466695500,MDEyOklzc3VlQ29tbWVudDQ2NjY5NTUwMA==,9599,simonw,2019-02-23T21:09:03Z,2019-02-23T21:09:03Z,OWNER,"Fixed in https://github.com/simonw/sqlite-utils/commit/228d595f7d10994f34e948888093c2cd290267c4 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403922644,Problems handling column names containing spaces or - , https://github.com/simonw/sqlite-utils/issues/8#issuecomment-464341721,https://api.github.com/repos/simonw/sqlite-utils/issues/8,464341721,MDEyOklzc3VlQ29tbWVudDQ2NDM0MTcyMQ==,82988,psychemedia,2019-02-16T12:08:41Z,2019-02-16T12:08:41Z,NONE,We also get an error if a column name contains a `.`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403922644,Problems handling column names containing spaces or - , https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029703503,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029703503,IC_kwDOCGYnMM49YAdP,9599,simonw,2022-02-04T06:46:32Z,2022-02-04T06:46:32Z,OWNER,Shipped in 3.23: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-23,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029683977,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029683977,IC_kwDOCGYnMM49X7sJ,9599,simonw,2022-02-04T05:58:15Z,2022-02-04T05:58:15Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#spatialite-helpers,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029317527,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029317527,IC_kwDOCGYnMM49WiOX,25778,eyeseast,2022-02-03T19:18:02Z,2022-02-03T19:18:02Z,CONTRIBUTOR,"Taking part of the conversation from #385 here. > Would sqlite-utils add-geometry-column ... be a good CLI enhancement. for example? Yes. And also `sqlite-utils create-spatial-index` would be great to have. My plan would be to add those once the Python API is settled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1013698557,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1013698557,IC_kwDOCGYnMM48a8_9,25778,eyeseast,2022-01-15T15:15:22Z,2022-01-15T15:15:22Z,CONTRIBUTOR,@simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012413729,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012413729,IC_kwDOCGYnMM48WDUh,25778,eyeseast,2022-01-13T18:50:00Z,2022-01-13T18:50:00Z,CONTRIBUTOR,"One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012253198,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012253198,IC_kwDOCGYnMM48VcIO,25778,eyeseast,2022-01-13T15:39:14Z,2022-01-13T15:39:14Z,CONTRIBUTOR,"Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (`gis.py` or something) just so it's easier to find later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012230212,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012230212,IC_kwDOCGYnMM48VWhE,25778,eyeseast,2022-01-13T15:15:13Z,2022-01-13T15:15:13Z,CONTRIBUTOR,"Some proposals I'd add to sqlite-utils: Some version of this, from [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite/blob/main/geojson_to_sqlite/utils.py#L124-L130): ```python def init_spatialite(db, lib): db.conn.enable_load_extension(True) db.conn.load_extension(lib) # Initialize SpatiaLite if not yet initialized if ""spatial_ref_sys"" in db.table_names(): return db.conn.execute(""select InitSpatialMetadata(1)"") ``` Also a function for creating a spatial index: ```python db.conn.execute(""select CreateSpatialIndex(?, ?)"", [table, ""geometry""]) ``` I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like: ```sh sqlite-utils spatial-index spatial.db table-name column-name ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012158895,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012158895,IC_kwDOCGYnMM48VFGv,25778,eyeseast,2022-01-13T13:55:59Z,2022-01-13T13:55:59Z,CONTRIBUTOR,"Came here to add this. I might pick it up. Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite, https://github.com/simonw/sqlite-utils/issues/77#issuecomment-580527238,https://api.github.com/repos/simonw/sqlite-utils/issues/77,580527238,MDEyOklzc3VlQ29tbWVudDU4MDUyNzIzOA==,9599,simonw,2020-01-31T00:34:02Z,2020-01-31T00:34:02Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#python-api-conversions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557825032,Ability to insert data that is transformed by a SQL function, https://github.com/simonw/sqlite-utils/issues/77#issuecomment-580515506,https://api.github.com/repos/simonw/sqlite-utils/issues/77,580515506,MDEyOklzc3VlQ29tbWVudDU4MDUxNTUwNg==,9599,simonw,2020-01-30T23:48:41Z,2020-01-30T23:48:41Z,OWNER,"Potential design: a `conversions={}` option. Used like this: ```python db[table].insert(record, conversions={""geom"": ""GeomFromText(?, 4326)""}) ``` The `conversions=` key would be supported on `.insert()`, `.insert_all()`, `.upsert()` etc. It could also be passed to the `db.table()` constructor function: ```python table = db.table( ""features"", pk=""id"", conversions={ ""geom"": ""GeomFromText(?, 4326)"" } ) # Then used like this: table.insert(record) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557825032,Ability to insert data that is transformed by a SQL function, https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614440032,https://api.github.com/repos/simonw/sqlite-utils/issues/76,614440032,MDEyOklzc3VlQ29tbWVudDYxNDQ0MDAzMg==,10501166,metab0t,2020-04-16T06:23:29Z,2020-04-16T06:23:29Z,NONE,Thanks for your hard work!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",549287310,order_by mechanism, https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614400533,https://api.github.com/repos/simonw/sqlite-utils/issues/76,614400533,MDEyOklzc3VlQ29tbWVudDYxNDQwMDUzMw==,9599,simonw,2020-04-16T03:51:26Z,2020-04-16T03:51:26Z,OWNER,Documentation here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#listing-rows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",549287310,order_by mechanism, https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614400454,https://api.github.com/repos/simonw/sqlite-utils/issues/76,614400454,MDEyOklzc3VlQ29tbWVudDYxNDQwMDQ1NA==,9599,simonw,2020-04-16T03:51:01Z,2020-04-16T03:51:01Z,OWNER,Released in 2.6,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",549287310,order_by mechanism, https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614354219,https://api.github.com/repos/simonw/sqlite-utils/issues/76,614354219,MDEyOklzc3VlQ29tbWVudDYxNDM1NDIxOQ==,9599,simonw,2020-04-16T01:01:34Z,2020-04-16T01:01:34Z,OWNER,"I think a neat way to do this would be with an optional argument for `.rows_where()`: ```python rows = db[""table""].rows_where(""age > 10"", order_by=""age desc"") ``` If you want everything you can use this: ```python rows = db[""table""].rows_where(order_by=""age desc"") ``` It's a tiny bit weird calling `.rows_where()` without a where clause, but I think it makes sense here - especially since `.rows` is a property that can't take any arguments - though under the hood it actually does this: https://github.com/simonw/sqlite-utils/blob/ad6ac19470a67867b96cb4c086450b8e4e46bf02/sqlite_utils/db.py#L436-L443","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",549287310,order_by mechanism, https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573389669,https://api.github.com/repos/simonw/sqlite-utils/issues/74,573389669,MDEyOklzc3VlQ29tbWVudDU3MzM4OTY2OQ==,15092,jayvdb,2020-01-12T07:21:17Z,2020-01-12T07:21:17Z,CONTRIBUTOR,"I guess there is some extra flag for ` CliRunner.invoke` to check exitcode and raise the exception, or that should be an extra assert added.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546073980,Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column, https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573388052,https://api.github.com/repos/simonw/sqlite-utils/issues/74,573388052,MDEyOklzc3VlQ29tbWVudDU3MzM4ODA1Mg==,15092,jayvdb,2020-01-12T06:51:30Z,2020-01-12T06:51:30Z,CONTRIBUTOR,"Thanks. That showed me that there was a click cli runner error, and setting `export LANG=en_US.UTF-8` fixed it. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546073980,Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column, https://github.com/simonw/sqlite-utils/issues/74#issuecomment-572871797,https://api.github.com/repos/simonw/sqlite-utils/issues/74,572871797,MDEyOklzc3VlQ29tbWVudDU3Mjg3MTc5Nw==,9599,simonw,2020-01-10T04:47:55Z,2020-01-10T04:47:55Z,OWNER,"This is odd. I'd love to see more about that result object. Could you try running `pytest --pdb` and then `result.exit_code, result.exception` in the PDB prompt, something like this? ``` $ pytest --pdb ========================================================= test session starts ========================================================= platform darwin -- Python 3.7.4, pytest-5.2.2, py-1.8.0, pluggy-0.13.0 rootdir: /Users/simonw/Dropbox/Development/sqlite-utils plugins: cov-2.8.1 collected 216 items tests/test_black.py s [ 0%] tests/test_cli.py F >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> db_path = '/private/var/folders/bl/5x847xbj2yb7xmp7f2tz7l280000gn/T/pytest-of-simonw/pytest-3/test_tables0/test.db' def test_tables(db_path): result = CliRunner().invoke(cli.cli, [""tables1"", db_path]) > assert '[{""table"": ""Gosh""},\n {""table"": ""Gosh2""}]' == result.output.strip() E assert '[{""table"": ""...e"": ""Gosh2""}]' == '' E - [{""table"": ""Gosh""}, E - {""table"": ""Gosh2""}] tests/test_cli.py:28: AssertionError >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > /Users/simonw/Dropbox/Development/sqlite-utils/tests/test_cli.py(28)test_tables() -> assert '[{""table"": ""Gosh""},\n {""table"": ""Gosh2""}]' == result.output.strip() (Pdb) result.exit_code, result.exception (1, OperationalError('near ""/"": syntax error')) ``` That should show the exception that caused the script to fail to run.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546073980,Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-580745213,https://api.github.com/repos/simonw/sqlite-utils/issues/73,580745213,MDEyOklzc3VlQ29tbWVudDU4MDc0NTIxMw==,82988,psychemedia,2020-01-31T14:02:38Z,2020-01-31T14:21:09Z,NONE,"So the conundrum continues.. The simple test case above now runs, but if I upsert a large number of new records (successfully) and then try to upsert a fewer number of new records to a different table, I get the same error. If I run the same upserts again (which in the first case means there are no new records to add, because they were already added), the second upsert works correctly. It feels as if the number of items added via an upsert >> the number of items I try to add in an upsert immediately after, I get the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916,upsert_all() throws issue when upserting to empty table, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-573047321,https://api.github.com/repos/simonw/sqlite-utils/issues/73,573047321,MDEyOklzc3VlQ29tbWVudDU3MzA0NzMyMQ==,82988,psychemedia,2020-01-10T14:02:56Z,2020-01-10T14:09:23Z,NONE,"Hmmm... just tried with installs from pip and the repo (v2.0.0 and v2.0.1) and I get the error each time (start of second run through the second loop). Could it be sqlite3? I'm on 3.30.1. UPDATE: just tried it on jupyter.org/try and I get the error there, too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916,upsert_all() throws issue when upserting to empty table, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-572870032,https://api.github.com/repos/simonw/sqlite-utils/issues/73,572870032,MDEyOklzc3VlQ29tbWVudDU3Mjg3MDAzMg==,9599,simonw,2020-01-10T04:38:41Z,2020-01-10T04:38:41Z,OWNER,"Odd.. I'm not able to replicate that error. Here's what I got: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916,upsert_all() throws issue when upserting to empty table, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-571138093,https://api.github.com/repos/simonw/sqlite-utils/issues/73,571138093,MDEyOklzc3VlQ29tbWVudDU3MTEzODA5Mw==,82988,psychemedia,2020-01-06T13:28:31Z,2020-01-06T13:28:31Z,NONE,"I think I actually had several issues in play... The missing key was one, but I think there is also an issue as per below. For example, in the following: ```python def init_testdb(dbname='test.db'): if os.path.exists(dbname): os.remove(dbname) conn = sqlite3.connect(dbname) db = Database(conn) return conn, db conn, db = init_testdb() c = conn.cursor() c.executescript('CREATE TABLE ""test1"" (""Col1"" TEXT, ""Col2"" TEXT, PRIMARY KEY (""Col1""));') c.executescript('CREATE TABLE ""test2"" (""Col1"" TEXT, ""Col2"" TEXT, PRIMARY KEY (""Col1""));') print('Test 1...') for i in range(3): db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) print('Test 2...') for i in range(3): db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}, {'Col1':'c','Col2':'x'}], pk=('Col1')) print('Done...') --------------------------------------------------------------------------- Test 1... Test 2... IndexError: list index out of range --------------------------------------------------------------------------- IndexError Traceback (most recent call last) in 22 print('Test 2...') 23 for i in range(3): ---> 24 db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) 25 db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}, 26 {'Col1':'c','Col2':'x'}], pk=('Col1')) /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts) 1157 alter=alter, 1158 extracts=extracts, -> 1159 upsert=True, 1160 ) 1161 /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert) 1097 # self.last_rowid will be 0 if a ""INSERT OR IGNORE"" happened 1098 if (hash_id or pk) and self.last_rowid: -> 1099 row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0] 1100 if hash_id: 1101 self.last_pk = row[hash_id] IndexError: list index out of range ``` the first test works but the second fails. Is the length of the list of items being upserted leaking somewhere?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916,upsert_all() throws issue when upserting to empty table, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-570931650,https://api.github.com/repos/simonw/sqlite-utils/issues/73,570931650,MDEyOklzc3VlQ29tbWVudDU3MDkzMTY1MA==,9599,simonw,2020-01-05T17:34:33Z,2020-01-05T17:34:33Z,OWNER,Released as 2.0.1 https://github.com/simonw/sqlite-utils/releases/tag/2.0.1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916,upsert_all() throws issue when upserting to empty table, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-570930239,https://api.github.com/repos/simonw/sqlite-utils/issues/73,570930239,MDEyOklzc3VlQ29tbWVudDU3MDkzMDIzOQ==,9599,simonw,2020-01-05T17:15:18Z,2020-01-05T17:15:18Z,OWNER,I think this is because you forgot to include a `pk=` argument. I'll change the code to throw a more useful error in this case.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916,upsert_all() throws issue when upserting to empty table, https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569234571,https://api.github.com/repos/simonw/sqlite-utils/issues/71,569234571,MDEyOklzc3VlQ29tbWVudDU2OTIzNDU3MQ==,9599,simonw,2019-12-27T09:48:48Z,2019-12-27T09:48:48Z,OWNER,That fixed it: https://travis-ci.com/simonw/sqlite-utils/builds/142443259,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",542814756,Tests are failing due to missing FTS5, https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569234096,https://api.github.com/repos/simonw/sqlite-utils/issues/71,569234096,MDEyOklzc3VlQ29tbWVudDU2OTIzNDA5Ng==,9599,simonw,2019-12-27T09:45:52Z,2019-12-27T09:45:52Z,OWNER,I'll try `bionic`: https://docs.travis-ci.com/user/reference/bionic/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",542814756,Tests are failing due to missing FTS5, https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569233996,https://api.github.com/repos/simonw/sqlite-utils/issues/71,569233996,MDEyOklzc3VlQ29tbWVudDU2OTIzMzk5Ng==,9599,simonw,2019-12-27T09:45:17Z,2019-12-27T09:45:17Z,OWNER,"It looks like those backports no longer include sqlite3 - Google Searches still find it but when you click through to launchpad you get 404s: https://launchpad.net/~jonathonf/+archive/ubuntu/codelite/+build/10511920 Maybe Travis have a newer Ubuntu I can use that ships with FTS5 in its SQLite?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",542814756,Tests are failing due to missing FTS5, https://github.com/simonw/sqlite-utils/issues/70#issuecomment-575799104,https://api.github.com/repos/simonw/sqlite-utils/issues/70,575799104,MDEyOklzc3VlQ29tbWVudDU3NTc5OTEwNA==,26292069,LucasElArruda,2020-01-17T21:20:17Z,2020-01-17T21:20:17Z,NONE,"Omg sorry I took so long to reply! On SQL we can say how the foreign key behaves when it is deleted or updated on the parent table (see https://www.sqlitetutorial.net/sqlite-foreign-key/ for more details). I did not see clearly how to create tables with this feature on sqlite-utils library.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",539204432,Implement ON DELETE and ON UPDATE actions for foreign keys, https://github.com/simonw/sqlite-utils/issues/70#issuecomment-569130037,https://api.github.com/repos/simonw/sqlite-utils/issues/70,569130037,MDEyOklzc3VlQ29tbWVudDU2OTEzMDAzNw==,9599,simonw,2019-12-26T20:39:04Z,2019-12-26T20:39:04Z,OWNER,"I hadn't thought about those at all. Are you suggesting a utility mechanism in the library for setting it up so that, for a specific foreign key, rows are deleted from other tables if the row they are pointing at is deleted?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",539204432,Implement ON DELETE and ON UPDATE actions for foreign keys, https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011906,https://api.github.com/repos/simonw/sqlite-utils/issues/7,458011906,MDEyOklzc3VlQ29tbWVudDQ1ODAxMTkwNg==,9599,simonw,2019-01-28T06:25:55Z,2019-01-28T06:25:55Z,OWNER,"I tested this with a script called `churn_em_out.py` ``` i = 0 while True: i += 1 print( '{""id"": I, ""another"": ""row"", ""number"": J}'.replace(""I"", str(i)).replace( ""J"", str(i + 1) ) ) ``` Then I ran this: ``` python churn_em_out.py | \ sqlite-utils insert /tmp/getbig.db stats - \ --nl --batch-size=10000 ``` And used `watch 'ls -lah /tmp/getbig.db'` to watch the file growing as it had 10,000 lines of junk committed in batches. The memory used by the process never grew about around 50MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403625674,.insert_all() should accept a generator and process it efficiently, https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011885,https://api.github.com/repos/simonw/sqlite-utils/issues/7,458011885,MDEyOklzc3VlQ29tbWVudDQ1ODAxMTg4NQ==,9599,simonw,2019-01-28T06:25:48Z,2019-01-28T06:25:48Z,OWNER,Re-opening for the second bit involving the cli tool.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403625674,.insert_all() should accept a generator and process it efficiently, https://github.com/simonw/sqlite-utils/issues/7#issuecomment-457980966,https://api.github.com/repos/simonw/sqlite-utils/issues/7,457980966,MDEyOklzc3VlQ29tbWVudDQ1Nzk4MDk2Ng==,9599,simonw,2019-01-28T02:29:32Z,2019-01-28T02:29:32Z,OWNER,"Remember to remove this TODO (and turn the `[]` into `()` on this line) as part of this task: https://github.com/simonw/sqlite-utils/blob/5309c5c7755818323a0f5353bad0de98ecc866be/sqlite_utils/cli.py#L78-L80","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403625674,.insert_all() should accept a generator and process it efficiently, https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710768396,https://api.github.com/repos/simonw/sqlite-utils/issues/69,710768396,MDEyOklzc3VlQ29tbWVudDcxMDc2ODM5Ng==,30607,aborruso,2020-10-17T07:46:59Z,2020-10-17T07:46:59Z,NONE,Great @simonw thank you very much,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534507142,Feature request: enable extensions loading, https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710405658,https://api.github.com/repos/simonw/sqlite-utils/issues/69,710405658,MDEyOklzc3VlQ29tbWVudDcxMDQwNTY1OA==,9599,simonw,2020-10-16T18:42:48Z,2020-10-16T18:42:48Z,OWNER,"Did some work on this for #134 in 7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d I still need to add `--load-extension` to other CLI methods, see #137. Closing this issue in favour of that one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534507142,Feature request: enable extensions loading, https://github.com/simonw/sqlite-utils/issues/69#issuecomment-569130196,https://api.github.com/repos/simonw/sqlite-utils/issues/69,569130196,MDEyOklzc3VlQ29tbWVudDU2OTEzMDE5Ng==,9599,simonw,2019-12-26T20:40:21Z,2019-12-26T20:40:21Z,OWNER,"This is a good idea. Datasette has this in the form of the `--load-extension` CLI argument, e.g. for SpatiaLite here: https://datasette.readthedocs.io/en/stable/spatialite.html#installation Having that available for `sqlite-utils` definitely makes sense.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",534507142,Feature request: enable extensions loading, https://github.com/simonw/sqlite-utils/issues/68#issuecomment-695695776,https://api.github.com/repos/simonw/sqlite-utils/issues/68,695695776,MDEyOklzc3VlQ29tbWVudDY5NTY5NTc3Ng==,9599,simonw,2020-09-20T04:25:47Z,2020-09-20T04:25:47Z,OWNER,This is a dupe of #130 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531583658,Add support for porter stemming in FTS, https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569844426,https://api.github.com/repos/simonw/sqlite-utils/issues/66,569844426,MDEyOklzc3VlQ29tbWVudDU2OTg0NDQyNg==,9599,simonw,2019-12-31T01:30:20Z,2019-12-31T01:30:20Z,OWNER,"I shipped 2.0 - release notes here: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v2 I also wrote about it on my blog: https://simonwillison.net/2019/Dec/30/sqlite-utils-2/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569588216,https://api.github.com/repos/simonw/sqlite-utils/issues/66,569588216,MDEyOklzc3VlQ29tbWVudDU2OTU4ODIxNg==,9599,simonw,2019-12-30T05:31:45Z,2019-12-30T05:31:45Z,OWNER,Last step: update changelog and ship 2.0. Then I can close this issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569226620,https://api.github.com/repos/simonw/sqlite-utils/issues/66,569226620,MDEyOklzc3VlQ29tbWVudDU2OTIyNjYyMA==,9599,simonw,2019-12-27T09:05:29Z,2019-12-27T09:05:36Z,OWNER,"I'm going to start by ignoring the existing `upsert` entirely and implementing `.insert(..., replace=True)` and `$ sqlite-utils insert --replace`. Including updating the tests. Then I'll figure out how to implement the new `.upsert()` / `$ sqlite-utils upsert`. Then I'll update the documentation, and ship `sqlite-utils` 2.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569131397,https://api.github.com/repos/simonw/sqlite-utils/issues/66,569131397,MDEyOklzc3VlQ29tbWVudDU2OTEzMTM5Nw==,9599,simonw,2019-12-26T20:49:11Z,2019-12-26T20:49:11Z,OWNER,Don't forget to update the documentation. This will be quite an involved task.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-555690319,https://api.github.com/repos/simonw/sqlite-utils/issues/66,555690319,MDEyOklzc3VlQ29tbWVudDU1NTY5MDMxOQ==,9599,simonw,2019-11-19T20:10:17Z,2019-11-19T20:10:17Z,OWNER,"Thinking about this further: I believe every time I've personally used `upsert` in the past (either with the Python library or the CLI tool) I've actually wanted the new behaviour, where ""upsert"" means ""update existing record with these changes, or insert a new record if one does not exist"". So I'm happy with `upsert` doing that, and `insert --replace` being added as an option that does what `upsert` does ta the moment. I'll still ship it as version 2.0 since it's technically a breaking change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-554565198,https://api.github.com/repos/simonw/sqlite-utils/issues/66,554565198,MDEyOklzc3VlQ29tbWVudDU1NDU2NTE5OA==,9599,simonw,2019-11-15T23:12:28Z,2019-11-15T23:12:28Z,OWNER,"Urgh this is going to be quite a bit of work, especially in the CLI module which shares an implementation for `upsert` and `insert` in a way that looks like it will have to be unwrapped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553574011,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553574011,MDEyOklzc3VlQ29tbWVudDU1MzU3NDAxMQ==,9599,simonw,2019-11-13T19:53:45Z,2019-11-13T19:53:45Z,OWNER,"First step: add a `replace=True` argument to `insert()` and `insert_all()` that does the same thing as the current `upsert=True` https://github.com/simonw/sqlite-utils/blob/8dab9fd1ccf571e188eec9ccf606a0c50fccf200/sqlite_utils/db.py#L938-L946","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553540146,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553540146,MDEyOklzc3VlQ29tbWVudDU1MzU0MDE0Ng==,9599,simonw,2019-11-13T18:33:30Z,2019-11-13T18:33:30Z,OWNER,"Maybe instead of inventing a new term I should tell people to use `.insert(..., replace=True)` directly. That matches `ignore=True`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553528850,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553528850,MDEyOklzc3VlQ29tbWVudDU1MzUyODg1MA==,9599,simonw,2019-11-13T18:04:20Z,2019-11-13T18:04:20Z,OWNER,This is going to affect the design of the CLI subcommands as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553528386,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553528386,MDEyOklzc3VlQ29tbWVudDU1MzUyODM4Ng==,9599,simonw,2019-11-13T18:03:10Z,2019-11-13T18:03:54Z,OWNER,"Maybe `inplace()` (combining ""insert"" and ""replace"")? It could be an alias for `.insert(..., replace=True)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553527384,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553527384,MDEyOklzc3VlQ29tbWVudDU1MzUyNzM4NA==,9599,simonw,2019-11-13T18:00:41Z,2019-11-13T18:00:41Z,OWNER,Is `replace()` a good name here? It doesn't really convey the idea that a brand new record will be created if there isn't an existing one to replace.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553526685,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553526685,MDEyOklzc3VlQ29tbWVudDU1MzUyNjY4NQ==,9599,simonw,2019-11-13T17:58:59Z,2019-11-13T17:58:59Z,OWNER,"This warrants making a backwards compatible change, which means I'll need to bump the major version number and release 2.0. I'm going to rename the existing `upsert()` and `upsert_all()` methods to `replace()` and `replace_all()` - then write new `upsert()` and `upsert_all()` methods that implement the correct behavior.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553171414,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553171414,MDEyOklzc3VlQ29tbWVudDU1MzE3MTQxNA==,9599,simonw,2019-11-12T23:52:35Z,2019-11-12T23:52:35Z,OWNER,"If I do implement the correct definition of `.upsert()` I think I'll use this pattern, since it works in versions of SQLite prior to 3.24: ```sql INSERT OR IGNORE INTO book(id) VALUES(1001); UPDATE book SET name = 'Programming' WHERE id = 1001; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553171011,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553171011,MDEyOklzc3VlQ29tbWVudDU1MzE3MTAxMQ==,9599,simonw,2019-11-12T23:50:52Z,2019-11-12T23:50:52Z,OWNER,"Fixing this is going to be a real pain. There's lots of code out there that uses `sqlite-utils` with the expectation that `upsert()` behaves as it currently does. Maybe I need to introduce new terms for both of these different patterns and deprecate the existing `.upsert()` and `.upsert_all()` since their behaviour can't be changed? Or maybe I fix this and ship `sqlite-utils 2.0` with a breaking change?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553170650,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553170650,MDEyOklzc3VlQ29tbWVudDU1MzE3MDY1MA==,9599,simonw,2019-11-12T23:49:29Z,2019-11-12T23:49:29Z,OWNER,This relates to this bug: https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549233778,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/63#issuecomment-549430429,https://api.github.com/repos/simonw/sqlite-utils/issues/63,549430429,MDEyOklzc3VlQ29tbWVudDU0OTQzMDQyOQ==,9599,simonw,2019-11-04T16:20:35Z,2019-11-04T16:20:35Z,OWNER,"I don't think we need this. We already have a `dogs.create_index([""name""], if_not_exists=True)` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517241040,ensure_index() method, https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549435364,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549435364,MDEyOklzc3VlQ29tbWVudDU0OTQzNTM2NA==,9599,simonw,2019-11-04T16:30:34Z,2019-11-04T16:30:34Z,OWNER,Released as 1.12.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,[enhancement] Method to delete a row in python, https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549429512,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549429512,MDEyOklzc3VlQ29tbWVudDU0OTQyOTUxMg==,9599,simonw,2019-11-04T16:18:48Z,2019-11-04T16:18:48Z,OWNER,Documentation for `.delete_where()`: https://github.com/simonw/sqlite-utils/blob/169ea455fc1f1d5e5b6e44cb339ba7ffa9d49c31/docs/python-api.rst#deleting-multiple-records,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,[enhancement] Method to delete a row in python, https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549425364,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549425364,MDEyOklzc3VlQ29tbWVudDU0OTQyNTM2NA==,9599,simonw,2019-11-04T16:09:56Z,2019-11-04T16:09:56Z,OWNER,"Since we have `table.rows_where(where, where_args)` it makes sense to me to also support `table.delete_where(where, where_args)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,[enhancement] Method to delete a row in python, https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549425012,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549425012,MDEyOklzc3VlQ29tbWVudDU0OTQyNTAxMg==,9599,simonw,2019-11-04T16:09:08Z,2019-11-04T16:09:08Z,OWNER,Documentation for `table.delete()`: https://github.com/simonw/sqlite-utils/blob/19073d6d972fad9d68dd74c28544cd29083f1c12/docs/python-api.rst#deleting-a-specific-record,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,[enhancement] Method to delete a row in python, https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549418134,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549418134,MDEyOklzc3VlQ29tbWVudDU0OTQxODEzNA==,9599,simonw,2019-11-04T15:54:18Z,2019-11-04T15:54:18Z,OWNER,"This is a good idea. The individual row version of it can work like the `.update(pk, ...)` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,[enhancement] Method to delete a row in python, https://github.com/simonw/sqlite-utils/issues/61#issuecomment-549432592,https://api.github.com/repos/simonw/sqlite-utils/issues/61,549432592,MDEyOklzc3VlQ29tbWVudDU0OTQzMjU5Mg==,9599,simonw,2019-11-04T16:25:01Z,2019-11-04T16:25:01Z,OWNER,"Yeah I've thought about this a bit and I'm OK leaving it out. The core idea of `sqlite-utils` is that if you can create a list, iterator or generator of Python dictionaries you can efficiently insert those into a SQLite table. The `--csv` function is actually implemented as just a few lines of code which turn that incoming CSV into a generator of dictionaries: https://github.com/simonw/sqlite-utils/blob/169ea455fc1f1d5e5b6e44cb339ba7ffa9d49c31/sqlite_utils/cli.py#L364-L368 I could turn this into a reusable helper function but since it wouldn't have anything to do with database inserts (it would just be a helper that turns a CSV into a generator of dictionaries) it doesn't feel like it fits well in the Python library section of this package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",491219910,importing CSV to SQLite as library, https://github.com/simonw/sqlite-utils/issues/61#issuecomment-533818697,https://api.github.com/repos/simonw/sqlite-utils/issues/61,533818697,MDEyOklzc3VlQ29tbWVudDUzMzgxODY5Nw==,49260,amjith,2019-09-21T18:09:01Z,2019-09-21T18:09:28Z,CONTRIBUTOR,"@witeshadow The library version doesn't have helpers around CSV (at least not from what I can see in the code). But here's a snippet that makes it easy to insert from CSV using the library. ``` import csv from sqlite_utils import Database # CSV Reader csv_file = open(""filename.csv"") # open the csv file. reader = csv.reader(csv_file) # Create a CSV reader headers = next(reader) # First line is the header docs = (dict(zip(headers, row)) for row in reader) # Now you can use the `sqlite_utils` library. db = Database(""my_database.db"") db[""table_name""].insert_all(docs) ``` This snippet is adapted from reading the CLI source code on how it implements the csv option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",491219910,importing CSV to SQLite as library, https://github.com/simonw/sqlite-utils/issues/6#issuecomment-457978729,https://api.github.com/repos/simonw/sqlite-utils/issues/6,457978729,MDEyOklzc3VlQ29tbWVudDQ1Nzk3ODcyOQ==,9599,simonw,2019-01-28T02:12:19Z,2019-01-28T02:12:19Z,OWNER,Will need to solve #7 for this to become truly efficient.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403624090,"""sqlite-utils insert"" should support newline-delimited JSON", https://github.com/simonw/sqlite-utils/issues/58#issuecomment-710399593,https://api.github.com/repos/simonw/sqlite-utils/issues/58,710399593,MDEyOklzc3VlQ29tbWVudDcxMDM5OTU5Mw==,9599,simonw,2020-10-16T18:39:31Z,2020-10-16T18:39:31Z,OWNER,I don't think this is valuable enough to justify adding to the library - especially since you can execute FTS search against views by joining to an FTS table built against an underlying table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488293926,Support enabling FTS on views, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638926655,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1638926655,IC_kwDOCGYnMM5hsAk_,9599,simonw,2023-07-17T21:42:37Z,2023-07-17T21:42:37Z,OWNER,"I really like this. I'm also interested in: - Plugins that make new custom SQL functions available - similar to this Datasette hook: https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette - Plugins that register functions that can be used as recipes for `sqlite-utils convert` https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes The upload-data-to-Datasette problem is planned to be solved by a future version of https://github.com/simonw/dclient ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638910473,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1638910473,IC_kwDOCGYnMM5hr8oJ,15178711,asg017,2023-07-17T21:27:41Z,2023-07-17T21:27:41Z,NONE,"Another use-case: I want to make a `sqlite-utils` plugin that'll help me insert data into Datasette. ```bash sqlite-utils insert-datasette \ --token $DATASETTE_API_KEY \ https://latest.datasette.io/fixtures/my-table \ 'select ...' ``` This could also be a datasette plugin (ex `datasette upload-data ...`, but you can also think of `sqlite-utils` plugins that upload to S3, a postgres table, other DBMS's, etc.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627598570,https://api.github.com/repos/simonw/sqlite-utils/issues/566,1627598570,IC_kwDOCGYnMM5hAy7q,9599,simonw,2023-07-09T04:13:34Z,2023-07-09T04:13:34Z,OWNER,On consulting https://pypi.org/project/tabulate/ it looks like most of those formats don't actually makes sense without headers - so the right thing here might be to raise an error if `--fmt` and `--no-headers` are used at the same time.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795219865,`--no-headers` doesn't work on most formats, https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627597872,https://api.github.com/repos/simonw/sqlite-utils/issues/566,1627597872,IC_kwDOCGYnMM5hAyww,9599,simonw,2023-07-09T04:09:56Z,2023-07-09T04:09:56Z,OWNER,"Thanks, looks like a bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795219865,`--no-headers` doesn't work on most formats, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1618380888,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1618380888,IC_kwDOCGYnMM5gdohY,9599,simonw,2023-07-03T14:09:11Z,2023-07-03T14:09:31Z,OWNER,"For the CLI: ```bash sqlite-utils rename-table data.db old_table_name new_table_name ``` For the Python code, should it go on Table or on Database? ```python db[""foo""].rename_table(""bar"") db.rename_table(""foo"", ""bar"") ``` I think I like the second better, it's slightly more clear. Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming utilities, https://github.com/simonw/sqlite-utils/issues/563#issuecomment-1617395444,https://api.github.com/repos/simonw/sqlite-utils/issues/563,1617395444,IC_kwDOCGYnMM5gZ370,9599,simonw,2023-07-03T05:44:43Z,2023-07-03T05:44:43Z,OWNER,Documentation at the bottom of this section: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-csv-or-tsv-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1785360409,`--empty-null` option when importing CSV, https://github.com/simonw/sqlite-utils/issues/562#issuecomment-1616782404,https://api.github.com/repos/simonw/sqlite-utils/issues/562,1616782404,IC_kwDOCGYnMM5gXiRE,9599,simonw,2023-07-02T19:24:14Z,2023-07-02T19:26:39Z,OWNER,"[Dataclasses](https://docs.python.org/3/library/dataclasses.html) were added in Python 3.7 and `sqlite-utils` was originally written for Python 3.6 - but both 3.6 and 3.7 are EOL now. The thing that makes Dataclasses particularly interesting is the potential to use type annotations with them to help specify the types of the related SQLite columns. Example for https://datasette.io/content/users ```sql CREATE TABLE [users] ( [login] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [avatar_url] TEXT, [gravatar_id] TEXT, [html_url] TEXT, [type] TEXT, [site_admin] INTEGER, [name] TEXT ); ``` And the dataclass: ```python from dataclasses import dataclass @dataclass class User: id: int login: str node_id: str avatar_url: str gravatar_id: str html_url: str type: str site_admin: int name: str ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1784794489,Explore the intersection between sqlite-utils and dataclasses, https://github.com/simonw/sqlite-utils/issues/561#issuecomment-1610040517,https://api.github.com/repos/simonw/sqlite-utils/issues/561,1610040517,IC_kwDOCGYnMM5f90TF,9599,simonw,2023-06-27T18:44:31Z,2023-06-27T18:44:38Z,OWNER,"Got this working: ```bash sqlite-utils insert /tmp/playground.db Playground_Submission_Data \ ~/Downloads/Playground_Submission_Data.csv --csv --stop-after 2000 ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1777548699,`--stop-after` option for `insert` and `upsert` commands, https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1590531892,https://api.github.com/repos/simonw/sqlite-utils/issues/557,1590531892,IC_kwDOCGYnMM5ezZc0,7908073,chapmanjacobd,2023-06-14T06:09:21Z,2023-06-14T06:09:21Z,CONTRIBUTOR,"I put together a [simple script](https://github.com/chapmanjacobd/library/blob/42129c5ebe15f9d74653c0f5ca4ed0c991d383e0/xklb/scripts/dedupe_db.py) to upsert and remove duplicate rows based on business keys. If anyone has similar problems with above this might help ``` CREATE TABLE my_table ( id INTEGER PRIMARY KEY, column1 TEXT, column2 TEXT, column3 TEXT ); INSERT INTO my_table (column1, column2, column3) VALUES ('Value 1', 'Duplicate 1', 'Duplicate A'), ('Value 2', 'Duplicate 2', 'Duplicate B'), ('Value 3', 'Duplicate 2', 'Duplicate C'), ('Value 4', 'Duplicate 3', 'Duplicate D'), ('Value 5', 'Duplicate 3', 'Duplicate E'), ('Value 6', 'Duplicate 3', 'Duplicate F'); ``` ``` library dedupe-db test.db my_table --bk column2 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740150327,Aliased ROWID option for tables created from alter=True commands, https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1577355134,https://api.github.com/repos/simonw/sqlite-utils/issues/557,1577355134,IC_kwDOCGYnMM5eBId-,7908073,chapmanjacobd,2023-06-05T19:26:26Z,2023-06-05T19:26:26Z,CONTRIBUTOR,"this isn't really actionable... I'm just being a whiny baby. I have tasted the milk of being able to use `upsert_all`, `insert_all`, etc without having to write DDL to create tables. The meat of the issue is that SQLITE doesn't make rowid stable between vacuums so it is not possible to take shortcuts","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740150327,Aliased ROWID option for tables created from alter=True commands, https://github.com/simonw/sqlite-utils/issues/556#issuecomment-1575310378,https://api.github.com/repos/simonw/sqlite-utils/issues/556,1575310378,IC_kwDOCGYnMM5d5VQq,601708,mcint,2023-06-04T01:21:15Z,2023-06-04T01:21:15Z,CONTRIBUTOR,"I've resolved my use, with the line-buffered output and while read loop for line buffered input, but I leave this here so the incremental saving or line-buffered use-case can be explicitly handled or rejected (or deferred).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740026046,Support storing incrementally piped values, https://github.com/simonw/sqlite-utils/issues/555#issuecomment-1592047502,https://api.github.com/repos/simonw/sqlite-utils/issues/555,1592047502,IC_kwDOCGYnMM5e5LeO,7908073,chapmanjacobd,2023-06-14T22:00:10Z,2023-06-14T22:01:57Z,CONTRIBUTOR,"You may want to try doing a performance comparison between this and just selecting all the ids with few constraints and then doing the filtering within python. That might seem like a lazy-programmer, inefficient way but queries with large resultsets are a different profile than what databases like SQLITE are designed for. That is not to say that SQLITE is slow or that python is always faster but when you start reading >20% of an index there is an equilibrium that is reached. Especially when adding in writing extra temp tables and stuff to memory/disk. And especially given the `NOT IN` style of query... You may also try chunking like this: ```py def chunks(lst, n) -> Generator: for i in range(0, len(lst), n): yield lst[i : i + n] SQLITE_PARAM_LIMIT = 32765 data = [] chunked = chunks(video_ids, consts.SQLITE_PARAM_LIMIT) for ids in chunked: data.expand( list( db.query( f""""""SELECT * from videos WHERE id in ("""""" + "","".join([""?""] * len(ids)) + "")"", (*ids,), ) ) ) ``` but that actually won't work with your `NOT IN` requirements. You need to query the full resultset to check any row. Since you are doing stuff with files/videos in SQLITE you might be interested in my side project: https://github.com/chapmanjacobd/library","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1733198948,Filter table by a large bunch of ids, https://github.com/simonw/sqlite-utils/issues/554#issuecomment-1557607516,https://api.github.com/repos/simonw/sqlite-utils/issues/554,1557607516,IC_kwDOCGYnMM5c1zRc,1231935,xavdid,2023-05-22T17:18:33Z,2023-05-22T17:18:33Z,NONE,"Oh and for context - this goes away if I use `.upsert` instead of `insert(..., ignore=True)`, but I don't want to update the value if it's written, just do an insert if it's new. The code is basically: ```py def save_items(table, items): db[""users""].insert(build_user(items[0]), pk=""id"",ignore=True) db[table].insert_all(items) if comments := fetch_comments(): save_items('comments', comments) if posts := fetch_posts(): save_items('posts', posts) ``` So either `comments` or `post` could create the relevant user if those items exist. In cases where they _both_ exist, I get this error. I need the `pk` because either call could create the table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1720096994,"`IndexError` when doing `.insert(..., pk='id')` after `insert_all`", https://github.com/simonw/sqlite-utils/issues/552#issuecomment-1556292204,https://api.github.com/repos/simonw/sqlite-utils/issues/552,1556292204,IC_kwDOCGYnMM5cwyJs,9599,simonw,2023-05-21T21:05:15Z,2023-05-21T21:05:15Z,OWNER,Now live at https://sqlite-utils.datasette.io/en/latest/installation.html#setting-up-shell-completion,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718612569,Document how to setup shell auto-completion, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556291915,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556291915,IC_kwDOCGYnMM5cwyFL,9599,simonw,2023-05-21T21:04:03Z,2023-05-21T21:04:03Z,OWNER,Now live at https://sqlite-utils.datasette.io/en/latest/cli.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556288270,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556288270,IC_kwDOCGYnMM5cwxMO,9599,simonw,2023-05-21T20:47:51Z,2023-05-21T20:47:51Z,OWNER,This page has all of the changes: https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556287599,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556287599,IC_kwDOCGYnMM5cwxBv,9599,simonw,2023-05-21T20:44:55Z,2023-05-21T20:44:55Z,OWNER,"Put this in a PR so I can preview it: - #553 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556265772,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556265772,IC_kwDOCGYnMM5cwrss,9599,simonw,2023-05-21T19:16:15Z,2023-05-21T19:16:15Z,OWNER,"Another option: That's using this markup: ``` Newline-delimited JSON ~~~~~~~~~~~~~~~~~~~~~~ Use ``--nl`` to get back newline-delimited JSON objects: .. code-block:: bash sqlite-utils dogs.db ""select * from dogs"" --nl .. code-block:: output {""id"": 1, ""age"": 4, ""name"": ""Cleo""} {""id"": 2, ""age"": 2, ""name"": ""Pancakes""} ``` And this extra CSS: ```css .highlight-output .highlight { border-left: 9px solid #30c94f; } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556263182,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556263182,IC_kwDOCGYnMM5cwrEO,9599,simonw,2023-05-21T19:06:48Z,2023-05-21T19:06:48Z,OWNER,"I could split them up into two blocks like this: I do miss the visual indication that one of these is the command and one is the output though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556262574,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556262574,IC_kwDOCGYnMM5cwq6u,9599,simonw,2023-05-21T19:04:59Z,2023-05-21T19:04:59Z,OWNER,"I wrote the docs like this because early examples include both the command and its output: https://sqlite-utils.datasette.io/en/stable/cli.html#returning-json ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556255309,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556255309,IC_kwDOCGYnMM5cwpJN,9599,simonw,2023-05-21T18:42:25Z,2023-05-21T18:42:25Z,OWNER,Tests passed here: https://github.com/simonw/sqlite-utils/actions/runs/5039119716,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7, https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556250236,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556250236,IC_kwDOCGYnMM5cwn58,9599,simonw,2023-05-21T18:25:26Z,2023-05-21T18:25:26Z,OWNER,"Relevant issues: - https://github.com/python/importlib_metadata/issues/406 - https://github.com/PyCQA/flake8/issues/1701 It looks to me like this is only a problem for `flake8` on Python 3.7 - 3.8 and higher work OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7, https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556249984,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556249984,IC_kwDOCGYnMM5cwn2A,9599,simonw,2023-05-21T18:24:48Z,2023-05-21T18:24:48Z,OWNER,"This is blocking: - #549","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7, https://github.com/simonw/sqlite-utils/issues/548#issuecomment-1556231832,https://api.github.com/repos/simonw/sqlite-utils/issues/548,1556231832,IC_kwDOCGYnMM5cwjaY,9599,simonw,2023-05-21T17:24:13Z,2023-05-21T17:24:13Z,OWNER,"Oh, I see why that is now: https://github.com/simonw/sqlite-utils/blob/6027f3ea6939a399aeef2578fca17efec0e539df/sqlite_utils/cli.py#L2670-L2679 This is because of the following command: sqlite-utils analyze-tables table1 table2 --column x Since you can pass multiple tables AND multiple columns, the tool currently assumes that the column(s) you specify may be available on a subset of the provided tables. I'm going to change this so if the column is not on ANY of those tables you get an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718576761,analyze-tables should validate provide --column names, https://github.com/simonw/sqlite-utils/issues/547#issuecomment-1556228395,https://api.github.com/repos/simonw/sqlite-utils/issues/547,1556228395,IC_kwDOCGYnMM5cwikr,9599,simonw,2023-05-21T17:11:15Z,2023-05-21T17:11:15Z,OWNER,"This will be a cosmetic change to the CLI output only - the options to save data to the database and the Python API function will continue to return `[(None, 158)]`. I can add an optimization though to avoid running the SQL count query if we know that it's all `null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718572201,No need to show common values if everything is null, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556269616,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556269616,IC_kwDOCGYnMM5cwsow,9599,simonw,2023-05-21T19:33:13Z,2023-05-21T19:33:13Z,OWNER,Now released: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-32,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556247818,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556247818,IC_kwDOCGYnMM5cwnUK,9599,simonw,2023-05-21T18:17:46Z,2023-05-21T18:17:46Z,OWNER,Draft documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556210844,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556210844,IC_kwDOCGYnMM5cweSc,9599,simonw,2023-05-21T15:44:10Z,2023-05-21T15:44:10Z,OWNER,"It looks like `nargs=-1` on a positional argument isn't yet supported - opened an issue here: - https://github.com/Textualize/trogon/issues/4","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556191894,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556191894,IC_kwDOCGYnMM5cwZqW,9599,simonw,2023-05-21T14:20:14Z,2023-05-21T14:20:14Z,OWNER,"Opened a feature request for customizing the help and command name: - https://github.com/Textualize/trogon/issues/2","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556190531,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556190531,IC_kwDOCGYnMM5cwZVD,9599,simonw,2023-05-21T14:13:43Z,2023-05-21T14:13:43Z,OWNER,"OK, this works! ![trogon](https://github.com/simonw/sqlite-utils/assets/9599/2ae194c5-ec82-471a-9d1b-b01b3f2632f3) To try it out, install that branch from GitHub: pip install https://github.com/simonw/sqlite-utils/archive/refs/heads/trogon.zip Then run this: sqlite-utils install trogon And this: sqlite-utils tui ","{""total_count"": 5, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 3, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556189823,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556189823,IC_kwDOCGYnMM5cwZJ_,9599,simonw,2023-05-21T14:09:59Z,2023-05-21T14:09:59Z,OWNER,"I don't want to add `trogon` as a default dependency because it's a little heavy - it pulls in all of Rich and Textual as well. People who use `sqlite-utils` just for its Python API won't benefit from this - it's a CLI feature only. But I have a `sqlite-utils install ...` command for helping people to install packages into the same virtual environment as `sqlite-utils` no matter how they installed that tool: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install So I can treat Trogon as an optional dependency and add the `sqlite-utils tui` command only if that package is also installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556225788,https://api.github.com/repos/simonw/sqlite-utils/issues/544,1556225788,IC_kwDOCGYnMM5cwh78,9599,simonw,2023-05-21T17:02:05Z,2023-05-21T17:02:05Z,OWNER,"New docs: - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-analyze-tables - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#analyze-tables - https://sqlite-utils.datasette.io/en/latest/python-api.html#analyzing-a-column - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.analyze_column New help output: ``` % sqlite-utils analyze-tables --help Usage: sqlite-utils analyze-tables [OPTIONS] PATH [TABLES]... Analyze the columns in one or more tables Example: sqlite-utils analyze-tables data.db trees Options: -c, --column TEXT Specific columns to analyze --save Save results to _analyze_tables table --common-limit INTEGER How many common values --no-most Skip most common values --no-least Skip least common values --load-extension TEXT Path to SQLite extension, with optional :entrypoint -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718515590,New options for analyze-tables --common-limit --no-most and --no-least, https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556211643,https://api.github.com/repos/simonw/sqlite-utils/issues/544,1556211643,IC_kwDOCGYnMM5cwee7,9599,simonw,2023-05-21T15:48:17Z,2023-05-21T15:48:17Z,OWNER,I generated the commit message in https://github.com/simonw/sqlite-utils/commit/1c1991b447a1ddd3d61d9d4a8a1d6a9da47ced20 using `git diff | llm --system 'describe this change'`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718515590,New options for analyze-tables --common-limit --no-most and --no-least, https://github.com/simonw/sqlite-utils/issues/542#issuecomment-1539052467,https://api.github.com/repos/simonw/sqlite-utils/issues/542,1539052467,IC_kwDOCGYnMM5bvBOz,9599,simonw,2023-05-08T21:07:41Z,2023-05-08T21:07:41Z,OWNER,"Relevant commits (will mostly revert these): - https://github.com/simonw/sqlite-utils/commit/455c35b512895c19bf922c2b804d750d27cb8dbd - https://github.com/simonw/sqlite-utils/commit/e0ec4c345129996011951e400388fd74865f65a2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1700936245,Remove `skip_false=True` and `--no-skip-false` in `sqlite-utils` 4.0, https://github.com/simonw/sqlite-utils/issues/541#issuecomment-1538963959,https://api.github.com/repos/simonw/sqlite-utils/issues/541,1538963959,IC_kwDOCGYnMM5burn3,9599,simonw,2023-05-08T19:59:34Z,2023-05-08T19:59:34Z,OWNER,"There are 8 failing tests left: ``` ==== short test summary info ==== FAILED tests/test_cli_memory.py::test_memory_csv[False-test] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_csv[False-t] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_csv[False-t1] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_tsv[False] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_dump[extra_args0] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_two_files_with_same_stem - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedate] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: .convert_value at 0x106bcaca0> FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedatetime] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: .convert_value at 0x106bc9620> ERROR tests/test_cli.py::test_upsert_analyze - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> ==== 8 failed, 894 passed, 4 skipped, 1 error in 6.27s ==== ``` Full traceback here: https://gist.github.com/simonw/b40b3e814729d6c02a0302a84ce54d9e","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1700840265,Get tests to pass with `pytest -Werror`, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537744000,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537744000,IC_kwDOCGYnMM5bqByA,42327,pquentin,2023-05-08T04:56:12Z,2023-05-08T04:56:12Z,NONE,"Hey @simonw, urllib3 maintainer here :wave: Sorry for breaking your CI. I understand you may prefer to pin the Python version, but note that specifying just `python: ""3""` will get you the latest. We use that in urllib3: https://github.com/urllib3/urllib3/blob/main/.readthedocs.yml I can open PRs to sqlite-utils / datasette if you're interested","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537514069,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537514069,IC_kwDOCGYnMM5bpJpV,9599,simonw,2023-05-07T18:40:18Z,2023-05-07T18:40:18Z,OWNER,"https://docs.readthedocs.io/en/stable/config-file/v2.html suggests: ```yaml build: os: ubuntu-22.04 tools: python: ""3.11"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513912,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537513912,IC_kwDOCGYnMM5bpJm4,9599,simonw,2023-05-07T18:39:29Z,2023-05-07T18:39:29Z,OWNER,"https://readthedocs.org/projects/sqlite-utils/builds/20513034/ said: > Problem in your project's configuration. Invalid ""python.version"": expected one of (2, 2.7, 3, 3.5, 3.6, 3.7, 3.8, pypy3.5), got 3.11","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513653,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537513653,IC_kwDOCGYnMM5bpJi1,9599,simonw,2023-05-07T18:37:59Z,2023-05-07T18:38:19Z,OWNER,"Useful comment here: - https://github.com/urllib3/urllib3/issues/2168#issuecomment-1537360928 > I faced the same issue. I switched to a different Python kernel (3.11.2) and it worked.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/54#issuecomment-524300388,https://api.github.com/repos/simonw/sqlite-utils/issues/54,524300388,MDEyOklzc3VlQ29tbWVudDUyNDMwMDM4OA==,20264,ftrain,2019-08-23T12:41:09Z,2019-08-23T12:41:09Z,NONE,Extremely cool and easy to understand. Thank you!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",480961330,"Ability to list views, and to access db[""view_name""].rows / rows_where / etc", https://github.com/simonw/sqlite-utils/issues/54#issuecomment-524297256,https://api.github.com/repos/simonw/sqlite-utils/issues/54,524297256,MDEyOklzc3VlQ29tbWVudDUyNDI5NzI1Ng==,9599,simonw,2019-08-23T12:30:20Z,2019-08-23T12:30:20Z,OWNER,Released in 1.10: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v1-10,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",480961330,"Ability to list views, and to access db[""view_name""].rows / rows_where / etc", https://github.com/simonw/sqlite-utils/issues/54#issuecomment-524294928,https://api.github.com/repos/simonw/sqlite-utils/issues/54,524294928,MDEyOklzc3VlQ29tbWVudDUyNDI5NDkyOA==,9599,simonw,2019-08-23T12:22:11Z,2019-08-23T12:22:11Z,OWNER,Documentation is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#listing-views,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",480961330,"Ability to list views, and to access db[""view_name""].rows / rows_where / etc", https://github.com/simonw/sqlite-utils/issues/54#issuecomment-524294383,https://api.github.com/repos/simonw/sqlite-utils/issues/54,524294383,MDEyOklzc3VlQ29tbWVudDUyNDI5NDM4Mw==,9599,simonw,2019-08-23T12:20:15Z,2019-08-23T12:20:15Z,OWNER,Closed by #55,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",480961330,"Ability to list views, and to access db[""view_name""].rows / rows_where / etc", https://github.com/simonw/sqlite-utils/issues/54#issuecomment-522236053,https://api.github.com/repos/simonw/sqlite-utils/issues/54,522236053,MDEyOklzc3VlQ29tbWVudDUyMjIzNjA1Mw==,9599,simonw,2019-08-17T13:22:26Z,2019-08-17T13:22:26Z,OWNER,"Good catch: `create_view()` is an earlier utility method from before I started adding methods like `db[""table""].rows` and `.rows_where()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",480961330,"Ability to list views, and to access db[""view_name""].rows / rows_where / etc", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537514610,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537514610,IC_kwDOCGYnMM5bpJxy,9599,simonw,2023-05-07T18:43:24Z,2023-05-07T18:43:24Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#returning-raw-data-such-as-binary-content - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#memory","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507676,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507676,IC_kwDOCGYnMM5bpIFc,9599,simonw,2023-05-07T18:09:43Z,2023-05-07T18:09:54Z,OWNER,"This worked: ```bash sqlite-utils memory /tmp/books3.json:nl \ 'select name from books3' --raw-lines > titles.txt ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507525,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507525,IC_kwDOCGYnMM5bpIDF,9599,simonw,2023-05-07T18:09:09Z,2023-05-07T18:09:09Z,OWNER,"I'm tempted to upgrade `--raw` to do this instead, but that would be a breaking change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507394,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507394,IC_kwDOCGYnMM5bpIBC,9599,simonw,2023-05-07T18:08:44Z,2023-05-07T18:08:44Z,OWNER,"Prototype: ```diff diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 153e5f9..c830518 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -124,6 +124,7 @@ See :ref:`cli_query`. --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row + --raw-lines Raw output, first column of each row -p, --param ... Named :parameters for SQL query --functions TEXT Python code defining one or more custom SQL functions @@ -192,6 +193,7 @@ See :ref:`cli_memory`. --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row + --raw-lines Raw output, first column of each row -p, --param ... Named :parameters for SQL query --encoding TEXT Character encoding for CSV input, defaults to utf-8 diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index d25b1df..da0e4b6 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -1653,6 +1653,7 @@ def drop_view(path, view, ignore, load_extension): ) @output_options @click.option(""-r"", ""--raw"", is_flag=True, help=""Raw output, first column of first row"") +@click.option(""--raw-lines"", is_flag=True, help=""Raw output, first column of each row"") @click.option( ""-p"", ""--param"", @@ -1677,6 +1678,7 @@ def query( fmt, json_cols, raw, + raw_lines, param, load_extension, functions, @@ -1700,7 +1702,19 @@ def query( _register_functions(db, functions) _execute_query( - db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols + db, + sql, + param, + raw, + raw_lines, + table, + csv, + tsv, + no_headers, + fmt, + nl, + arrays, + json_cols, ) @@ -1728,6 +1742,7 @@ def query( ) @output_options @click.option(""-r"", ""--raw"", is_flag=True, help=""Raw output, first column of first row"") +@click.option(""--raw-lines"", is_flag=True, help=""Raw output, first column of each row"") @click.option( ""-p"", ""--param"", @@ -1773,6 +1788,7 @@ def memory( fmt, json_cols, raw, + raw_lines, param, encoding, no_detect_types, @@ -1879,12 +1895,36 @@ def memory( _register_functions(db, functions) _execute_query( - db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols + db, + sql, + param, + raw, + raw_lines, + table, + csv, + tsv, + no_headers, + fmt, + nl, + arrays, + json_cols, ) def _execute_query( - db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols + db, + sql, + param, + raw, + raw_lines, + table, + csv, + tsv, + no_headers, + fmt, + nl, + arrays, + json_cols, ): with db.conn: try: @@ -1903,6 +1943,13 @@ def _execute_query( sys.stdout.buffer.write(data) else: sys.stdout.write(str(data)) + elif raw_lines: + for row in cursor: + data = row[0] + if isinstance(data, bytes): + sys.stdout.buffer.write(data + b""\n"") + else: + sys.stdout.write(str(data) + ""\n"") elif fmt or table: print( tabulate.tabulate( ``` Needs tests and more documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538975545,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538975545,IC_kwDOCGYnMM5buuc5,1231935,xavdid,2023-05-08T20:06:35Z,2023-05-08T20:06:35Z,NONE,"perfect, thank you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538921774,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538921774,IC_kwDOCGYnMM5buhUu,9599,simonw,2023-05-08T19:24:41Z,2023-05-08T19:24:41Z,OWNER,That fix seems to work!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538910894,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538910894,IC_kwDOCGYnMM5buequ,9599,simonw,2023-05-08T19:16:52Z,2023-05-08T19:17:00Z,OWNER,"How about if I had logic which checked that all not-null columns were provided in the call to `upsert_all()` - and if they were, modified the `INSERT OR IGNORE INTO` to include a placeholder value for those columns that would then be fixed by the later `UPDATE`? Something like this: ```python [ ('INSERT OR IGNORE INTO [comments]([id], name) VALUES(?, ?);', [1, '']), ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1]) ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538903556,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538903556,IC_kwDOCGYnMM5buc4E,9599,simonw,2023-05-08T19:11:24Z,2023-05-08T19:13:23Z,OWNER,"I could detect if this happens using `cursor.rowcount` - not sure how I would recover from it though. This would also require some major re-engineering, since currently it all works by generating a list of SQL queries in advance and applying them inside a loop in `.insert_chunk()`: https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2839-L2878 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538893329,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538893329,IC_kwDOCGYnMM5buaYR,9599,simonw,2023-05-08T19:04:47Z,2023-05-08T19:04:47Z,OWNER,"This feels like a fundamental flaw in the way upserts are implemented by `sqlite-utils`. One fix would be to switch to using the `UPSERT` feature in SQLite: https://www.sqlite.org/lang_UPSERT.html But... > UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04). I still want to support SQLite versions earlier than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538889482,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538889482,IC_kwDOCGYnMM5buZcK,9599,simonw,2023-05-08T19:02:38Z,2023-05-08T19:02:38Z,OWNER,"Here's the code at fault: https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2774-L2788","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538887361,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538887361,IC_kwDOCGYnMM5buY7B,9599,simonw,2023-05-08T19:01:20Z,2023-05-08T19:01:20Z,OWNER,"Here's the problem: ```python import sqlite3 db = sqlite3.connect("":memory:"") db.execute('create table foo (id integer primary key, name not null)') db.execute('insert into foo (id) values (1)') ``` Produces: ``` IntegrityError: NOT NULL constraint failed: foo.name ``` But this: ```python db.execute('insert or ignore into foo (id) values (1)') ``` Completes without an exception.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538801855,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538801855,IC_kwDOCGYnMM5buEC_,9599,simonw,2023-05-08T18:00:17Z,2023-05-08T18:00:17Z,OWNER,"From time in the debugger, after creating the table it ends up doing this: ``` (Pdb) queries_and_params [ ('INSERT OR IGNORE INTO [comments]([id]) VALUES(?);', [1]), ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1]) ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538793817,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538793817,IC_kwDOCGYnMM5buCFZ,9599,simonw,2023-05-08T17:55:10Z,2023-05-08T17:55:10Z,OWNER,"Confirmed - I added this test and it fails: ```python def test_upsert_all_not_null(fresh_db): # https://github.com/simonw/sqlite-utils/issues/538 fresh_db[""comments""].upsert_all( [{""id"": 1, ""name"": ""Cleo""}], pk=""id"", not_null=[""name""], ) assert list(fresh_db[""comments""].rows) == [{""id"": 1, ""name"": ""Cleo""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592617271,https://api.github.com/repos/simonw/sqlite-utils/issues/535,1592617271,IC_kwDOCGYnMM5e7Wk3,13780613,erlend-aasland,2023-06-15T08:39:49Z,2023-06-15T08:39:49Z,NONE,"> piping to `jq` is good enough usually ... or `python -m json.tool`[^1], if you don't have `jq` installed. [^1]: no fancy colouring, like `jq`; only pretty-printing","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1655860104,rows: --transpose or psql extended view-like functionality, https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592052320,https://api.github.com/repos/simonw/sqlite-utils/issues/535,1592052320,IC_kwDOCGYnMM5e5Mpg,7908073,chapmanjacobd,2023-06-14T22:05:28Z,2023-06-14T22:05:28Z,CONTRIBUTOR,piping to `jq` is good enough usually,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1655860104,rows: --transpose or psql extended view-like functionality, https://github.com/simonw/sqlite-utils/issues/534#issuecomment-1538933540,https://api.github.com/repos/simonw/sqlite-utils/issues/534,1538933540,IC_kwDOCGYnMM5bukMk,9599,simonw,2023-05-08T19:34:37Z,2023-05-08T19:34:37Z,OWNER,"On macOS this shows the same warning: ``` % python -Wdefault $(which sqlite-utils) insert dogs.db dogs dogs.csv --csv [############------------------------] 35% [####################################] 100%/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs.csv' encoding='utf-8-sig'> insert_upsert_implementation( ResourceWarning: Enable tracemalloc to get the object allocation traceback ``` The file itself is a `click.File` which is automatically closed - https://click.palletsprojects.com/en/8.1.x/api/#click.File - but it looks like it's the `_io.TextIOWrapper` which is not being closed: https://github.com/simonw/sqlite-utils/blob/2376c452a56b0c3e75e7ca698273434e32945304/sqlite_utils/cli.py#L949-L956","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1622640374, ResourceWarning: unclosed file, https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465303378,https://api.github.com/repos/simonw/sqlite-utils/issues/533,1465303378,IC_kwDOCGYnMM5XVsFS,9599,simonw,2023-03-12T21:24:04Z,2023-03-12T21:24:04Z,OWNER,"Upgraded to Sphinx 6 locally and got the same error: ``` % just docs Cogging README.md Cogging docs/changelog.rst Cogging docs/cli-reference.rst Cogging docs/cli.rst Cogging docs/contributing.rst Cogging docs/index.rst Cogging docs/installation.rst Cogging docs/python-api.rst Cogging docs/reference.rst sphinx-autobuild -a -b html ""."" ""_build"" --watch ../sqlite_utils [sphinx-autobuild] > sphinx-build -b html -a /Users/simon/Dropbox/Development/sqlite-utils/docs /Users/simon/Dropbox/Development/sqlite-utils/docs/_build Running Sphinx v6.1.3 loading pickled environment... failed failed: Can't get attribute '_stable_repr_object' on building [mo]: all of 0 po files writing output... building [html]: all source files updating environment: [new config] 8 added, 0 changed, 0 removed reading sources... [ 12%] changelog Exception occurred: File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/sphinx/ext/extlinks.py"", line 103, in role title = caption % part ~~~~~~~~^~~~~~ TypeError: not all arguments converted during string formatting The full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1ey36c1n.log, if you want to report the issue to the developers. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620516340,ReadTheDocs error: not all arguments converted during string formatting, https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465302936,https://api.github.com/repos/simonw/sqlite-utils/issues/533,1465302936,IC_kwDOCGYnMM5XVr-Y,9599,simonw,2023-03-12T21:22:09Z,2023-03-12T21:22:09Z,OWNER,"Could be the same problem as: - https://github.com/simonw/datasette/issues/1972 Which I fixed in https://github.com/simonw/datasette/commit/3af313e165215696af899e772f47bf7c27873ae3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620516340,ReadTheDocs error: not all arguments converted during string formatting, https://github.com/simonw/sqlite-utils/issues/532#issuecomment-1539009453,https://api.github.com/repos/simonw/sqlite-utils/issues/532,1539009453,IC_kwDOCGYnMM5bu2ut,9599,simonw,2023-05-08T20:30:29Z,2023-05-08T20:30:42Z,OWNER,"Here's an improvement: ``` % sqlite-utils insert /tmp/b.db blah /tmp/blah.txt [####################################] 100% Error: Invalid JSON - use --csv for CSV or --tsv for TSV files JSON error: Expecting value: line 1 column 1 (char 0) ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620254998,Show more information when JSON can't be imported with sqlite-utils insert, https://github.com/simonw/sqlite-utils/issues/532#issuecomment-1539006509,https://api.github.com/repos/simonw/sqlite-utils/issues/532,1539006509,IC_kwDOCGYnMM5bu2At,9599,simonw,2023-05-08T20:28:56Z,2023-05-08T20:28:56Z,OWNER,Was this a newline-delimited JSON file perhaps?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620254998,Show more information when JSON can't be imported with sqlite-utils insert, https://github.com/simonw/sqlite-utils/issues/530#issuecomment-1539018912,https://api.github.com/repos/simonw/sqlite-utils/issues/530,1539018912,IC_kwDOCGYnMM5bu5Cg,9599,simonw,2023-05-08T20:39:00Z,2023-05-08T20:39:00Z,OWNER,"I think the natural place to add these in the Python library API would be https://sqlite-utils.datasette.io/en/stable/python-api.html#adding-foreign-key-constraints - maybe something like this: ```python db[""books""].add_foreign_key(""author_id"", ""authors"", ""id"", on_delete=RESTRICT) ``` Then for the CLI tool could be added to https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key ``` sqlite-utils add-foreign-key my.db books author_id authors id --on-update SET_NULL ``` I wouldn't support these for the other methods of adding foreign keys - `foreign_keys(...)` for the various `.insert()` etc methods and the `add_foreign_keys(...)` bulk menthod.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1595340692,"add ability to configure ""on delete"" and ""on update"" attributes of foreign keys:", https://github.com/simonw/sqlite-utils/issues/530#issuecomment-1539015064,https://api.github.com/repos/simonw/sqlite-utils/issues/530,1539015064,IC_kwDOCGYnMM5bu4GY,9599,simonw,2023-05-08T20:35:07Z,2023-05-08T20:35:07Z,OWNER,"Wow, this is a neat feature I didn't know about. Looks like there are a bunch of options: - NO ACTION (default) - RESTRICT: application is prohibited from deleting a parent key when there exists one or more child keys mapped to it - SET NULL: when a parent key is deleted the child key columns of all rows in the child table that mapped to the parent key are set to contain SQL NULL values - SET DEFAULT: set a specific default - CASCADE: propagates the delete or update operation on the parent key to each dependent child key","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1595340692,"add ability to configure ""on delete"" and ""on update"" attributes of foreign keys:", https://github.com/simonw/sqlite-utils/issues/529#issuecomment-1592110694,https://api.github.com/repos/simonw/sqlite-utils/issues/529,1592110694,IC_kwDOCGYnMM5e5a5m,7908073,chapmanjacobd,2023-06-14T23:11:47Z,2023-06-14T23:12:12Z,CONTRIBUTOR,"sorry i was wrong. `sqlite-utils --raw-lines` works correctly ``` sqlite-utils --raw-lines :memory: ""SELECT * FROM (VALUES ('test'), ('line2'))"" | cat -A test$ line2$ sqlite-utils --csv --no-headers :memory: ""SELECT * FROM (VALUES ('test'), ('line2'))"" | cat -A test$ line2$ ``` I think this was fixed somewhat recently","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1581090327,Microsoft line endings, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1540900733,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1540900733,IC_kwDOCGYnMM5b2Ed9,167893,mcarpenter,2023-05-09T21:15:05Z,2023-05-09T21:15:05Z,CONTRIBUTOR,"Sorry, I completely missed your first comment whilst on Easter break. This looks like a good practical compromise before v4. Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539051724,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1539051724,IC_kwDOCGYnMM5bvBDM,9599,simonw,2023-05-08T21:07:04Z,2023-05-08T21:07:04Z,OWNER,"Updated documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-data-in-columns - https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#convert","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539035838,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1539035838,IC_kwDOCGYnMM5bu9K-,9599,simonw,2023-05-08T20:55:00Z,2023-05-08T20:55:00Z,OWNER,"I'm going to go with `--no-skip-false` as the CLI option. It's ugly, but this whole thing is ugly. I'm going to make a note to remove this misfeature in `sqlite-utils` 4.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539033736,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1539033736,IC_kwDOCGYnMM5bu8qI,9599,simonw,2023-05-08T20:52:51Z,2023-05-08T20:52:51Z,OWNER,"OK, I implemented that at the Python API level. I need to decide how it should work for the `sqlite-utils convert` command too: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#convert","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1506223848,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1506223848,IC_kwDOCGYnMM5Zxybo,9599,simonw,2023-04-13T02:08:16Z,2023-04-13T02:08:16Z,OWNER,"I agree, this is a design flaw. It's technically a breaking change. As such, I would need to bump to v4 to responsibly release this. I'd rather bundle in a few more breaking changes before shipping that version. One thing we could do here is add an extra argument to `.convert()` - something like this: ```python table.convert(col, lambda x: x+1, skip_false=False) ``` This would trigger the new, improved behaviour without breaking existing code that depends on how it works at the moment. Then in `sqlite-utils` 4 we can change the default of that option. What do you think? (I'm open to suggestions for better names for this parameter too)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1539108140,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1539108140,IC_kwDOCGYnMM5bvO0s,9599,simonw,2023-05-08T21:59:41Z,2023-05-08T21:59:41Z,OWNER,That original example passes against `main` now.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail, https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1539101853,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1539101853,IC_kwDOCGYnMM5bvNSd,9599,simonw,2023-05-08T21:52:44Z,2023-05-08T21:52:44Z,OWNER,I like the `lambda-{uuid}` idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail, https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1435318713,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1435318713,IC_kwDOCGYnMM5VjTm5,167893,mcarpenter,2023-02-17T21:55:01Z,2023-02-17T21:55:01Z,CONTRIBUTOR,"Meanwhile, a cheap workaround is to invalidate the registered function cache: ``` python table.convert(...) db._registered_functions = set() table.convert(...) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail, https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1423387341,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1423387341,IC_kwDOCGYnMM5U1yrN,167893,mcarpenter,2023-02-08T23:48:52Z,2023-02-09T00:17:30Z,CONTRIBUTOR,PR below,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1422681850,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1422681850,IC_kwDOCGYnMM5UzGb6,21095447,4l1fe,2023-02-08T14:25:50Z,2023-02-08T14:29:09Z,NONE,"I live the patch here for others: _original code_ ```shell $ which sqlite-utils | xargs cat ``` ```python #!/usr/bin/python3 # -*- coding: utf-8 -*- import re import sys from sqlite_utils.cli import cli if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) sys.exit(cli()) ``` _patched/sqlite-utils.py_ ```python #!/usr/bin/python3 # -*- coding: utf-8 -*- import re import sys from sqlite_utils.cli import cli # New imports from unittest.mock import patch from sqlite_utils.cli import VALID_COLUMN_TYPES if __name__ == '__main__': # Choices of the option `--type` cli.commands['transform'].params[2].type.types[1].choices.append('DATETIME') # The dicts has to be extended with a new type with patch.dict('sqlite_utils.db.COLUMN_TYPE_MAPPING', {'DATETIME': 'DATETIME'}),\ patch('sqlite_utils.cli.VALID_COLUMN_TYPES', VALID_COLUMN_TYPES + (""DATETIME"", )): # Command is unchanged sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) sys.exit(cli()) ``` And now it's working ```bash $ sqlite-utils schema events.sqlite cards.chunk.get CREATE TABLE ""cards.chunk.get"" ( [id] INTEGER PRIMARY KEY NOT NULL, [timestamp] TEXT, ) $ python patched/sqlite-utils.py transform events.sqlite cards.chunk.get --type timestamp DATETIME $ sqlite-utils schema events.sqlite cards.chunk.get CREATE TABLE ""cards.chunk.get"" ( [id] INTEGER PRIMARY KEY NOT NULL, [timestamp] DATETIME, ) ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421177666,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421177666,IC_kwDOCGYnMM5UtXNC,21095447,4l1fe,2023-02-07T17:39:00Z,2023-02-07T17:39:00Z,NONE,"> lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed I've got your concern. Let's see if we will be replied on it and i'll close the issue some later. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421081939,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421081939,IC_kwDOCGYnMM5Us_1T,193185,cldellow,2023-02-07T16:42:25Z,2023-02-07T16:43:42Z,NONE,"Ha, yes, I might end up making something very niche. That's OK. I'm building a UI for [Datasette](https://datasette.io/) that lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed. And so on. I had hoped that I could just lean on sqlite-utils, but I think it's a little too surprising.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421055590,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421055590,IC_kwDOCGYnMM5Us5Zm,21095447,4l1fe,2023-02-07T16:25:31Z,2023-02-07T16:25:31Z,NONE,"> Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178 > > I suspect you could overwrite the datetime entry to achieve what you want And thank you for pointing me to it. At least, i can make a monkey patch for my need...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421052195,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421052195,IC_kwDOCGYnMM5Us4kj,21095447,4l1fe,2023-02-07T16:23:17Z,2023-02-07T16:23:57Z,NONE,"Isn't your suggestion too fundamental for the utility? The bigger flexibility, the bigger complexity. Your idea make sense defenitely, but how often do you make schema changes? And how many people could benefit from it, what do you think?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421033725,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421033725,IC_kwDOCGYnMM5Us0D9,193185,cldellow,2023-02-07T16:12:13Z,2023-02-07T16:12:13Z,NONE,"I think the bigger issue is that `sqlite-utils` mixes mechanism (it implements the [12-step way to alter SQLite tables](https://www.sqlite.org/lang_altertable.html#otheralter)) and policy (it has an opinionated stance on what column types should be used). That might be a design choice to make it accessible to users by providing a reasonable set of defaults, but it doesn't quite fit my use case. It might make sense to extract a separate library that provides just the mechanisms, and then `sqlite-utils` would sit on top of that library with its opinionated set of policies. That would be a very big change, though. I might take a stab at extracting the library, but just for the table schema migration piece, not all the other features that `sqlite-utils` supports. I wouldn't expect `sqlite-utils` to depend on it. Part of my motivation is that I want to provide some other abilities, too, like support for CHECK constraints. I see that the issue in this repo (https://github.com/simonw/sqlite-utils/issues/358) proposes a bunch of short-hand constraints, which I wouldn't want to accidentally expose to people -- I want a layer that is a 1:1 mapping to SQLite.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421022917,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421022917,IC_kwDOCGYnMM5UsxbF,21095447,4l1fe,2023-02-07T16:06:03Z,2023-02-07T16:08:58Z,NONE,"> Do you see a way to enable it without affecting existing users or bumping the major version number? I don't see a clean solution, only extending code with a side variable that tells us we want to apply advanced types instead of basic. it could be a similiar command like `tranform-v2 --type column DATETIME` or a cli option `transform --adv-type column DATETIME` along with a dict that contains the advanced types. Then with knowledge that we run an advanced command we take that dictionary somehow, we can wrap the current and new dictionaries by a superdict and work with it everywhere according to the knowledge. This way shouldn't affect users who are using the previous lib versions and it have to be merged in the next major one. But this way looks a bad design, too messy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420992261,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420992261,IC_kwDOCGYnMM5Usp8F,193185,cldellow,2023-02-07T15:45:58Z,2023-02-07T15:45:58Z,NONE,"I'd support that, but I'm not the author of this library. One challenge is that would be a breaking change. Do you see a way to enable it without affecting existing users or bumping the major version number?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420966995,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420966995,IC_kwDOCGYnMM5UsjxT,21095447,4l1fe,2023-02-07T15:29:28Z,2023-02-07T15:29:28Z,NONE,"I could, of course. Doest it worth bringing such the improvement to the library?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420809773,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420809773,IC_kwDOCGYnMM5Ur9Yt,193185,cldellow,2023-02-07T13:53:01Z,2023-02-07T13:53:01Z,NONE,"Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178 I suspect you could overwrite the datetime entry to achieve what you want","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420496447,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420496447,IC_kwDOCGYnMM5Uqw4_,21095447,4l1fe,2023-02-07T09:57:38Z,2023-02-07T09:57:38Z,NONE,"> That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work. It works, but a column becomes `TEXT` ```python In [1]: import sqlite_utils In [2]: db = sqlite_utils.Database('events.sqlite') In [3]: table = db['cards.chunk.get'] In [4]: table.columns_dict Out[4]: {'id': int, 'timestamp': float, 'data_chunk_number': int, 'user_id': str, 'meta_duplication_source_id': int, 'context_sort_attribute': str, 'context_sort_order': str} In [5]: from datetime import datetime In [7]: table.transform(types={'timestamp': datetime}) In [8]: table.columns_dict Out[8]: {'id': int, 'timestamp': str, 'data_chunk_number': int, 'user_id': str, 'meta_duplication_source_id': int, 'context_sort_attribute': str, 'context_sort_order': str} ``` ```bash ❯ sqlite-utils schema events.sqlite cards.chunk.get CREATE TABLE ""cards.chunk.get"" ( [id] INTEGER PRIMARY KEY NOT NULL, [timestamp] TEXT, ... ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419740776,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1419740776,IC_kwDOCGYnMM5Un4Zo,193185,cldellow,2023-02-06T20:59:01Z,2023-02-06T20:59:01Z,NONE,"That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419734229,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1419734229,IC_kwDOCGYnMM5Un2zV,193185,cldellow,2023-02-06T20:53:28Z,2023-02-06T21:16:29Z,NONE,"I think it's not currently possible: sqlite-utils requires that it be one of `integer`, `text`, `float`, `blob` ([see code](https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/cli.py#L2266)) IMO, this is a bit of friction and it would be nice if it was more permissive. SQLite permits developers to use any data type when creating a table. For example, this is a perfectly cromulent sqlite session that creates a table with columns of type `baz` and `bar`: ``` sqlite> create table foo(column1 baz, column2 bar); sqlite> .schema foo CREATE TABLE foo(column1 baz, column2 bar); sqlite> select * from pragma_table_info('foo'); cid name type notnull dflt_value pk ---------- ---------- ---------- ---------- ---------- ---------- 0 column1 baz 0 0 1 column2 bar 0 0 ``` The idea is that the application developer will know what meaning to ascribe to those types. For example, I'm working on a plugin to Datasette. Dates are tricky to handle. If you have some existing rows, you can look at the values in them to know how a user is serializing the dates -- as an ISO 8601 string? An RFC 3339 string? With millisecond precision? With timezone offset? But if you don't yet have any rows, you have to guess. If the column is of type `TEXT`, you don't even know that it's meant to hold a date! In this case, my plugin will look to see if the column is of type `DATE` or `DATETIME`, and assume a certain representation when writing. Perhaps there is an argument that sqlite-utils is trying to conform to SQLite's strict mode, and that is why it limits the choices. In strict mode, SQLite requires that the data type be one of `INT`, `INTEGER`, `REAL`, `TEXT`, `BLOB`, `ANY`. But that can't be the case -- sqlite-utils supports `FLOAT`, which is not one of the valid types in strict mode, and it rejects `INT`, `REAL` and `ANY`, which _are_ valid.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419390560,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1419390560,IC_kwDOCGYnMM5Umi5g,21095447,4l1fe,2023-02-06T16:43:47Z,2023-02-06T16:43:47Z,NONE,"> SQLite doesn't have a native `DATETIME` type. It stores dates internally as strings and then has [functions](https://www.sqlite.org/lang_datefunc.html) to work with date-like strings. Yes it's weird. That's correct. But my issue is about the application level libraries that, i suppose, have better data understanding if see a specific type such as `DATETIME`. I'm writing data with **dataset** i've mentioned. The lib changes its behavior depending on a type. I saw different behavior with types `DATETIME, FLOAT, TEXT`. Dataset, for their part, is built upon Sqlalchemy, you know what it is. To be honest, i didn't dive into the details of why the behavior changes, but when i altered manually by other util a type of column to `DATETIME` things got back to normal. On the matter, can i achieve it with Sqlite Utils at the moment?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419357290,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1419357290,IC_kwDOCGYnMM5Umaxq,25778,eyeseast,2023-02-06T16:21:44Z,2023-02-06T16:21:44Z,CONTRIBUTOR,SQLite doesn't have a native `DATETIME` type. It stores dates internally as strings and then has [functions](https://www.sqlite.org/lang_datefunc.html) to work with date-like strings. Yes it's weird.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`, https://github.com/simonw/sqlite-utils/issues/523#issuecomment-1407264466,https://api.github.com/repos/simonw/sqlite-utils/issues/523,1407264466,IC_kwDOCGYnMM5T4SbS,536941,fgregg,2023-01-28T02:41:14Z,2023-01-28T02:41:14Z,CONTRIBUTOR,"I also often then run another little script to cast all empty strings to null, but i save that for another issue if this gets accepted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560651350,Feature request: trim all leading and trailing white space for all columns for all tables in a database, https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1539109587,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1539109587,IC_kwDOCGYnMM5bvPLT,9599,simonw,2023-05-08T22:00:46Z,2023-05-08T22:00:46Z,OWNER,"> Hey, isn't this essentially the same issue as #448 ? Yes it is, good catch!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,rows_from_file() raises confusing error if file-like object is not in binary mode, https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1421571810,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1421571810,IC_kwDOCGYnMM5Uu3bi,167893,mcarpenter,2023-02-07T22:43:09Z,2023-02-07T22:43:09Z,CONTRIBUTOR,"Hey, isn't this essentially the same issue as #448 ?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,rows_from_file() raises confusing error if file-like object is not in binary mode, https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1369333759,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1369333759,IC_kwDOCGYnMM5Rnl__,9599,simonw,2023-01-03T02:23:43Z,2023-01-03T02:23:43Z,OWNER,"The documentation here does at least say the following: https://sqlite-utils.datasette.io/en/3.30/python-api.html#reading-rows-from-a-file > - **fp** (*BinaryIO*) -- a file-like object containing binary data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,rows_from_file() raises confusing error if file-like object is not in binary mode, https://github.com/simonw/sqlite-utils/issues/52#issuecomment-550694197,https://api.github.com/repos/simonw/sqlite-utils/issues/52,550694197,MDEyOklzc3VlQ29tbWVudDU1MDY5NDE5Nw==,9599,simonw,2019-11-07T04:11:05Z,2019-11-07T04:11:05Z,OWNER,I just hit this bug while running `healthkit-to-sqlite`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",476413293,Throws error if .insert_all() / .upsert_all() called with empty list, https://github.com/simonw/sqlite-utils/issues/517#issuecomment-1344965367,https://api.github.com/repos/simonw/sqlite-utils/issues/517,1344965367,IC_kwDOCGYnMM5QKor3,9599,simonw,2022-12-10T01:26:31Z,2022-12-10T01:26:31Z,OWNER,At some point I should drop it from all of these other projects too: https://cs.github.com/?scopeName=All+repos&scope=&q=user%3Asimonw+%223.6%22+path%3A.github%2Fworkflows%2F*,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1487757143,Drop support for Python 3.6, https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339844639,https://api.github.com/repos/simonw/sqlite-utils/issues/516,1339844639,IC_kwDOCGYnMM5P3Ggf,122043,briandorsey,2022-12-06T19:08:13Z,2022-12-06T19:08:13Z,NONE,"Reference: tqdm (https://tqdm.github.io/) shows a progress bar when total is known, and falls back to counting units of work done for streams. File input vs. stdin seems like a similar situation. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1479914599,Feature request: output number of ignored/replaced rows for insert command, https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339839767,https://api.github.com/repos/simonw/sqlite-utils/issues/516,1339839767,IC_kwDOCGYnMM5P3FUX,122043,briandorsey,2022-12-06T19:04:17Z,2022-12-06T19:04:17Z,NONE,"Current behavior is different when importing via stdin vs. a file. Imports from a file give a progress bar. For this new request, I'd love to see total imported and total ignored/replaced in both cases. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1479914599,Feature request: output number of ignored/replaced rows for insert command, https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339837520,https://api.github.com/repos/simonw/sqlite-utils/issues/516,1339837520,IC_kwDOCGYnMM5P3ExQ,122043,briandorsey,2022-12-06T19:02:30Z,2022-12-06T19:02:30Z,NONE,"`--verbose` or `--verbosity=ABC` were the flags I looked for. Expected to see them at a global level near `--version`. But only sharing because that's where I looked first, I don't have a strong opinion on the exact wording/location. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1479914599,Feature request: output number of ignored/replaced rows for insert command, https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339834918,https://api.github.com/repos/simonw/sqlite-utils/issues/516,1339834918,IC_kwDOCGYnMM5P3EIm,9599,simonw,2022-12-06T19:00:18Z,2022-12-06T19:00:35Z,OWNER,"Right now the command produces no output at all. Maybe a `--verbose` mode that writes these numbers to standard error (or even standard output since it's an option)? Is there a better name than `--verbose` for this? `--summary` perhaps?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1479914599,Feature request: output number of ignored/replaced rows for insert command, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539100300,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539100300,IC_kwDOCGYnMM5bvM6M,9599,simonw,2023-05-08T21:50:51Z,2023-05-08T21:50:51Z,OWNER,Seeing as `sqlite-utils` doesn't currently provide mechanisms for adding `check` constraints like this I'm going to leave this - I'm happy with the fix I put in for the `not null` constraints.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539099703,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539099703,IC_kwDOCGYnMM5bvMw3,9599,simonw,2023-05-08T21:50:06Z,2023-05-08T21:50:06Z,OWNER,"Applying the fix from the PR here doesn't fix the above problem either: - https://github.com/simonw/sqlite-utils/pull/515 So it looks like these kinds of `check` constraints currently aren't compatible with how `upsert()` works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539094287,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539094287,IC_kwDOCGYnMM5bvLcP,9599,simonw,2023-05-08T21:44:11Z,2023-05-08T21:44:11Z,OWNER,"OK, this fails silently: ```python import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute('''CREATE TABLE employees ( id INTEGER PRIMARY KEY, name TEXT, age INTEGER, salary REAL, CHECK (salary is not null and salary > 0) );''') db[""employees""].upsert({""id"": 1, ""name"": ""Bob""}, pk=""id"") list(db[""employees""].rows) ```` It outputs: ```python [] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539079507,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539079507,IC_kwDOCGYnMM5bvH1T,9599,simonw,2023-05-08T21:28:37Z,2023-05-08T21:28:37Z,OWNER,"> This means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted. Huh... on that basis, it's possible my fix in https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304 is incomplete. I only covered the 'not null' case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539078429,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539078429,IC_kwDOCGYnMM5bvHkd,9599,simonw,2023-05-08T21:27:40Z,2023-05-08T21:27:40Z,OWNER,"Dupe of: - #538","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316530539,https://api.github.com/repos/simonw/sqlite-utils/issues/512,1316530539,IC_kwDOCGYnMM5OeKlr,9599,simonw,2022-11-16T07:49:50Z,2022-11-16T07:49:50Z,OWNER,Tests passed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450952393,mypy failures in CI, https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316447182,https://api.github.com/repos/simonw/sqlite-utils/issues/512,1316447182,IC_kwDOCGYnMM5Od2PO,9599,simonw,2022-11-16T06:32:31Z,2022-11-16T06:32:31Z,OWNER,"Test failed again: https://github.com/simonw/sqlite-utils/actions/runs/3476950474/jobs/5812663096 `E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/s/spatialite/libsqlite3-mod-spatialite_4.3.0a-6build1_amd64.deb Unable to connect to azure.archive.ubuntu.com:http:` That looks like an intermittent error. I'll try running it again in the morning.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450952393,mypy failures in CI, https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316437748,https://api.github.com/repos/simonw/sqlite-utils/issues/512,1316437748,IC_kwDOCGYnMM5Odz70,9599,simonw,2022-11-16T06:24:31Z,2022-11-16T06:24:31Z,OWNER,"``` sqlite-utils % pipx run no_implicit_optional . Calculating full-repo metadata... Executing codemod... 11.43s 98% complete, 0.24s estimated for 5 files to go... ``` Then: ``` Finished codemodding 239 files! - Transformed 239 files successfully. - Skipped 0 files. - Failed to codemod 0 files. - 0 warnings were generated. ``` Here's the diff: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a06f4b7..e819d17 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -297,12 +297,12 @@ class Database: def __init__( self, - filename_or_conn: Union[str, pathlib.Path, sqlite3.Connection] = None, + filename_or_conn: Optional[Union[str, pathlib.Path, sqlite3.Connection]] = None, memory: bool = False, - memory_name: str = None, + memory_name: Optional[str] = None, recreate: bool = False, recursive_triggers: bool = True, - tracer: Callable = None, + tracer: Optional[Callable] = None, use_counts_table: bool = False, ): assert (filename_or_conn is not None and (not memory and not memory_name)) or ( @@ -341,7 +341,7 @@ class Database: self.conn.close() @contextlib.contextmanager - def tracer(self, tracer: Callable = None): + def tracer(self, tracer: Optional[Callable] = None): """""" Context manager to temporarily set a tracer function - all executed SQL queries will be passed to this. @@ -378,7 +378,7 @@ class Database: def register_function( self, - fn: Callable = None, + fn: Optional[Callable] = None, deterministic: bool = False, replace: bool = False, name: Optional[str] = None, @@ -879,7 +879,7 @@ class Database: pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, hash_id: Optional[str] = None, hash_id_columns: Optional[Iterable[str]] = None, @@ -1129,7 +1129,7 @@ class Database: sql += "" [{}]"".format(name) self.execute(sql) - def init_spatialite(self, path: str = None) -> bool: + def init_spatialite(self, path: Optional[str] = None) -> bool: """""" The ``init_spatialite`` method will load and initialize the SpatiaLite extension. The ``path`` argument should be an absolute path to the compiled extension, which @@ -1182,7 +1182,7 @@ class Queryable: def count_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, ) -> int: """""" @@ -1213,12 +1213,12 @@ class Queryable: def rows_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, - order_by: str = None, + order_by: Optional[str] = None, select: str = ""*"", - limit: int = None, - offset: int = None, + limit: Optional[int] = None, + offset: Optional[int] = None, ) -> Generator[dict, None, None]: """""" Iterate over every row in this table or view that matches the specified where clause. @@ -1251,11 +1251,11 @@ class Queryable: def pks_and_rows_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, - order_by: str = None, - limit: int = None, - offset: int = None, + order_by: Optional[str] = None, + limit: Optional[int] = None, + offset: Optional[int] = None, ) -> Generator[Tuple[Any, Dict], None, None]: """""" Like ``.rows_where()`` but returns ``(pk, row)`` pairs - ``pk`` can be a single value or tuple. @@ -1345,7 +1345,7 @@ class Table(Queryable): pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, batch_size: int = 100, hash_id: Optional[str] = None, @@ -1545,7 +1545,7 @@ class Table(Queryable): pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, hash_id: Optional[str] = None, hash_id_columns: Optional[Iterable[str]] = None, @@ -2464,7 +2464,7 @@ class Table(Queryable): columns: Optional[Iterable[str]] = None, limit: Optional[int] = None, offset: Optional[int] = None, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, quote: bool = False, ) -> Generator[dict, None, None]: @@ -2527,7 +2527,7 @@ class Table(Queryable): def delete_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, analyze: bool = False, ) -> ""Table"": ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450952393,mypy failures in CI, https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304320521,https://api.github.com/repos/simonw/sqlite-utils/issues/511,1304320521,IC_kwDOCGYnMM5NvloJ,7908073,chapmanjacobd,2022-11-04T22:54:09Z,2022-11-04T22:59:54Z,CONTRIBUTOR,I ran `PRAGMA integrity_check` and it returned `ok`. but then I tried restoring from a backup and I didn't get this `IntegrityError: constraint failed` error. So I think it was just something wrong with my database. If it happens again I will first try to reindex and see if that fixes the issue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1436539554,"[insert_all, upsert_all] IntegrityError: constraint failed", https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304078945,https://api.github.com/repos/simonw/sqlite-utils/issues/511,1304078945,IC_kwDOCGYnMM5Nuqph,7908073,chapmanjacobd,2022-11-04T19:38:36Z,2022-11-04T20:13:17Z,CONTRIBUTOR,"Even more bizarre, the source db only has one record and the target table has no conflicting record: ``` 875 0.3s lb:/ (main|✚2) [0|0]🌺 sqlite-utils tube_71.db 'select * from media where path = ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz""' | jq [ { ""size"": null, ""time_created"": null, ""play_count"": 1, ""language"": null, ""view_count"": null, ""width"": null, ""height"": null, ""fps"": null, ""average_rating"": null, ""live_status"": null, ""age_limit"": null, ""uploader"": null, ""time_played"": 0, ""path"": ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz"", ""id"": ""088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz/074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv"", ""ie_key"": ""ArchiveOrg"", ""playlist_path"": ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz"", ""duration"": 1424.05, ""tags"": null, ""title"": ""074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv"" } ] 875 0.3s lb:/ (main|✚2) [0|0]🥧 sqlite-utils video.db 'select * from media where path = ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz""' | jq [] ``` I've been able to use this code successfully several times before so not sure what's causing the issue. I guess the way that I'm handling multiple databases is an issue, though it hasn't ever inserted into the source db, not sure what's different. The only reasonable explanation is that it is trying to insert into the source db from the source db for some reason? Or maybe sqlite3 is checking the source db for primary key violation because the table name is the same","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1436539554,"[insert_all, upsert_all] IntegrityError: constraint failed", https://github.com/simonw/sqlite-utils/issues/510#issuecomment-1320394127,https://api.github.com/repos/simonw/sqlite-utils/issues/510,1320394127,IC_kwDOCGYnMM5Os52P,1176293,ar-jan,2022-11-18T18:37:51Z,2022-11-18T18:37:51Z,NONE,"I guess it is not incorrect when it says the version is `4`, though it is confusing. Maybe it doesn't even refer to FTS4/FTS5 versions, but something else? In any case, it's not related to sqlite-utils, but SQLite itself.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434911255,Cannot enable FTS5 despite it being available, https://github.com/simonw/sqlite-utils/issues/510#issuecomment-1318777114,https://api.github.com/repos/simonw/sqlite-utils/issues/510,1318777114,IC_kwDOCGYnMM5OmvEa,7908073,chapmanjacobd,2022-11-17T15:09:47Z,2022-11-17T15:09:47Z,CONTRIBUTOR,"why close? is the only problem that the _config table that incorrectly says 4 for fts5? if so, that's still something that should be fixed","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434911255,Cannot enable FTS5 despite it being available, https://github.com/simonw/sqlite-utils/issues/510#issuecomment-1318431389,https://api.github.com/repos/simonw/sqlite-utils/issues/510,1318431389,IC_kwDOCGYnMM5Olaqd,1176293,ar-jan,2022-11-17T10:36:28Z,2022-11-17T10:36:28Z,NONE,The virtual table's _config `version: 4` seems to indicate FTS5.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434911255,Cannot enable FTS5 despite it being available, https://github.com/simonw/sqlite-utils/issues/507#issuecomment-1297859539,https://api.github.com/repos/simonw/sqlite-utils/issues/507,1297859539,IC_kwDOCGYnMM5NW8PT,7908073,chapmanjacobd,2022-11-01T00:40:16Z,2022-11-01T00:40:16Z,CONTRIBUTOR,"Ideally people could fix their data if they run into this issue. If you are using filenames try [convmv](https://linux.die.net/man/1/convmv) ``` convmv --preserve-mtimes -f utf8 -t utf8 --notest -i -r . ``` maybe this script will also help: ```py import argparse, shutil from pathlib import Path import ftfy from xklb import utils from xklb.utils import log def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument(""paths"", nargs='*') parser.add_argument(""--verbose"", ""-v"", action=""count"", default=0) args = parser.parse_args() log.info(utils.dict_filter_bool(args.__dict__)) return args def rename_invalid_paths() -> None: args = parse_args() for path in args.paths: log.info(path) for p in sorted([str(p) for p in Path(path).rglob(""*"")], key=len): fixed = ftfy.fix_text(p, uncurl_quotes=False).replace(""\r\n"", ""\n"").replace(""\r"", ""\n"").replace(""\n"", """") if p != fixed: try: shutil.move(p, fixed) except FileNotFoundError: log.warning(""FileNotFound. %s"", p) else: log.info(fixed) if __name__ == ""__main__"": rename_invalid_paths() ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430325103,conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character, https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298879701,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1298879701,IC_kwDOCGYnMM5Na1TV,9599,simonw,2022-11-01T17:37:13Z,2022-11-01T17:37:13Z,OWNER,"The question I was originally trying to answer here was this: how many rows were actually inserted by that call to `.insert_all()`? I don't know that `.rowcount` would ever be useful here, since the ""correct"" answer depends on other factors - had I determined to ignore or replace records with a primary key that matches an existing record for example? So I think if people need `rowcount` they can get it by using a `cursor` directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298877872,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1298877872,IC_kwDOCGYnMM5Na02w,9599,simonw,2022-11-01T17:35:30Z,2022-11-01T17:35:30Z,OWNER,"This may not make sense. First, `.last_rowid` is a property on table - but that doesn't make sense for `rowcount` since it should clearly be a property on the database itself (you can run a query directly using `db.execute()` without going through a `Table` object). So I tried this prototype: ```diff diff --git a/docs/python-api.rst b/docs/python-api.rst index 206e5e6..78d3a8d 100644 --- a/docs/python-api.rst +++ b/docs/python-api.rst @@ -186,6 +186,15 @@ The ``db.query(sql)`` function executes a SQL query and returns an iterator over # {'name': 'Cleo'} # {'name': 'Pancakes'} +After executing a query the ``db.rowcount`` property on that database instance will reflect the number of rows affected by any insert, update or delete operations performed by that query: + +.. code-block:: python + + db = Database(memory=True) + db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}]) + print(db.rowcount) + # Outputs: 2 + .. _python_api_execute: db.execute(sql, params) diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a06f4b7..c19c2dd 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -294,6 +294,8 @@ class Database: _counts_table_name = ""_counts"" use_counts_table = False + # Number of rows inserted, updated or deleted + rowcount: Optional[int] = None def __init__( self, @@ -480,9 +482,11 @@ class Database: if self._tracer: self._tracer(sql, parameters) if parameters is not None: - return self.conn.execute(sql, parameters) + cursor = self.conn.execute(sql, parameters) else: - return self.conn.execute(sql) + cursor = self.conn.execute(sql) + self.rowcount = cursor.rowcount + return cursor def executescript(self, sql: str) -> sqlite3.Cursor: """""" ``` But this happens: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}])
>>> db.rowcount -1 ``` Turning on query tracing demonstrates why: ```pycon >>> db = Database(memory=True, tracer=print) PRAGMA recursive_triggers=on; None >>> db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}]) select name from sqlite_master where type = 'view' None select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'view' None CREATE TABLE [dogs] ( [name] TEXT ); None select name from sqlite_master where type = 'view' None INSERT INTO [dogs] ([name]) VALUES (?), (?); ['Cleo', 'Pancakes'] select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'table' None PRAGMA table_info([dogs]) None
>>> ``` The `.insert_all()` function does a bunch of other queries too, so `.rowcount` is quickly over-ridden by the same result from extra queries that it executed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1296358636,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1296358636,IC_kwDOCGYnMM5NRNzs,9599,simonw,2022-10-30T21:52:11Z,2022-10-30T21:52:11Z,OWNER,This could work in a similar way to `db.insert(...).last_rowid`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/sqlite-utils/issues/505#issuecomment-1291216193,https://api.github.com/repos/simonw/sqlite-utils/issues/505,1291216193,IC_kwDOCGYnMM5M9mVB,9599,simonw,2022-10-25T22:41:16Z,2022-10-25T22:41:16Z,OWNER,Tweeted about it here: https://twitter.com/simonw/status/1585038766678609921,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423182778,Release sqlite-utils 3.30, https://github.com/simonw/sqlite-utils/issues/505#issuecomment-1291203911,https://api.github.com/repos/simonw/sqlite-utils/issues/505,1291203911,IC_kwDOCGYnMM5M9jVH,9599,simonw,2022-10-25T22:21:02Z,2022-10-25T22:21:02Z,OWNER,"- Now tested against Python 3.11. ([#502](https://github.com/simonw/sqlite-utils/issues/502)) - New `table.search_sql(include_rank=True)` option, which adds a `rank` column to the generated SQL. Thanks, Jacob Chapman. ([#480](https://github.com/simonw/sqlite-utils/pull/480)) - Progress bars now display for newline-delimited JSON files using the `--nl` option. Thanks, Mischa Untaga. ([#485](https://github.com/simonw/sqlite-utils/issues/485)) - New `db.close()` method. ([#504](https://github.com/simonw/sqlite-utils/issues/504)) - Conversion functions passed to [table.convert(...)](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-convert) can now return lists or dictionaries, which will be inserted into the database as JSON strings. ([#495](https://github.com/simonw/sqlite-utils/issues/495)) - `sqlite-utils install` and `sqlite-utils uninstall` commands for installing packages into the same virtual environment as `sqlite-utils`, [described here](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install). ([#483](https://github.com/simonw/sqlite-utils/issues/483)) - New [sqlite_utils.utils.flatten()](https://sqlite-utils.datasette.io/en/stable/reference.html#reference-utils-flatten) utility function. ([#500](https://github.com/simonw/sqlite-utils/issues/500)) - Documentation on [using Just](https://sqlite-utils.datasette.io/en/stable/contributing.html#contributing-just) to run tests, linters and build documentation. - Documentation now covers the [Release process](https://sqlite-utils.datasette.io/en/stable/contributing.html#release-process) for this package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423182778,Release sqlite-utils 3.30, https://github.com/simonw/sqlite-utils/issues/504#issuecomment-1291136971,https://api.github.com/repos/simonw/sqlite-utils/issues/504,1291136971,IC_kwDOCGYnMM5M9S_L,9599,simonw,2022-10-25T21:00:29Z,2022-10-25T21:00:29Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.close,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423069384,"db.close() method, calling db.conn.close()", https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291124413,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291124413,IC_kwDOCGYnMM5M9P69,9599,simonw,2022-10-25T20:47:34Z,2022-10-25T20:47:34Z,OWNER,TIL about this: https://til.simonwillison.net/python/os-remove-windows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291122389,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291122389,IC_kwDOCGYnMM5M9PbV,9599,simonw,2022-10-25T20:45:43Z,2022-10-25T20:45:43Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291115986,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291115986,IC_kwDOCGYnMM5M9N3S,9599,simonw,2022-10-25T20:39:24Z,2022-10-25T20:39:24Z,OWNER,"Used `psutil` to confirm that closing a SQLite connection closes the underlying file: https://til.simonwillison.net/python/too-many-open-files-psutil ```pycon >>> import psutil >>> import sqlite3 >>> for f in psutil.Process().open_files(): print(f) ... >>> sqlite3.connect(""/tmp/blah.db"") >>> conn = _ >>> for f in psutil.Process().open_files(): print(f) ... popenfile(path='/private/tmp/blah.db', fd=3) >>> conn.close() >>> for f in psutil.Process().open_files(): print(f) ... >>> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291111357,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291111357,IC_kwDOCGYnMM5M9Mu9,9599,simonw,2022-10-25T20:36:06Z,2022-10-25T20:36:06Z,OWNER,... or maybe Windows doesn't like attempts to remove a file that the process has opened?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291103021,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291103021,IC_kwDOCGYnMM5M9Kst,9599,simonw,2022-10-25T20:32:01Z,2022-10-25T20:32:01Z,OWNER,"This test reliably fails on Windows with Python 3.11. I'm going to skip the test for the moment to get back to green CI... but I'll leave this issue open. This is definitely concerning, I just don't have the right local environment to solve this at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291093581,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291093581,IC_kwDOCGYnMM5M9IZN,9599,simonw,2022-10-25T20:23:00Z,2022-10-25T20:23:00Z,OWNER,"I'm not hugely happy with my fix there: https://github.com/simonw/sqlite-utils/blob/c5d7ec1dd71fa1dce829bc8bb82b639018befd63/sqlite_utils/db.py#L321-L328 The problem here was that in the case where the `os.remove()` failed the `self.conn` property was NOT being set to a valid connection - which caused `__repr__` to fail later on. So now I catch the `os.remove()` error, set `self.conn` to a memory connection, then raise the error again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291088108,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291088108,IC_kwDOCGYnMM5M9HDs,9599,simonw,2022-10-25T20:17:36Z,2022-10-25T20:17:36Z,OWNER,"Now `mypy` is failing: ``` sqlite_utils/db.py:474: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:476: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:486: error: Item ""None"" of ""Optional[Any]"" has no attribute ""executescript"" sqlite_utils/db.py:603: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__enter__"" sqlite_utils/db.py:603: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__exit__"" sqlite_utils/db.py:604: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:607: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:1082: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__enter__"" sqlite_utils/db.py:1082: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__exit__"" sqlite_utils/db.py:1083: error: Item ""None"" of ""Optional[Any]"" has no attribute ""cursor"" sqlite_utils/db.py:1155: error: Item ""None"" of ""Optional[Any]"" has no attribute ""enable_load_extension"" sqlite_utils/db.py:1156: error: Item ""None"" of ""Optional[Any]"" has no attribute ""load_extension"" Found 12 errors in 1 file (checked 51 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291083188,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291083188,IC_kwDOCGYnMM5M9F20,9599,simonw,2022-10-25T20:12:52Z,2022-10-25T20:12:52Z,OWNER,"Failed again, but just noticed this: https://github.com/simonw/sqlite-utils/actions/runs/3323932266/jobs/5494890223 ``` > Database(filepath, recreate=True)[""t2""].insert({""foo"": ""bar""}) tests\test_recreate.py:31: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError(""'Database' object has no attribute 'conn'"") raised in repr()] Database object at 0x29fc125aa90> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291076031,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291076031,IC_kwDOCGYnMM5M9EG_,9599,simonw,2022-10-25T20:06:28Z,2022-10-25T20:06:28Z,OWNER,"This is the failing test: https://github.com/simonw/sqlite-utils/blob/7b2d1c0ffd0b874e280292b926f328a61cb31e2c/tests/test_recreate.py#L21-L32 I'm going to try a different way of creating the temporary file: https://docs.pytest.org/en/7.1.x/how-to/tmp_path.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291071627,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291071627,IC_kwDOCGYnMM5M9DCL,9599,simonw,2022-10-25T20:02:18Z,2022-10-25T20:02:18Z,OWNER,Passes on Windows with other Python versions for some reason.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/502#issuecomment-1291029761,https://api.github.com/repos/simonw/sqlite-utils/issues/502,1291029761,IC_kwDOCGYnMM5M840B,9599,simonw,2022-10-25T19:21:44Z,2022-10-25T19:21:44Z,OWNER,"Replicated locally using a fresh virtual environment with Python 3.11 and: pytest -k test_query_invalid_function","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422954582,Fix tests for Python 3.11, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282830806,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282830806,IC_kwDOCGYnMM5MdnHW,9599,simonw,2022-10-18T18:23:36Z,2022-10-18T18:23:36Z,OWNER,"Tests pass now. Updated docs: - https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query - and many other places on that page","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282819035,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282819035,IC_kwDOCGYnMM5MdkPb,9599,simonw,2022-10-18T18:15:05Z,2022-10-18T18:15:05Z,OWNER,I'm going to skip the cog test on Python 3.6 to address this. The documentation on the website will show the available list of options for 3.7 and higher.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282817901,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282817901,IC_kwDOCGYnMM5Mdj9t,9599,simonw,2022-10-18T18:14:35Z,2022-10-18T18:14:35Z,OWNER,"Now the 3.6 tests fail - because the new release of tabulate dropped support for that Python version (so on Python 3.6 you get an older version): https://github.com/simonw/sqlite-utils/actions/runs/3275842849/jobs/5391181675 https://github.com/astanin/python-tabulate/blame/20c6370d5da2dae89b305bfb6c7f12a0f8b7236c/pyproject.toml#L22 shows minimum is 3.7 now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282813168,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282813168,IC_kwDOCGYnMM5Mdizw,9599,simonw,2022-10-18T18:12:15Z,2022-10-18T18:12:15Z,OWNER,"Here's the new Tabulate release: - https://github.com/astanin/python-tabulate/releases/tag/v0.9.0 - https://github.com/astanin/python-tabulate/compare/v0.8.10...v0.9.0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282800547,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282800547,IC_kwDOCGYnMM5Mdfuj,9599,simonw,2022-10-18T18:02:09Z,2022-10-18T18:02:09Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-flatten,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282780770,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282780770,IC_kwDOCGYnMM5Mda5i,9599,simonw,2022-10-18T17:45:56Z,2022-10-18T17:46:05Z,OWNER,I think the public interface is a `flatten(row)` function that does `dict(_flatten(row))`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282779755,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282779755,IC_kwDOCGYnMM5Mdapr,9599,simonw,2022-10-18T17:45:10Z,2022-10-18T17:45:10Z,OWNER,It should go in `sqlite_utils.utils` - documented here: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite-utils-utils,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282778928,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282778928,IC_kwDOCGYnMM5Mdacw,9599,simonw,2022-10-18T17:44:20Z,2022-10-18T17:44:20Z,OWNER,"Here's how it works: https://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/sqlite_utils/cli.py#L1847-L1848 https://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/sqlite_utils/cli.py#L1082-L1088","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515752204,https://api.github.com/repos/simonw/sqlite-utils/issues/50,515752204,MDEyOklzc3VlQ29tbWVudDUxNTc1MjIwNA==,9599,simonw,2019-07-28T10:48:14Z,2019-07-28T10:48:14Z,OWNER,"Here's the diff where I tried to use `.executemany()` and ran into the `lastrowid` problem: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index ef55976..7f85759 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -881,13 +881,10 @@ class Table: or_what=or_what, table=self.name, columns="", "".join(""[{}]"".format(c) for c in all_columns), - rows="", "".join( - """""" + rows="""""" ({placeholders}) """""".format( - placeholders="", "".join([""?""] * len(all_columns)) - ) - for record in chunk + placeholders="", "".join([""?""] * len(all_columns)) ), ) values = [] @@ -902,15 +899,15 @@ class Table: extract_table = extracts[key] value = self.db[extract_table].lookup({""value"": value}) record_values.append(value) - values.extend(record_values) + values.append(record_values) with self.db.conn: try: - result = self.db.conn.execute(sql, values) + result = self.db.conn.executemany(sql, values) except sqlite3.OperationalError as e: if alter and ("" has no column "" in e.args[0]): # Attempt to add any missing columns, then try again self.add_missing_columns(chunk) - result = self.db.conn.execute(sql, values) + result = self.db.conn.executemany(sql, values) else: raise self.last_rowid = result.lastrowid ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473083260,"""Too many SQL variables"" on large inserts", https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515752129,https://api.github.com/repos/simonw/sqlite-utils/issues/50,515752129,MDEyOklzc3VlQ29tbWVudDUxNTc1MjEyOQ==,9599,simonw,2019-07-28T10:46:49Z,2019-07-28T10:46:49Z,OWNER,"The problem with `.executemany()` is it breaks `lastrowid`: > This read-only attribute provides the rowid of the last modified row. It is only set if you issued an INSERT or a REPLACE statement using the execute() method. For operations other than INSERT or REPLACE or when executemany() is called, lastrowid is set to None. So I think I need to continue to use my existing way of executing bulk inserts (with a giant repeated `INSERT INTO ... VALUES` block) but ensure that I calculate the chunk size such that I don't ever try to pass more than 999 values at once.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473083260,"""Too many SQL variables"" on large inserts", https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515751719,https://api.github.com/repos/simonw/sqlite-utils/issues/50,515751719,MDEyOklzc3VlQ29tbWVudDUxNTc1MTcxOQ==,9599,simonw,2019-07-28T10:40:11Z,2019-07-28T10:40:11Z,OWNER,"I think the fix here is for me to switch to using `executemany()` - example from the Python docs: https://docs.python.org/3/library/sqlite3.html ```python purchases = [('2006-03-28', 'BUY', 'IBM', 1000, 45.00), ('2006-04-05', 'BUY', 'MSFT', 1000, 72.00), ('2006-04-06', 'SELL', 'IBM', 500, 53.00), ] c.executemany('INSERT INTO stocks VALUES (?,?,?,?,?)', purchases) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473083260,"""Too many SQL variables"" on large inserts", https://github.com/simonw/sqlite-utils/issues/50#issuecomment-1303660293,https://api.github.com/repos/simonw/sqlite-utils/issues/50,1303660293,IC_kwDOCGYnMM5NtEcF,7908073,chapmanjacobd,2022-11-04T14:38:36Z,2022-11-04T14:38:36Z,CONTRIBUTOR,where did you see the limit as 999? I believe the limit has been 32766 for quite some time. If you could detect which one this could speed up batch insert of some types of data significantly,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473083260,"""Too many SQL variables"" on large inserts", https://github.com/simonw/sqlite-utils/issues/497#issuecomment-1291146850,https://api.github.com/repos/simonw/sqlite-utils/issues/497,1291146850,IC_kwDOCGYnMM5M9VZi,9599,simonw,2022-10-25T21:09:28Z,2022-10-25T21:09:28Z,OWNER,"Yeah, `table.columns` and `table.columns_dict` are meant to handle this: https://sqlite-utils.datasette.io/en/stable/python-api.html#columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393212964,column_names, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1532481862,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1532481862,IC_kwDOCGYnMM5bV9FG,1231935,xavdid,2023-05-03T05:53:26Z,2023-05-03T05:53:26Z,NONE,"Would love to put our heads together for improvements here. I _think_ anything that is `argname=DEFAULT` needs to be typed as `argname: str | Default = DEFAULT` (replacing `str` with the appropriate type(s)). We may be able to get clever and tie the types to that key in the `_defaults` dict (definitely possible in Typescript, but I'm less familiar with advanced python types). Right now, all args are typed as `Default`, which means all callers get type errors. As for table/view, given that they don't have the same methods, it would be nice to be able to get one or the other specifically. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1294408928,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1294408928,IC_kwDOCGYnMM5NJxzg,39538958,justmars,2022-10-28T03:36:56Z,2022-10-28T03:37:50Z,NONE,"With respect to the typing of Table class itself, my interim solution: ```python from sqlite_utils.db import Table def tbl(self, table_name: str) -> Table: tbl = self.db[table_name] if isinstance(tbl, Table): return tbl raise Exception(f""Missing {table_name=}"") ``` With respect to @chapmanjacobd concern on the `DEFAULT` being an empty class, have also been using `# type: ignore`, e.g. ```python @classmethod def insert_list(cls, areas: list[str]): return meta.tbl(meta.Areas).insert_all( ({""area"": a} for a in areas), ignore=True # type: ignore ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1291170072,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1291170072,IC_kwDOCGYnMM5M9bEY,9599,simonw,2022-10-25T21:36:12Z,2022-10-25T21:36:12Z,OWNER,"I was going to suggest using `db.table(name)` instead of `db[name]` - but it looks like that method will have the same problem: https://github.com/simonw/sqlite-utils/blob/defa2974c6d3abc19be28d6b319649b8028dc966/sqlite_utils/db.py#L497-L506 I could change `sqlite-utils` so `db.table(name)` always returns a table and you need to call `db.view(name)` if you want to access a view - that would require bumping to 4.0 though. I'm not convinced that's the best approach here either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1291167887,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1291167887,IC_kwDOCGYnMM5M9aiP,9599,simonw,2022-10-25T21:33:25Z,2022-10-25T21:33:25Z,OWNER,"I do care about this, but I'm not hugely experienced with types yet so I'm open to suggestions about how to do it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291159549,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291159549,IC_kwDOCGYnMM5M9Yf9,9599,simonw,2022-10-25T21:23:01Z,2022-10-25T21:23:01Z,OWNER,"I've decided not to explicitly document this, since it's consistent with how other parts of the library work already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291152433,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291152433,IC_kwDOCGYnMM5M9Wwx,9599,simonw,2022-10-25T21:14:54Z,2022-10-25T21:14:54Z,OWNER,"There is a case where the function can return a dictionary at the moment: `multi=True` ```python table.convert( ""title"", lambda v: {""upper"": v.upper(), ""lower"": v.lower()}, multi=True ) ``` But I think this change is still compatible with that. if you don't use `multi=True` then the return value will be stringified. If you DO use `multi=True` then something like this could work: ```python table.convert( ""title"", lambda v: {""upper"": {""str"": v.upper()}, ""lower"": {""str"": v.lower()}}, multi=True ) ``` This would result in a `upper` and `lower` column, each containing the JSON string `{""str"": ""UPPERCASE""}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291149509,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291149509,IC_kwDOCGYnMM5M9WDF,9599,simonw,2022-10-25T21:12:11Z,2022-10-25T21:12:11Z,OWNER,"This makes sense to me. There are other places in the codebase where JSON is automatically stringified: https://github.com/simonw/sqlite-utils/blob/c7e4308e6f49d929704163531632e558f9646e4a/sqlite_utils/db.py#L2759-L2766 I don't see why the return value from a convert function shouldn't do the same thing. Since this will result in previous errors working, I don't think it warrants a major version bump either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/494#issuecomment-1258521333,https://api.github.com/repos/simonw/sqlite-utils/issues/494,1258521333,IC_kwDOCGYnMM5LA4L1,9599,simonw,2022-09-26T19:32:36Z,2022-09-26T19:32:36Z,OWNER,Tweeted about it too: https://twitter.com/simonw/status/1574481628507668480,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386593843,Document how to use Just, https://github.com/simonw/sqlite-utils/issues/494#issuecomment-1258516872,https://api.github.com/repos/simonw/sqlite-utils/issues/494,1258516872,IC_kwDOCGYnMM5LA3GI,9599,simonw,2022-09-26T19:28:36Z,2022-09-26T19:28:36Z,OWNER,New documentation: https://sqlite-utils.datasette.io/en/latest/contributing.html#using-just-and-pipenv,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386593843,Document how to use Just, https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1291166273,https://api.github.com/repos/simonw/sqlite-utils/issues/493,1291166273,IC_kwDOCGYnMM5M9aJB,9599,simonw,2022-10-25T21:31:15Z,2022-10-25T21:31:15Z,OWNER,"Based on the docs here I tried the following too: https://docutils.sourceforge.io/docs/user/smartquotes.html#description - `\--` - `\\--` - `\\-\\-` - `\-\-` But none of them had the desired effect in this particular piece of markup: the :ref:`insert \--convert ` I think because this is text inside a `:ref:` block, not regular text. Consider the following: The \--convert and the :ref:`insert \--convert ` and It's rendered like this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386562662,Tiny typographical error in install/uninstall docs, https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1264219650,https://api.github.com/repos/simonw/sqlite-utils/issues/493,1264219650,IC_kwDOCGYnMM5LWnYC,7908073,chapmanjacobd,2022-10-01T03:22:50Z,2022-10-01T03:23:58Z,CONTRIBUTOR,"this is likely what you are looking for: https://stackoverflow.com/a/51076749/697964 but yeah I would say just disable smart quotes","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386562662,Tiny typographical error in install/uninstall docs, https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1258476455,https://api.github.com/repos/simonw/sqlite-utils/issues/493,1258476455,IC_kwDOCGYnMM5LAtOn,9599,simonw,2022-09-26T19:01:49Z,2022-09-26T19:01:49Z,OWNER,"I tried the tips in https://stackoverflow.com/questions/15258831/how-to-handle-two-dashes-in-rest (not the settings change though, because I might want smart quotes elsewhere) and they didn't work. Maybe I should disable smart quotes entirely? I feel like there should be an escaping trick that works here though. I tried `insert -\\-convert` but it didn't help.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386562662,Tiny typographical error in install/uninstall docs, https://github.com/simonw/sqlite-utils/issues/492#issuecomment-1258446128,https://api.github.com/repos/simonw/sqlite-utils/issues/492,1258446128,IC_kwDOCGYnMM5LAl0w,9599,simonw,2022-09-26T18:32:14Z,2022-09-26T18:33:19Z,OWNER,"This idea would make more sense if there was a good mechanism to say ""run the conversion script held in this file"" as opposed to passing it as an option. This would also make having to remember bash escaping rules ([see tip](https://til.simonwillison.net/zsh/argument-heredoc)) much easier! `shot-scraper` has that for `--javascript`, using the `--input` option: https://shot-scraper.datasette.io/en/stable/javascript.html#shot-scraper-javascript-help Maybe `--convert-script` would work here? Or `--convert-file`? It should accept `-` for stdin too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386530156,Idea: ability to pass extra variables to `--convert` scripts, https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1264218914,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1264218914,IC_kwDOCGYnMM5LWnMi,7908073,chapmanjacobd,2022-10-01T03:18:36Z,2023-06-14T22:14:24Z,CONTRIBUTOR,"> some good concrete use-cases in mind I actually found myself wanting something like this the past couple days. The use-case was databases with slightly different schema but same table names. here is a full script: ``` import argparse from pathlib import Path from sqlite_utils import Database def connect(args, conn=None, **kwargs) -> Database: db = Database(conn or args.database, **kwargs) with db.conn: db.conn.execute(""PRAGMA main.cache_size = 8000"") return db def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument(""database"") parser.add_argument(""dbs_folder"") parser.add_argument(""--db"", ""-db"", help=argparse.SUPPRESS) parser.add_argument(""--verbose"", ""-v"", action=""count"", default=0) args = parser.parse_args() if args.db: args.database = args.db Path(args.database).touch() args.db = connect(args) return args def merge_db(args, source_db): source_db = str(Path(source_db).resolve()) s_db = connect(argparse.Namespace(database=source_db, verbose = args.verbose)) for table in s_db.table_names(): data = s_db[table].rows args.db[table].insert_all(data, alter=True, replace=True) args.db.conn.commit() def merge_directory(): args = parse_args() source_dbs = list(Path(args.dbs_folder).glob('*.db')) for s_db in source_dbs: merge_db(args, s_db) if __name__ == '__main__': merge_directory() ``` edit: I've made some improvements to this and put it on PyPI: ``` $ pip install xklb $ lb merge-db -h usage: library merge-dbs DEST_DB SOURCE_DB ... [--only-target-columns] [--only-new-rows] [--upsert] [--pk PK ...] [--table TABLE ...] Merge-DBs will insert new rows from source dbs to target db, table by table. If primary key(s) are provided, and there is an existing row with the same PK, the default action is to delete the existing row and insert the new row replacing all existing fields. Upsert mode will update matching PK rows such that if a source row has a NULL field and the destination row has a value then the value will be preserved instead of changed to the source row's NULL value. Ignore mode (--only-new-rows) will insert only rows which don't already exist in the destination db Test first by using temp databases as the destination db. Try out different modes / flags until you are satisfied with the behavior of the program library merge-dbs --pk path (mktemp --suffix .db) tv.db movies.db Merge database data and tables library merge-dbs --upsert --pk path video.db tv.db movies.db library merge-dbs --only-target-columns --only-new-rows --table media,playlists --pk path audio-fts.db audio.db library merge-dbs --pk id --only-tables subreddits reddit/81_New_Music.db audio.db library merge-dbs --only-new-rows --pk subreddit,path --only-tables reddit_posts reddit/81_New_Music.db audio.db -v positional arguments: database source_dbs ``` Also if you want to dedupe a table based on a ""business key"" which isn't explicitly your primary key(s) you can run this: ``` $ lb dedupe-db -h usage: library dedupe-dbs DATABASE TABLE --bk BUSINESS_KEYS [--pk PRIMARY_KEYS] [--only-columns COLUMNS] Dedupe your database (not to be confused with the dedupe subcommand) It should not need to be said but *backup* your database before trying this tool! Dedupe-DB will help remove duplicate rows based on non-primary-key business keys library dedupe-db ./video.db media --bk path If --primary-keys is not provided table metadata primary keys will be used If --only-columns is not provided all non-primary and non-business key columns will be upserted positional arguments: database table options: -h, --help show this help message and exit --skip-0 --only-columns ONLY_COLUMNS Comma separated column names to upsert --primary-keys PRIMARY_KEYS, --pk PRIMARY_KEYS Comma separated primary keys --business-keys BUSINESS_KEYS, --bk BUSINESS_KEYS Comma separated business keys ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables, https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258712931,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1258712931,IC_kwDOCGYnMM5LBm9j,25778,eyeseast,2022-09-26T22:31:58Z,2022-09-26T22:31:58Z,CONTRIBUTOR,"Right. The backup command will copy tables completely, but in the case of conflicting table names, the destination gets overwritten silently. That might not be what you want here. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables, https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258697384,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1258697384,IC_kwDOCGYnMM5LBjKo,9599,simonw,2022-09-26T22:12:45Z,2022-09-26T22:12:45Z,OWNER,That feels like a slightly different command to me - maybe `sqlite-utils backup data.db data-backup.db`? It doesn't have any of the mechanics for merging tables together. Could be a useful feature separately though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables, https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258508215,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1258508215,IC_kwDOCGYnMM5LA0-3,25778,eyeseast,2022-09-26T19:22:14Z,2022-09-26T19:22:14Z,CONTRIBUTOR,"This might be fairly straightforward using SQLite's backup utility: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.backup ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables, https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258450447,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1258450447,IC_kwDOCGYnMM5LAm4P,9599,simonw,2022-09-26T18:36:23Z,2022-09-26T18:36:23Z,OWNER,This is also the kind of feature that would need to express itself in both the Python library and the CLI utility.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables, https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258449887,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1258449887,IC_kwDOCGYnMM5LAmvf,9599,simonw,2022-09-26T18:35:50Z,2022-09-26T18:35:50Z,OWNER,"This is a really interesting idea. I'm nervous about needing to set the rules for how duplicate tables should be merged though. This feels like a complex topic - one where there isn't necessarily an obviously ""correct"" way of doing it, but where different problems that people are solving might need different merging approaches. Likewise, merging isn't just a database-to-database thing at that point - I could see a need for merging two tables using similar rules to those used for merging two databases. So I think I'd want to have some good concrete use-cases in mind before trying to design how something like this should work. Will leave this thread open for people to drop those in!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables, https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1256858763,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1256858763,IC_kwDOCGYnMM5K6iSL,7908073,chapmanjacobd,2022-09-24T04:50:59Z,2022-09-24T04:52:08Z,CONTRIBUTOR,"Instead of outputting binary data to stdout the interface might be better like this ``` sqlite-utils merge animals.db cats.db dogs.db ``` similar to `zip`, `ogr2ogr`, etc Actually I think this might already be possible within `ogr2ogr`. I don't believe spatial data is a requirement though it might add an `ogc_id` column or something ``` cp cats.db animals.db ogr2ogr -append animals.db dogs.db ogr2ogr -append animals.db another.db ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables, https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1258437060,https://api.github.com/repos/simonw/sqlite-utils/issues/490,1258437060,IC_kwDOCGYnMM5LAjnE,9599,simonw,2022-09-26T18:24:44Z,2022-09-26T18:24:44Z,OWNER,Just saw your great write-up on this: https://jeqo.github.io/notes/2022-09-24-ingest-logs-sqlite/,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1382457780,Ability to insert multi-line files, https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1257072258,https://api.github.com/repos/simonw/sqlite-utils/issues/490,1257072258,IC_kwDOCGYnMM5K7WaC,6180701,jeqo,2022-09-24T22:01:05Z,2022-09-24T22:01:05Z,NONE,"For completeness, the regex requires a bit more dark magic to capture the following lines, here is a _working_ expression: https://regex101.com/r/rsuEcs/1 ``` sqlite-utils insert /tmp/log.db log multiline.log --text --convert "" import re r = re.compile(r'^(?P\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})(?:\:\s)(?P(.*\s\s.*|.*)+)', re.MULTILINE) def convert(text): return [m.groupdict() for m in r.finditer(text)] "" ``` ``` BEGIN TRANSACTION; CREATE TABLE [log] ( [datetime] TEXT, [log] TEXT ); INSERT INTO ""log"" VALUES('2022-03-01T12:04:52','Here is a log message that spans multiple lines'); INSERT INTO ""log"" VALUES('2022-03-01T12:04:52','This is a single line'); INSERT INTO ""log"" VALUES('2022-03-01T12:04:52','Here is another message that spans multiple lines'); COMMIT; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1382457780,Ability to insert multi-line files, https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1257063174,https://api.github.com/repos/simonw/sqlite-utils/issues/490,1257063174,IC_kwDOCGYnMM5K7UMG,6180701,jeqo,2022-09-24T20:50:15Z,2022-09-24T20:50:15Z,NONE,🤯 this is beautiful. Thanks @simonw !,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1382457780,Ability to insert multi-line files, https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1256428818,https://api.github.com/repos/simonw/sqlite-utils/issues/490,1256428818,IC_kwDOCGYnMM5K45US,9599,simonw,2022-09-23T16:37:58Z,2022-09-23T16:38:35Z,OWNER,"It should be possible to achieve this with the `--text` option: https://sqlite-utils.datasette.io/en/stable/cli.html?highlight=text#convert-with-text Given an example like this in `multiline.log`: ``` 2022-03-01T12:04:52: Here is a log message that spans multiple lines 2022-03-01T12:04:52: This is a single line 2022-03-01T12:04:52: Here is another message that spans multiple lines ``` You should be able to run something like this: ``` sqlite-utils insert /tmp/log.db log multiline.log --text --convert "" import re r = re.compile(r'^(?P\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}):(?P.*)', re.MULTILINE) def convert(text): return [m.groupdict() for m in r.finditer(text)] "" ``` After running this I get: ``` sqlite-utils rows /tmp/log.db log [{""datetime"": ""2022-03-01T12:04:52"", ""log"": "" Here is a log message""}, {""datetime"": ""2022-03-01T12:04:52"", ""log"": "" This is a single line""}, {""datetime"": ""2022-03-01T12:04:52"", ""log"": "" Here is another message""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1382457780,Ability to insert multi-line files, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710461468,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710461468,MDEyOklzc3VlQ29tbWVudDcxMDQ2MTQ2OA==,9599,simonw,2020-10-16T19:18:19Z,2020-10-16T19:18:19Z,OWNER,"Reconsidering: #89 was a feature request that relates to this, so maybe this is worth implementing after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710397574,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710397574,MDEyOklzc3VlQ29tbWVudDcxMDM5NzU3NA==,9599,simonw,2020-10-16T18:38:21Z,2020-10-16T18:38:21Z,OWNER,"I'm not going to implement this. I'll leave `extract=...` as it is right now, suitable for quick simple single-column operations on input, but if users want to do something more complicated involving multiple columns they should use the `table.extract()` method after the initial insert instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710395444,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710395444,MDEyOklzc3VlQ29tbWVudDcxMDM5NTQ0NA==,9599,simonw,2020-10-16T18:37:10Z,2020-10-16T18:37:10Z,OWNER,"But this begins to feel too complicated, given that `table.extract()` can already be used to achieve the same thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710393550,MDEyOklzc3VlQ29tbWVudDcxMDM5MzU1MA==,9599,simonw,2020-10-16T18:35:57Z,2020-10-16T18:36:39Z,OWNER,"If I want to support that most complicated example, I think the option to pass a `Extracts()` object to `extracts=` is the best way to do it: ```python fresh_db.table(""tree"", extracts=[Extract( columns=(""CommonName"", ""LatinName""), table=""Species"", fk_column=""species_id"", rename={""CommonName"": ""name"", ""LatinName"": ""latin""} )]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710390915,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710390915,MDEyOklzc3VlQ29tbWVudDcxMDM5MDkxNQ==,9599,simonw,2020-10-16T18:34:26Z,2020-10-16T18:34:50Z,OWNER,"Here's the most complex example of `.extracts()`: ```python db[""Trees""].extract( [""CommonName"", ""LatinName""], table=""Species"", fk_column=""species_id"", rename={""CommonName"": ""name"", ""LatinName"": ""latin""} ) ``` Resulting in: ```sql CREATE TABLE [Species] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [latin] TEXT ) ``` From https://sqlite-utils.readthedocs.io/en/stable/python-api.html#extracting-columns-into-a-separate-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710364942,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710364942,MDEyOklzc3VlQ29tbWVudDcxMDM2NDk0Mg==,9599,simonw,2020-10-16T18:18:48Z,2020-10-16T18:18:48Z,OWNER,"I think there is. It's a nice existing feature, and I don't think adding tuple support to it would be a huge lift.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710363789,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710363789,MDEyOklzc3VlQ29tbWVudDcxMDM2Mzc4OQ==,9599,simonw,2020-10-16T18:18:05Z,2020-10-16T18:18:05Z,OWNER,I wonder if there's value in extending the `extracts=` option at all given the existence of `table.extract()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710359724,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710359724,MDEyOklzc3VlQ29tbWVudDcxMDM1OTcyNA==,9599,simonw,2020-10-16T18:15:31Z,2020-10-16T18:15:31Z,OWNER,"Using a tuple would work: ```python fresh_db.table(""tree"", extracts=[(""common_name"", ""latin_name"")]) ``` Or to define a custom name: ```python fresh_db.table(""tree"", extracts={(""common_name"", ""latin_name""): ""names""}) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710346830,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710346830,MDEyOklzc3VlQ29tbWVudDcxMDM0NjgzMA==,9599,simonw,2020-10-16T18:08:52Z,2020-10-16T18:09:21Z,OWNER,"The new `.extract()` method can handle multiple columns: https://github.com/simonw/sqlite-utils/blob/2c541fac352632e23e40b0d21e3f233f7a744a57/tests/test_extract.py#L70-L87","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-514509307,https://api.github.com/repos/simonw/sqlite-utils/issues/49,514509307,MDEyOklzc3VlQ29tbWVudDUxNDUwOTMwNw==,9599,simonw,2019-07-24T07:09:43Z,2019-07-24T07:10:21Z,OWNER,"This syntax should be shared with #42 as much as possible. Maybe something based on a namedtuple would work, since those are already used in the library. ```python workouts = db.table(""workouts"", extracts=[Extract( columns=[""source"", ""source_version""], table=""Sources"" )]) ``` Since namedtuples cannot have default values this should probably be a class instead. Actually it looks like there is a trick for defaults here: https://stackoverflow.com/a/18348004","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248621072,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248621072,IC_kwDOCGYnMM5KbHIQ,9599,simonw,2022-09-15T20:56:09Z,2022-09-15T20:56:09Z,OWNER,"Prototype so far: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 767b170..d96c507 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -1762,6 +1762,17 @@ def query( is_flag=True, help=""Analyze resulting tables and output results"", ) +@click.option(""--key"", help=""read data from this key of the root object"") +@click.option( + ""--auto-key"", + is_flag=True, + help=""Find a key in the root object that is a list of objects"", +) +@click.option( + ""--analyze"", + is_flag=True, + help=""Analyze resulting tables and output results"", +) @load_extension_option def memory( paths, @@ -1784,6 +1795,8 @@ def memory( schema, dump, save, + key, + auto_key, analyze, load_extension, ): @@ -1838,7 +1851,9 @@ def memory( csv_table = stem stem_counts[stem] = stem_counts.get(stem, 1) + 1 csv_fp = csv_path.open(""rb"") - rows, format_used = rows_from_file(csv_fp, format=format, encoding=encoding) + rows, format_used = rows_from_file( + csv_fp, format=format, encoding=encoding, key=key, auto_key=auto_key + ) tracker = None if format_used in (Format.CSV, Format.TSV) and not no_detect_types: tracker = TypeTracker() diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py index 8754554..2e69c26 100644 --- a/sqlite_utils/utils.py +++ b/sqlite_utils/utils.py @@ -231,6 +231,8 @@ def rows_from_file( encoding: Optional[str] = None, ignore_extras: Optional[bool] = False, extras_key: Optional[str] = None, + key: Optional[str] = None, + auto_key: Optional[bool] = False, ) -> Tuple[Iterable[dict], Format]: """""" Load a sequence of dictionaries from a file-like object containing one of four different formats. @@ -271,13 +273,31 @@ def rows_from_file( :param encoding: the character encoding to use when reading CSV/TSV data :param ignore_extras: ignore any extra fields on rows :param extras_key: put any extra fields in a list with this key + :param key: read data from this key of the root object + :param auto_key: find a key in the root object that is a list of objects """""" if ignore_extras and extras_key: raise ValueError(""Cannot use ignore_extras= and extras_key= together"") + if key and auto_key: + raise ValueError(""Cannot use key= and auto_key= together"") if format == Format.JSON: decoded = json.load(fp) if isinstance(decoded, dict): - decoded = [decoded] + if auto_key: + list_keys = [ + k + for k in decoded + if isinstance(decoded[k], list) + and decoded[k] + and all(isinstance(o, dict) for o in decoded[k]) + ] + if len(list_keys) == 1: + decoded = decoded[list_keys[0]] + elif key: + # Raises KeyError, I think that's OK + decoded = decoded[key] + if not isinstance(decoded, list): + decoded = [decoded] if not isinstance(decoded, list): raise RowsFromFileBadJSON(""JSON must be a list or a dictionary"") return decoded, Format.JSON @@ -305,7 +325,9 @@ def rows_from_file( first_bytes = buffered.peek(2048).strip() if first_bytes.startswith(b""["") or first_bytes.startswith(b""{""): # TODO: Detect newline-JSON - return rows_from_file(buffered, format=Format.JSON) + return rows_from_file( + buffered, format=Format.JSON, key=key, auto_key=auto_key + ) else: dialect = csv.Sniffer().sniff( first_bytes.decode(encoding or ""utf-8-sig"", ""ignore"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248522618,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248522618,IC_kwDOCGYnMM5KavF6,9599,simonw,2022-09-15T19:29:20Z,2022-09-15T19:29:20Z,OWNER,I think refactoring `sqlite-utils insert` to use `rows_from_file` needs to happen as part of this work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248512739,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248512739,IC_kwDOCGYnMM5Kasrj,9599,simonw,2022-09-15T19:18:24Z,2022-09-15T19:21:01Z,OWNER,"Why doesn't `sqlite-utils insert` use the `rows_from_file` function I wonder? https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841 says: > I can refactor `sqlite-utils insert` to use this new code too. Maybe I forgot to do that?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248501824,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248501824,IC_kwDOCGYnMM5KaqBA,9599,simonw,2022-09-15T19:10:48Z,2022-09-15T19:10:48Z,OWNER,"This feels pretty good: ``` % sqlite-utils memory ~/Downloads/CVR_Export_20220908084311/*.json --schema --auto-key CREATE TABLE [BallotTypeContestManifest] ( [BallotTypeId] INTEGER, [ContestId] INTEGER ); CREATE VIEW t1 AS select * from [BallotTypeContestManifest]; CREATE VIEW t AS select * from [BallotTypeContestManifest]; CREATE TABLE [BallotTypeManifest] ( [Description] TEXT, [Id] INTEGER, [ExternalId] TEXT ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248484094,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248484094,IC_kwDOCGYnMM5Kalr-,9599,simonw,2022-09-15T18:56:31Z,2022-09-15T18:56:31Z,OWNER,"Actually I quite like `--key X` - it could work for single nested objects too. You could insert a single record like this: ```json { ""record"" { ""id"": 1 } } ``` ``` sqlite-utils insert db.db records record.json --key record ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248481303,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248481303,IC_kwDOCGYnMM5KalAX,9599,simonw,2022-09-15T18:54:30Z,2022-09-15T18:55:14Z,OWNER,"Maybe this would make more sense as a mechanism where you can say ""Use the data in the key called X"" - but there's a special option for ""figure out that key automatically"". The syntax then could be: `--list-key List` Or for automatic detection: `--list-key-auto` Could also go with `--key List` and `--key-auto` - but would that be as obvious as `--list-key`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248479485,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248479485,IC_kwDOCGYnMM5Kakj9,9599,simonw,2022-09-15T18:52:52Z,2022-09-15T18:53:45Z,OWNER,"The most similar option I have at the moment is probably `--flatten`. What would good names for this option be? - `--auto-list` - `--auto-key` - `--inner-key` - `--auto-json` - `--find-list` - `--find-key` Those are all bad. Another option: introduce a new explicit format for it. Right now the explicit formats you can use are: https://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/docs/cli-reference.rst#L153-L158 So I could add a `:autojson` format.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248475718,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248475718,IC_kwDOCGYnMM5KajpG,9599,simonw,2022-09-15T18:49:05Z,2022-09-15T18:49:53Z,OWNER,"Here's how I used my prototype to build [that Gist](https://gist.github.com/simonw/0e6901974a14ab7d56c2746a04d72c8c): sqlite-utils memory ~/Downloads/CVR_Export_20220908084311/*.json --schema > database.sql ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248474806,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248474806,IC_kwDOCGYnMM5Kaja2,9599,simonw,2022-09-15T18:48:09Z,2022-09-15T18:48:09Z,OWNER,"Built a prototype of this that works really well: ```diff diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py index c0b7bf1..f9a482c 100644 --- a/sqlite_utils/utils.py +++ b/sqlite_utils/utils.py @@ -272,7 +272,19 @@ def rows_from_file( if format == Format.JSON: decoded = json.load(fp) if isinstance(decoded, dict): - decoded = [decoded] + # TODO: Solve for if this isn't what people want + # Does it have just one key that is a list of dicts? + list_keys = [ + k + for k in decoded + if isinstance(decoded[k], list) + and decoded[k] + and all(isinstance(o, dict) for o in decoded[k]) + ] + if len(list_keys) == 1: + decoded = decoded[list_keys[0]] + else: + decoded = [decoded] if not isinstance(decoded, list): raise RowsFromFileBadJSON(""JSON must be a list or a dictionary"") return decoded, Format.JSON ``` I used that to build this: https://gist.github.com/simonw/0e6901974a14ab7d56c2746a04d72c8c One problem though: right now, if you do this `sqlite-utils` treats it as a single object and adds a `tags` column with JSON in it: ``` echo '{""title"": ""Hi"", ""tags"": [{""t"": ""one""}]}` | sqlite-utils insert db.db t - ``` If I implement this new mechanism the above line would behave differently - which would be a backwards incompatible change. So I probably need some kind of opt-in mechanism for this. And I need a good name for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1364141224,https://api.github.com/repos/simonw/sqlite-utils/issues/488,1364141224,IC_kwDOCGYnMM5RTySo,1176293,ar-jan,2022-12-23T17:38:55Z,2022-12-23T17:38:55Z,NONE,"> text columns containing empty strings should not be rewritten to null. I would actually appreciate an option to do just that for text columns as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373224657,`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float, https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1254033981,https://api.github.com/repos/simonw/sqlite-utils/issues/488,1254033981,IC_kwDOCGYnMM5Kvwo9,9599,simonw,2022-09-21T17:49:32Z,2022-09-21T17:50:10Z,OWNER,"It looks like SQLite has a `SELECT NULLIF(value, '')` function which returns `null` if that value is equal to `''`. We need to only apply that function to columns that we know to be of type integer or float though - text columns containing empty strings should not be rewritten to null.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373224657,`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float, https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1254032378,https://api.github.com/repos/simonw/sqlite-utils/issues/488,1254032378,IC_kwDOCGYnMM5KvwP6,9599,simonw,2022-09-21T17:47:54Z,2022-09-21T17:47:54Z,OWNER,"New tests should go in: https://github.com/simonw/sqlite-utils/blob/main/tests/test_transform.py I think the implementation fix needs to go near here: https://github.com/simonw/sqlite-utils/blob/0b315d3fa83c1584eaeec32f24912898621e437a/sqlite_utils/db.py#L1770-L1775 The trick is going to be teaching that generated SQL to know which columns are `integer` or `float` and to convert `""""` to `null` as part of that operation. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373224657,`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float, https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1254029808,https://api.github.com/repos/simonw/sqlite-utils/issues/488,1254029808,IC_kwDOCGYnMM5Kvvnw,9599,simonw,2022-09-21T17:45:20Z,2022-09-21T17:45:41Z,OWNER,"No, I'm going to say that this is a bug - it's WEIRD having a `integer` or `float` column containing an empty string. I'm OK changing that - I very much doubt anyone is relying on this functionality. So no need for a new option here - just fixing the bug is sensible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373224657,`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float, https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1246971764,https://api.github.com/repos/simonw/sqlite-utils/issues/488,1246971764,IC_kwDOCGYnMM5KU0d0,9599,simonw,2022-09-14T15:52:14Z,2022-09-14T15:52:14Z,OWNER,"Frustratingly I think this counts as a backwards-incompatible change. Could have it be opt-in with a new option / method parameter, and then change it to the default if I release a `sqlite-utils 4`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373224657,`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float, https://github.com/simonw/sqlite-utils/issues/487#issuecomment-1242513223,https://api.github.com/repos/simonw/sqlite-utils/issues/487,1242513223,IC_kwDOCGYnMM5KDz9H,540968,ryanfox,2022-09-09T22:01:10Z,2022-09-09T22:01:10Z,NONE,"Perfect. Apologies, I searched but didn’t see that one for some reason. Thank you. On Fri, Sep 9, 2022 at 3:04 PM Simon Willison ***@***.***> wrote: > This isn't supported yet - there's an older issue for that here: > > - #117 > > — > Reply to this email directly, view it on GitHub > , > or unsubscribe > > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1367835380,Specify foreign key against compound key in other table, https://github.com/simonw/sqlite-utils/issues/487#issuecomment-1242409766,https://api.github.com/repos/simonw/sqlite-utils/issues/487,1242409766,IC_kwDOCGYnMM5KDasm,9599,simonw,2022-09-09T20:04:30Z,2022-09-09T20:04:30Z,OWNER,"This isn't supported yet - there's an older issue for that here: - #117 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1367835380,Specify foreign key against compound key in other table, https://github.com/simonw/sqlite-utils/issues/485#issuecomment-1248597643,https://api.github.com/repos/simonw/sqlite-utils/issues/485,1248597643,IC_kwDOCGYnMM5KbBaL,9599,simonw,2022-09-15T20:39:39Z,2022-09-15T20:39:52Z,OWNER,"A note from PR #486: https://github.com/simonw/sqlite-utils/issues/486#issuecomment-1248591268_ > ``` > sqlite-utils insert /tmp/t3.db t /tmp/big.json > [####################################] 100% > ``` > This is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%. I decided to land this anyway. If a streaming JSON parser is added later it will start to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366423176,Progressbar not shown when inserting/upserting jsonlines file, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239772256,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239772256,IC_kwDOCGYnMM5J5Wxg,9599,simonw,2022-09-07T19:07:51Z,2022-09-07T19:09:52Z,OWNER,"Or... I could automatically register multiple copies of the function of different arities! If I'm going to do something like that though I need to think carefully about how functions that have keyword-only arguments should work: https://peps.python.org/pep-3102/ ```python def compare(a, b, *ignore, key=None): ... ``` I should think about how these work with `db.register_function()` anyway, since SQL functions cannot support keyword arguments.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239766987,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239766987,IC_kwDOCGYnMM5J5VfL,9599,simonw,2022-09-07T19:01:49Z,2022-09-07T19:01:49Z,OWNER,"OK with this: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index c51b101..93d82a9 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -30,6 +30,7 @@ from .utils import ( Format, TypeTracker, ) +from . import recipes CONTEXT_SETTINGS = dict(help_option_names=[""-h"", ""--help""]) @@ -3029,7 +3030,7 @@ def _load_extensions(db, load_extension): def _register_functions(db, functions): # Register any Python functions as SQL functions: sqlite3.enable_callback_tracebacks(True) - globals = {} + globals = {""r"": recipes, ""recipes"": recipes} try: exec(functions, globals) except SyntaxError as ex: @@ -3037,4 +3038,4 @@ def _register_functions(db, functions): # Register all callables in the locals dict: for name, value in globals.items(): if callable(value) and not name.startswith(""_""): - db.register_function(value, name=name) + db.register_function(value, name=name, ignore_defaults=True) diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 27c46b0..1407d23 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -370,6 +370,7 @@ class Database: self, fn: Callable = None, deterministic: bool = False, + ignore_defaults: bool = False, replace: bool = False, name: Optional[str] = None, ): @@ -397,7 +398,10 @@ class Database: def register(fn): fn_name = name or fn.__name__ - arity = len(inspect.signature(fn).parameters) + params = inspect.signature(fn).parameters + if ignore_defaults: + params = [p for p in params if params[p].default is inspect._empty] + arity = len(params) if not replace and (fn_name, arity) in self._registered_functions: return fn kwargs = {} ``` I can now do this: ``` % sqlite-utils memory --functions 'parsedate = r.parsedate' 'select parsedate(""1st jan"")' [{""parsedate(\""1st jan\"")"": ""2022-01-01""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239763997,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239763997,IC_kwDOCGYnMM5J5Uwd,9599,simonw,2022-09-07T18:57:59Z,2022-09-07T18:58:10Z,OWNER,"Here's how to detect defaults in the function signature: ```pycon >>> import inspect >>> def foo(a, b, c=1, d=2): ... pass ... >>> inspect.signature(foo) >>> inspect.signature(foo).parameters mappingproxy(OrderedDict([('a', ), ('b', ), ('c', ), ('d', )])) >>> inspect.signature(foo).parameters['c'] >>> dir(inspect.signature(foo).parameters['c']) ['KEYWORD_ONLY', 'POSITIONAL_ONLY', 'POSITIONAL_OR_KEYWORD', 'VAR_KEYWORD', 'VAR_POSITIONAL', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_annotation', '_default', '_kind', '_name', 'annotation', 'default', 'empty', 'kind', 'name', 'replace'] >>> inspect.signature(foo).parameters['c'].default 1 >>> inspect.signature(foo).parameters['a'].default ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239762561,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239762561,IC_kwDOCGYnMM5J5UaB,9599,simonw,2022-09-07T18:56:13Z,2022-09-07T18:56:13Z,OWNER,"I could do this: ```python # Register all callables in the locals dict: for name, value in globals.items(): if callable(value) and not name.startswith(""_""): db.register_function(value, name=name, ignore_params_with_defaults=True) ``` Introducing a new `ignore_params_with_defaults` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239762031,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239762031,IC_kwDOCGYnMM5J5URv,9599,simonw,2022-09-07T18:55:30Z,2022-09-07T18:55:30Z,OWNER,That would be a breaking change though - existing code that registers functions with default parameters should continue to work unchanged (unless I want to ship `sqlite-utils` 4.0).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239761280,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239761280,IC_kwDOCGYnMM5J5UGA,9599,simonw,2022-09-07T18:54:51Z,2022-09-07T18:54:51Z,OWNER,"I could teach this code here to only register the function using arguments that don't have default parameters: https://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/sqlite_utils/cli.py#L3037-L3040 Or even this code here: https://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/sqlite_utils/db.py#L398-L418 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239760001,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239760001,IC_kwDOCGYnMM5J5TyB,9599,simonw,2022-09-07T18:53:17Z,2022-09-07T18:53:17Z,OWNER,"So you would need to do this instead: ``` sqlite-utils memory 'select parsedate(""1st jan"")' --functions ' def parsedate(s): return r.parsedate(s) ' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239759022,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239759022,IC_kwDOCGYnMM5J5Tiu,9599,simonw,2022-09-07T18:52:08Z,2022-09-07T18:52:08Z,OWNER,"It's not quite that simple. I tried applying this patch: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index c51b101..33e4d90 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -30,6 +30,7 @@ from .utils import ( Format, TypeTracker, ) +from . import recipes CONTEXT_SETTINGS = dict(help_option_names=[""-h"", ""--help""]) @@ -3029,7 +3030,7 @@ def _load_extensions(db, load_extension): def _register_functions(db, functions): # Register any Python functions as SQL functions: sqlite3.enable_callback_tracebacks(True) - globals = {} + globals = {""r"": recipes, ""recipes"": recipes} try: exec(functions, globals) except SyntaxError as ex: ``` Then got this: ``` % sqlite-utils memory --functions 'parsedate = r.parsedate' 'select parsedate(""1st jan"")' Error: wrong number of arguments to function parsedate() % sqlite-utils memory --functions 'parsedate = r.parsedate' 'select parsedate(""1st jan"", 0, 0, 0)' [{""parsedate(\""1st jan\"", 0, 0, 0)"": ""2022-01-01""}] ``` The problem here is that the `parsedate` function signature looks like this: https://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/sqlite_utils/recipes.py#L8 But the code that register SQL functions introspects that signature, so creates a SQL function that requires four arguments.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239699276,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239699276,IC_kwDOCGYnMM5J5E9M,9599,simonw,2022-09-07T17:49:49Z,2022-09-07T17:49:49Z,OWNER,"This feature is a tiny bit weird though: the recipe functions are not exposed to SQL by default, they are instead designed to be used with `sqlite-utils convert`. I guess with `--functions` support you could do something like this: sqlite-utils data.db ""update mytable set col1 = parsedate(col1)"" --functions ""parsedate = r.parsedate""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239697643,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239697643,IC_kwDOCGYnMM5J5Ejr,9599,simonw,2022-09-07T17:48:00Z,2022-09-07T17:48:00Z,OWNER,Will also need to update documentation here: https://sqlite-utils.datasette.io/en/stable/cli.html#defining-custom-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1238607591,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1238607591,IC_kwDOCGYnMM5J06bn,9599,simonw,2022-09-06T20:16:39Z,2022-09-06T20:16:39Z,OWNER,"Here's the implementation for recipes at the moment: https://github.com/simonw/sqlite-utils/blob/5b969273f1244b1bcf3e4dc071cdf17dab35d5f8/sqlite_utils/utils.py#L434-L441 And here's the `--functions` implementation that doesn't expose them: https://github.com/simonw/sqlite-utils/blob/5b969273f1244b1bcf3e4dc071cdf17dab35d5f8/sqlite_utils/cli.py#L3029-L3040","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/483#issuecomment-1258479462,https://api.github.com/repos/simonw/sqlite-utils/issues/483,1258479462,IC_kwDOCGYnMM5LAt9m,9599,simonw,2022-09-26T19:04:29Z,2022-09-26T19:04:43Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-install - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-uninstall - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#install - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#uninstall ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363765916,`sqlite-utils install` command, https://github.com/simonw/sqlite-utils/issues/483#issuecomment-1258451968,https://api.github.com/repos/simonw/sqlite-utils/issues/483,1258451968,IC_kwDOCGYnMM5LAnQA,9599,simonw,2022-09-26T18:37:54Z,2022-09-26T18:40:41Z,OWNER,"The implementation of this can be an almost exact copy of Datasette's, which was added in this commit: https://github.com/simonw/datasette/commit/01fe5b740171bfaea3752fc5754431dac53777e3 Current code for that is here: https://github.com/simonw/datasette/blob/0.62/datasette/cli.py#L319-L340 - which is improved to use the `from runpy import run_module` function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363765916,`sqlite-utils install` command, https://github.com/simonw/sqlite-utils/issues/482#issuecomment-1236511153,https://api.github.com/repos/simonw/sqlite-utils/issues/482,1236511153,IC_kwDOCGYnMM5Js6mx,9599,simonw,2022-09-05T03:38:29Z,2022-09-05T03:38:29Z,OWNER,"This is an interesting research area! Running the same experiment as the one in https://www.cybertec-postgresql.com/en/column-order-in-postgresql-does-matter/ could demonstrate one way or another if column ordering makes a difference with regards to performance or not. My hunch is that even if it does this wouldn't be a feature to bake into `sqlite-utils` itself, beyond the existing features for changing the order of columns in a table (https://sqlite-utils.datasette.io/en/stable/python-api.html#changing-column-order).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1361355564,balanced table default column_order, https://github.com/simonw/sqlite-utils/issues/48#issuecomment-710402331,https://api.github.com/repos/simonw/sqlite-utils/issues/48,710402331,MDEyOklzc3VlQ29tbWVudDcxMDQwMjMzMQ==,9599,simonw,2020-10-16T18:41:06Z,2020-10-16T18:41:06Z,OWNER,I could use this demo from JupyterCon 2020 https://gist.github.com/simonw/656c21b5800d5e4624dec9930f00e093,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471818939,"Jupyter notebook demo of the library, launchable on Binder", https://github.com/simonw/sqlite-utils/issues/48#issuecomment-1021790707,https://api.github.com/repos/simonw/sqlite-utils/issues/48,1021790707,IC_kwDOCGYnMM4850nz,9599,simonw,2022-01-26T02:08:39Z,2022-01-26T02:08:39Z,OWNER,This is a dupe of #308.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471818939,"Jupyter notebook demo of the library, launchable on Binder", https://github.com/simonw/sqlite-utils/issues/478#issuecomment-1229563718,https://api.github.com/repos/simonw/sqlite-utils/issues/478,1229563718,IC_kwDOCGYnMM5JSadG,9599,simonw,2022-08-28T22:07:03Z,2022-08-28T22:07:03Z,OWNER,"What should happen if you run this: sqlite-utils tables data.db bad_table Where `bad_table` is a table that does not exist? I think it should error rather than failing silently. That way you can use the error code to confirm that a list of expected tables exist.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353481513,`sqlite-utils tables data.db table1 table2`, https://github.com/simonw/sqlite-utils/issues/477#issuecomment-1238815924,https://api.github.com/repos/simonw/sqlite-utils/issues/477,1238815924,IC_kwDOCGYnMM5J1tS0,49702524,thewchan,2022-09-07T01:46:24Z,2022-09-07T01:46:24Z,NONE,Will do!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353441389,Conda Forge, https://github.com/simonw/sqlite-utils/issues/477#issuecomment-1238608921,https://api.github.com/repos/simonw/sqlite-utils/issues/477,1238608921,IC_kwDOCGYnMM5J06wZ,9599,simonw,2022-09-06T20:18:12Z,2022-09-06T20:18:12Z,OWNER,"Hi @thewchan - thanks for that! Yes I'm interested, can you invite `swillison` @ Google's email provider please?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353441389,Conda Forge, https://github.com/simonw/sqlite-utils/issues/476#issuecomment-1229370823,https://api.github.com/repos/simonw/sqlite-utils/issues/476,1229370823,IC_kwDOCGYnMM5JRrXH,9599,simonw,2022-08-28T04:07:15Z,2022-08-28T04:07:15Z,OWNER,https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-29,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353196970,Release notes for 3.29, https://github.com/simonw/sqlite-utils/issues/476#issuecomment-1229320114,https://api.github.com/repos/simonw/sqlite-utils/issues/476,1229320114,IC_kwDOCGYnMM5JRe-y,9599,simonw,2022-08-27T23:26:48Z,2022-08-27T23:26:48Z,OWNER,"- #458 - the `register_function(name=...)` argument - New tutorial: [Cleaning data with sqlite-utils and Datasette](https://datasette.io/tutorials/clean-data) provides a tutorial introduction (and accompanying ten minute video) about using this tool. - New Discord community, https://discord.gg/Ass7bCAMDw - #469 `sqlite-utils rows --order` option - #471 `sqlite-utils query --functions` option - #472 Improved code compilation pattern - #473 Support entrypoints for `--load-extension` - #455 - #475 `table.default_values` property - #467","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353196970,Release notes for 3.29, https://github.com/simonw/sqlite-utils/issues/475#issuecomment-1229283063,https://api.github.com/repos/simonw/sqlite-utils/issues/475,1229283063,IC_kwDOCGYnMM5JRV73,9599,simonw,2022-08-27T22:44:45Z,2022-08-27T22:44:45Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#default-values - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.default_values","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353189941,table.default_values introspection property, https://github.com/simonw/sqlite-utils/issues/474#issuecomment-1232424050,https://api.github.com/repos/simonw/sqlite-utils/issues/474,1232424050,IC_kwDOCGYnMM5JdUxy,9599,simonw,2022-08-31T03:42:28Z,2022-08-31T03:42:28Z,OWNER,"Yeah, having a way of inserting a CSV file but saying ""only I want the `title` column to come first"" does make sense to me as a feature. It can use `-o` for consistency with https://sqlite-utils.datasette.io/en/stable/cli-reference.html#search and https://sqlite-utils.datasette.io/en/stable/cli-reference.html#rows and https://sqlite-utils.datasette.io/en/stable/cli-reference.html#transform","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353074021,Add an option for specifying column names when inserting CSV data, https://github.com/simonw/sqlite-utils/issues/474#issuecomment-1229449018,https://api.github.com/repos/simonw/sqlite-utils/issues/474,1229449018,IC_kwDOCGYnMM5JR-c6,14294,hubgit,2022-08-28T12:40:13Z,2022-08-28T12:40:13Z,NONE,"Creating the table before inserting is a useful workaround, thanks. It does require figuring out the `create table` syntax and listing all the fields manually, though, which loses some of the magic of sqlite-utils. I was expecting to find an option like `--headers=foo,bar` (or `--header-row='foo\tbar'`, if that would be easier) - not necessarily that exact syntax, but something that would essentially be treated the same as having a header row in the file.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353074021,Add an option for specifying column names when inserting CSV data, https://github.com/simonw/sqlite-utils/issues/474#issuecomment-1229265285,https://api.github.com/repos/simonw/sqlite-utils/issues/474,1229265285,IC_kwDOCGYnMM5JRRmF,9599,simonw,2022-08-27T20:52:53Z,2022-08-27T20:52:53Z,OWNER,"A couple of tricks I use here. Firstly, I often create the table before the import using the `sqlite-utils create-table` command: https://sqlite-utils.datasette.io/en/stable/cli.html#creating-tables The other current option is to use the `bulk` command, which lets you construct a custom SQL query to execute against every row from a CSV file: https://sqlite-utils.datasette.io/en/stable/cli.html#executing-sql-in-bulk Do either of those options work here or is there a useful new feature that would work better?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353074021,Add an option for specifying column names when inserting CSV data, https://github.com/simonw/sqlite-utils/issues/472#issuecomment-1229125890,https://api.github.com/repos/simonw/sqlite-utils/issues/472,1229125890,IC_kwDOCGYnMM5JQvkC,9599,simonw,2022-08-27T05:15:26Z,2022-08-27T05:15:26Z,OWNER,Here's the docs I get to update to remove the documented `global` hack: https://sqlite-utils.datasette.io/en/3.28/cli.html#cli-convert-complex,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352946135,Reuse the locals/globals fix from --functions for other code accepting options, https://github.com/simonw/sqlite-utils/issues/472#issuecomment-1229125614,https://api.github.com/repos/simonw/sqlite-utils/issues/472,1229125614,IC_kwDOCGYnMM5JQvfu,9599,simonw,2022-08-27T05:12:59Z,2022-08-27T05:12:59Z,OWNER,"Annoyingly this means the `--import` option is no longer required, but removing it would be a backwards incompatible breakage so I need to leave it in. I can minimize it in the documentation though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352946135,Reuse the locals/globals fix from --functions for other code accepting options, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1238873948,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1238873948,IC_kwDOCGYnMM5J17dc,9599,simonw,2022-09-07T03:46:26Z,2022-09-07T03:46:26Z,OWNER,"> Is it still nfortunately slow and tricky when playing with floats ? Not sure what you mean here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229430228,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229430228,IC_kwDOCGYnMM5JR53U,4312421,stonebig,2022-08-28T10:43:35Z,2022-08-28T10:43:35Z,NONE,"Is it still nfortunately slow and tricky when playing with floats ? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229125114,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229125114,IC_kwDOCGYnMM5JQvX6,9599,simonw,2022-08-27T05:08:58Z,2022-08-27T05:08:58Z,OWNER,"Testing `bulk --functions`: ``` % sqlite-utils create-table chickens.db chickens id integer name text name_upper text % echo 'id,name 1,Blue 2,Snowy 3,Azi 4,Lila 5,Suna 6,Cardi' | sqlite-utils bulk chickens.db ' insert into chickens (id, name, name_upper) values (:id, :name, myupper(:name)) ' - --functions ' def myupper(s): return s.upper() ' --csv % sqlite-utils rows chickens.db chickens [{""id"": 1, ""name"": ""Blue"", ""name_upper"": ""BLUE""}, {""id"": 2, ""name"": ""Snowy"", ""name_upper"": ""SNOWY""}, {""id"": 3, ""name"": ""Azi"", ""name_upper"": ""AZI""}, {""id"": 4, ""name"": ""Lila"", ""name_upper"": ""LILA""}, {""id"": 5, ""name"": ""Suna"", ""name_upper"": ""SUNA""}, {""id"": 6, ""name"": ""Cardi"", ""name_upper"": ""CARDI""}] ```","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229124549,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229124549,IC_kwDOCGYnMM5JQvPF,9599,simonw,2022-08-27T05:03:39Z,2022-08-27T05:03:39Z,OWNER,"I don't think I need separate documentation for `sqlite-utils memory` and `sqlite-tils bulk` since they work the same, and the `--help` text provides the necessary hints.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229124379,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229124379,IC_kwDOCGYnMM5JQvMb,9599,simonw,2022-08-27T05:02:21Z,2022-08-27T05:02:21Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#defining-custom-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120899,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229120899,IC_kwDOCGYnMM5JQuWD,9599,simonw,2022-08-27T04:31:35Z,2022-08-27T04:32:38Z,OWNER,I should add this `--functions` feature to the `memory` and `bulk` commands too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120653,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229120653,IC_kwDOCGYnMM5JQuSN,9599,simonw,2022-08-27T04:29:49Z,2022-08-27T04:30:03Z,OWNER,"Found a fix for that! I replaced this: ```python locals = {} globals = {} exec(functions, globals, locals) # Register all callables in the locals dict: for name, value in locals.items(): if callable(value): db.register_function(value, name=name) ``` With this: ```python globals = {} exec(functions, globals) # Register all callables in the globals dict: for name, value in globals.items(): if callable(value): db.register_function(value, name=name) ``` Because https://docs.python.org/3/library/functions.html#exec says: > If only *globals* is provided, it must be a dictionary (and not a subclass of dictionary), which will be used for both the global and the local variables. If *globals* and *locals* are given, they are used for the global and local variables, respectively.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120104,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229120104,IC_kwDOCGYnMM5JQuJo,9599,simonw,2022-08-27T04:25:39Z,2022-08-27T04:25:39Z,OWNER,"This works: ``` sqlite-utils :memory: 'select extract_domain(""https://www.google.com/blah"")' --functions ' from urllib.parse import urlparse def extract_domain(url): global urlparse return urlparse(url).netloc ' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229119999,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229119999,IC_kwDOCGYnMM5JQuH_,9599,simonw,2022-08-27T04:24:58Z,2022-08-27T04:24:58Z,OWNER,"I've encountered this problem before: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-convert-complex > ``` > $ sqlite-utils convert content.db articles score ' > import random > random.seed(10) > > def convert(value): > global random > return random.random() > ' > ``` > Note the `global random` line here. Due to the way the tool compiles Python code, this is necessary to ensure the `random` module is available within the `convert()` function. If you were to omit this you would see a `NameError: name 'random' is not defined` error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229119171,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229119171,IC_kwDOCGYnMM5JQt7D,9599,simonw,2022-08-27T04:18:28Z,2022-08-27T04:18:28Z,OWNER,"I tried this: ``` sqlite-utils :memory: 'select extract_domain(""https://www.google.com/blah"")' --functions ' from urllib.parse import urlparse def extract_domain(url): from urllib.parse import urlparse return urlparse(url).netloc ' ``` And got: ``` NameError: name 'urlparse' is not defined Error: user-defined function raised exception ``` But this worked OK: ``` % sqlite-utils :memory: 'select extract_domain(""https://www.google.com/blah"")' --functions ' def extract_domain(url): from urllib.parse import urlparse return urlparse(url).netloc ' [{""extract_domain(\""https://www.google.com/blah\"")"": ""www.google.com""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229118619,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229118619,IC_kwDOCGYnMM5JQtyb,9599,simonw,2022-08-27T04:14:52Z,2022-08-27T04:14:52Z,OWNER,"Quick prototype: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 43e76fa..5dee4f6 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -1633,6 +1633,9 @@ def drop_view(path, view, ignore, load_extension): type=(str, str), help=""Named :parameters for SQL query"", ) +@click.option( + ""--functions"", help=""Python code defining one or more custom SQL functions"" +) @load_extension_option def query( path, @@ -1649,6 +1652,7 @@ def query( raw, param, load_extension, + functions, ): """"""Execute SQL query and return the results as JSON @@ -1665,6 +1669,16 @@ def query( _load_extensions(db, load_extension) db.register_fts4_bm25() + # Register any Python functions as SQL functions: + if functions: + locals = {} + globals = {} + exec(functions, globals, locals) + # Register all callables in the locals dict: + for name, value in locals.items(): + if callable(value): + db.register_function(value, name=name) + _execute_query( db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols ) ``` Demo: ```bash % sqlite-utils :memory: 'select 1 + dog()' --functions ' quote> def dog(): quote> return 2 quote> ' [{""1 + dog()"": 3}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229116423,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229116423,IC_kwDOCGYnMM5JQtQH,9599,simonw,2022-08-27T04:00:52Z,2022-08-27T04:00:52Z,OWNER,"Alternative design would be `--function name definition` - like this: ``` sqlite-utils data.db 'update images set domain = extract_domain(url)' --function extract_domain ' from urllib.parse import urlparse return urlparse(url).netloc ' ``` I like the `--functions` design better because it leaves space for import statements at the top of the code block, and allows more than one function to be defined in a single go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229130422,https://api.github.com/repos/simonw/sqlite-utils/issues/470,1229130422,IC_kwDOCGYnMM5JQwq2,9599,simonw,2022-08-27T05:55:01Z,2022-08-27T05:55:01Z,OWNER,cc @asg017 since this feature was inspired by his work on Datasette.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932038,Upgrade `--load-extension` to accept entrypoints like Datasette, https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229130242,https://api.github.com/repos/simonw/sqlite-utils/issues/470,1229130242,IC_kwDOCGYnMM5JQwoC,9599,simonw,2022-08-27T05:53:31Z,2022-08-27T05:53:31Z,OWNER,"Tests passed in CI against Ubuntu: ``` tests/test_cli.py::test_load_extension[None-should_pass0-should_fail0] PASSED [ 19%] tests/test_cli.py::test_load_extension[sqlite3_ext_b_init-b-should_fail1] PASSED [ 19%] tests/test_cli.py::test_load_extension[sqlite3_ext_c_init-c-should_fail2] PASSED [ 19%] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932038,Upgrade `--load-extension` to accept entrypoints like Datasette, https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229130158,https://api.github.com/repos/simonw/sqlite-utils/issues/470,1229130158,IC_kwDOCGYnMM5JQwmu,9599,simonw,2022-08-27T05:52:46Z,2022-08-27T05:52:46Z,OWNER,"To run the tests locally on my Mac I needed to run Ubuntu in Docker (for the compilation step). This worked: docker run -it -v ""$(pwd):/app"" ubuntu:latest /bin/bash Then: ``` apt-get update && apt-get install python3-pip gcc libsqlite3-dev -y cd /app/tests gcc ext.c -fPIC -shared -o ext.so cd /app pip3 install -e '.[test]' pytest -k test_load_extension ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932038,Upgrade `--load-extension` to accept entrypoints like Datasette, https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229128950,https://api.github.com/repos/simonw/sqlite-utils/issues/470,1229128950,IC_kwDOCGYnMM5JQwT2,9599,simonw,2022-08-27T05:42:04Z,2022-08-27T05:42:04Z,OWNER,"Extension looks to be correctly compiled: ``` Run (cd tests && gcc ext.c -fPIC -shared -o ext.so && ls -lah) total 412K drwxr-xr-x 3 runner docker 4.0K Aug 27 05:40 . drwxr-xr-x 8 runner docker 4.0K Aug 27 05:40 .. -rw-r--r-- 1 runner docker 0 Aug 27 05:40 __init__.py -rw-r--r-- 1 runner docker 760 Aug 27 05:40 conftest.py -rw-r--r-- 1 runner docker 1.6K Aug 27 05:40 ext.c -rwxr-xr-x 1 runner docker 16K Aug 27 05:40 ext.so drwxr-xr-x 2 runner docker 4.0K Aug 27 05:40 sniff ``` So why doesn't this check pass? https://github.com/simonw/sqlite-utils/blob/3f694e51a0212a297f663e2fc7f9cdad8775a550/tests/test_cli.py#L25-L30","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932038,Upgrade `--load-extension` to accept entrypoints like Datasette, https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229128617,https://api.github.com/repos/simonw/sqlite-utils/issues/470,1229128617,IC_kwDOCGYnMM5JQwOp,9599,simonw,2022-08-27T05:39:06Z,2022-08-27T05:39:06Z,OWNER,"That didn't run the tests as expected on Ubuntu: https://github.com/simonw/sqlite-utils/runs/8048229213?check_suite_focus=true ``` tests/test_cli.py::test_load_extension[None-should_pass0-should_fail0] SKIPPED [ 19%] tests/test_cli.py::test_load_extension[sqlite3_ext_b_init-b-should_fail1] SKIPPED [ 19%] tests/test_cli.py::test_load_extension[sqlite3_ext_c_init-c-should_fail2] SKIPPED [ 19%] ``` That should have worked because of this: https://github.com/simonw/sqlite-utils/blob/3f694e51a0212a297f663e2fc7f9cdad8775a550/.github/workflows/test.yml#L38-L41 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932038,Upgrade `--load-extension` to accept entrypoints like Datasette, https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229115501,https://api.github.com/repos/simonw/sqlite-utils/issues/470,1229115501,IC_kwDOCGYnMM5JQtBt,9599,simonw,2022-08-27T03:54:26Z,2022-08-27T03:54:26Z,OWNER,I can borrow Alex's example extension for the tests too: https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/tests/ext.c,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932038,Upgrade `--load-extension` to accept entrypoints like Datasette, https://github.com/simonw/sqlite-utils/issues/469#issuecomment-1229120779,https://api.github.com/repos/simonw/sqlite-utils/issues/469,1229120779,IC_kwDOCGYnMM5JQuUL,9599,simonw,2022-08-27T04:30:48Z,2022-08-27T04:30:48Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352931464,sqlite-utils rows --order option, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1229311612,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1229311612,IC_kwDOCGYnMM5JRc58,9599,simonw,2022-08-27T23:19:41Z,2022-08-27T23:19:41Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#explicitly-creating-a-table - https://sqlite-utils.datasette.io/en/latest/cli.html#creating-tables ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1229206306,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1229206306,IC_kwDOCGYnMM5JRDMi,9599,simonw,2022-08-27T14:47:04Z,2022-08-27T14:47:04Z,OWNER,I could add a `--transform` option to `sqlite-utils create-table` too.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224388810,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224388810,IC_kwDOCGYnMM5I-rDK,9599,simonw,2022-08-23T17:21:16Z,2022-08-23T17:21:16Z,OWNER,Also needs comprehensive tests and documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224386951,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224386951,IC_kwDOCGYnMM5I-qmH,9599,simonw,2022-08-23T17:20:07Z,2022-08-23T17:20:07Z,OWNER,"Example of that prototype working: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""dogs""].create({""id"": int, ""name"": str}, pk=""id"")
>>> db[""dogs""].create({""id"": int, ""name"": str, ""age"": int}, pk=""id"", transform=True)
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224385575,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224385575,IC_kwDOCGYnMM5I-qQn,9599,simonw,2022-08-23T17:19:00Z,2022-08-23T17:19:00Z,OWNER,"Initial prototype: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 18a442a..03fd345 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -875,6 +875,7 @@ class Database: hash_id_columns: Optional[Iterable[str]] = None, extracts: Optional[Union[Dict[str, str], List[str]]] = None, if_not_exists: bool = False, + transform: bool = False, ) -> ""Table"": """""" Create a table with the specified name and the specified ``{column_name: type}`` columns. @@ -892,7 +893,39 @@ class Database: :param hash_id_columns: List of columns to be used when calculating the hash ID for a row :param extracts: List or dictionary of columns to be extracted during inserts, see :ref:`python_api_extracts` :param if_not_exists: Use ``CREATE TABLE IF NOT EXISTS`` - """""" + :param transform: If table already exists, transform it to fit the specified schema + """""" + # Transform table to match the new definition if table already exists: + if transform and self[name].exists(): + # First add missing columns and columns to drop + existing_columns = self[name].columns_dict + missing_columns = dict( + (col_name, col_type) + for col_name, col_type in columns.items() + if col_name not in existing_columns + ) + columns_to_drop = [ + column for column in existing_columns if column not in columns + ] + if missing_columns: + for col_name, col_type in missing_columns.items(): + self[name].add_column(col_name, col_type) + # Do we need to reset the column order? + column_order = None + if list(existing_columns) != list(columns): + column_order = list(columns) + # Only run .transform() if there is something to do + # TODO: this misses changes like pk= without also column changes + if columns_to_drop or missing_columns or column_order: + self[name].transform( + types=columns, + drop=columns_to_drop, + column_order=column_order, + not_null=not_null, + defaults=defaults, + pk=pk, + ) + return cast(Table, self[name]) sql = self.create_table_sql( name=name, columns=columns, @@ -1477,6 +1510,7 @@ class Table(Queryable): hash_id_columns: Optional[Iterable[str]] = None, extracts: Optional[Union[Dict[str, str], List[str]]] = None, if_not_exists: bool = False, + transform: bool = False, ) -> ""Table"": """""" Create a table with the specified columns. @@ -1508,6 +1542,7 @@ class Table(Queryable): hash_id_columns=hash_id_columns, extracts=extracts, if_not_exists=if_not_exists, + transform=transform, ) return self ``` Needs more thought about how things like just a change to `pk=` should work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224382336,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224382336,IC_kwDOCGYnMM5I-peA,50527,jefftriplett,2022-08-23T17:16:13Z,2022-08-23T17:16:13Z,CONTRIBUTOR,"> Should passing `alter=True` also drop any columns that aren't included in the new table structure? > > It could even spot column types that aren't correct and fix those. > > Is that consistent with the expectations set by how `alter=True` works elsewhere? I would lean towards not dropping them (or making a `drop=True` or `drop_columns=True`or `drop_missing_columns=True`) to work with existing tables easier. I do like that sqlite-utils mostly just works with existing tables but it's also nice to add to existing fields in a few cases. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224283367,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224283367,IC_kwDOCGYnMM5I-RTn,9599,simonw,2022-08-23T16:05:55Z,2022-08-23T16:05:55Z,OWNER,"... but that's what the `table.transform(...)` method does already! So maybe this is actually a `transform=True` parameter to `create()` that triggers `table.transform(...)` if necessary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224280225,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224280225,IC_kwDOCGYnMM5I-Qih,9599,simonw,2022-08-23T16:03:33Z,2022-08-23T16:03:33Z,OWNER,"Maybe there should be a separate `table.alter(...)` method that does the actual work here, with `.create(..., alter=True)` as syntactic sugar for triggering that if the table exists already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224278280,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224278280,IC_kwDOCGYnMM5I-QEI,9599,simonw,2022-08-23T16:02:07Z,2022-08-23T16:02:07Z,OWNER,"Thinking about this more, I think `alter=True` is a good name for this option even if it does more than the same option on `.insert()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224272854,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224272854,IC_kwDOCGYnMM5I-OvW,9599,simonw,2022-08-23T15:58:14Z,2022-08-23T15:58:14Z,OWNER,Could call it `ensure=True` here if it works differently enough from `alter=True` that the behavior could be confusing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224271324,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224271324,IC_kwDOCGYnMM5I-OXc,9599,simonw,2022-08-23T15:56:58Z,2022-08-23T15:56:58Z,OWNER,"Should passing `alter=True` also drop any columns that aren't included in the new table structure? It could even spot column types that aren't correct and fix those. Is that consistent with the expectations set by how `alter=True` works elsewhere?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224268273,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224268273,IC_kwDOCGYnMM5I-Nnx,9599,simonw,2022-08-23T15:54:24Z,2022-08-23T15:54:24Z,OWNER,I'm not crazy about having to pass both `alter=True` and `if_not_exists=True` - maybe `alter` should imply `if_not_exists`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224264744,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224264744,IC_kwDOCGYnMM5I-Mwo,9599,simonw,2022-08-23T15:51:29Z,2022-08-23T15:53:29Z,OWNER,"Jeff suggested `db[table].(..., alter=True)` for this. ```python db[""urls""].create( { ""url"": str, ""crawled"": bool, ""body"": str, ""headers"": dict, ""status"": int, ""status_text"": str, }, pk=""url"", defaults={""crawled"": False}, if_not_exists=True, alter=True, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214199975,https://api.github.com/repos/simonw/sqlite-utils/issues/464,1214199975,IC_kwDOCGYnMM5IXzin,9599,simonw,2022-08-13T18:09:02Z,2022-08-13T18:09:02Z,OWNER,I may be able to do this without that extra dependency: https://www.sphinx-doc.org/en/master/usage/extensions/linkcode.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1338001039,Link from documentation to source code, https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214188615,https://api.github.com/repos/simonw/sqlite-utils/issues/464,1214188615,IC_kwDOCGYnMM5IXwxH,9599,simonw,2022-08-13T16:50:16Z,2022-08-13T16:50:16Z,OWNER,Extracted a TIL: https://til.simonwillison.net/python/setup-py-from-url,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1338001039,Link from documentation to source code, https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214187919,https://api.github.com/repos/simonw/sqlite-utils/issues/464,1214187919,IC_kwDOCGYnMM5IXwmP,9599,simonw,2022-08-13T16:46:01Z,2022-08-13T16:49:36Z,OWNER,"Source links are now live here: https://sqlite-utils.datasette.io/en/latest/reference.html ![CleanShot 2022-08-13 at 09 49 02@2x](https://user-images.githubusercontent.com/9599/184503393-6559ab2b-cd02-4c85-98b9-62d10b3bdab8.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1338001039,Link from documentation to source code, https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214184631,https://api.github.com/repos/simonw/sqlite-utils/issues/464,1214184631,IC_kwDOCGYnMM5IXvy3,9599,simonw,2022-08-13T16:21:40Z,2022-08-13T16:21:40Z,OWNER,"I tried it out and ran into a bug, which I've reported and submitted a PR for: - https://github.com/beanbaginc/beanbag-docutils/pull/1 - https://github.com/beanbaginc/beanbag-docutils/pull/2 I'm going to ship a version that uses my fork of the repo here: https://github.com/simonw/beanbag-docutils/tree/bytes-in-url","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1338001039,Link from documentation to source code, https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214184512,https://api.github.com/repos/simonw/sqlite-utils/issues/464,1214184512,IC_kwDOCGYnMM5IXvxA,9599,simonw,2022-08-13T16:20:41Z,2022-08-13T16:20:41Z,OWNER,"https://github.com/beanbaginc/beanbag-docutils/blob/master/beanbag_docutils/sphinx/ext/github.py via https://twitter.com/chipx86/status/1558271211901050880 looks like a good option. It works using this Sphinx mechanism: https://www.sphinx-doc.org/en/master/usage/extensions/linkcode.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1338001039,Link from documentation to source code, https://github.com/simonw/sqlite-utils/issues/461#issuecomment-1203168339,https://api.github.com/repos/simonw/sqlite-utils/issues/461,1203168339,IC_kwDOCGYnMM5HtuRT,9599,simonw,2022-08-02T20:12:14Z,2022-08-02T20:12:14Z,OWNER,"The neat thing about these is that they can be embedded directly in a GitHub README. Not sure if they show up on PyPI project pages, though I expect that they would.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1326349129,Consider including animated SVG console demos, https://github.com/simonw/sqlite-utils/issues/46#issuecomment-593118471,https://api.github.com/repos/simonw/sqlite-utils/issues/46,593118471,MDEyOklzc3VlQ29tbWVudDU5MzExODQ3MQ==,9599,simonw,2020-03-01T16:53:40Z,2020-03-01T16:53:40Z,OWNER,That seems sensible to me. Opening a ticket for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471780443,extracts= option for insert/update/etc, https://github.com/simonw/sqlite-utils/issues/46#issuecomment-592999503,https://api.github.com/repos/simonw/sqlite-utils/issues/46,592999503,MDEyOklzc3VlQ29tbWVudDU5Mjk5OTUwMw==,35075,chrishas35,2020-02-29T22:08:20Z,2020-02-29T22:08:20Z,NONE,"@simonw any thoughts on allow extracts to specify the lookup column name? If I'm understanding the documentation right, `.lookup()` allows you to define the ""value"" column (the documentation uses name), but when you use `extracts` keyword as part of `.insert()`, `.upsert()` etc. the lookup must be done against a column named ""value"". I have an existing lookup table that I've populated with columns ""id"" and ""name"" as opposed to ""id"" and ""value"", and seems I can't use `extracts=`, unless I'm missing something... Initial thought on how to do this would be to allow the dictionary value to be a tuple of table name column pair... so: ``` table = db.table(""trees"", extracts={""species_id"": (""Species"", ""name""}) ``` I haven't dug too much into the existing code yet, but does this make sense? Worth doing? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471780443,extracts= option for insert/update/etc, https://github.com/simonw/sqlite-utils/issues/46#issuecomment-514273140,https://api.github.com/repos/simonw/sqlite-utils/issues/46,514273140,MDEyOklzc3VlQ29tbWVudDUxNDI3MzE0MA==,9599,simonw,2019-07-23T15:57:53Z,2019-07-23T15:57:53Z,OWNER,"This will play very well with the new ability to specify these arguments in the class constructor, like so: ```python table = db.table(""trees"", extracts={""species_id"": ""Species""}) table.insert_all([{ ""id"": 1, ""species_id"": ""Oak"", }, { ""id"": 2, ""species_id"": ""Oak"", }, { ""id"": 3, ""species_id"": ""Palm"", }])","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471780443,extracts= option for insert/update/etc, https://github.com/simonw/sqlite-utils/issues/458#issuecomment-1197509096,https://api.github.com/repos/simonw/sqlite-utils/issues/458,1197509096,IC_kwDOCGYnMM5HYIno,9599,simonw,2022-07-28T00:14:22Z,2022-07-28T00:14:22Z,OWNER,Updated documentation: https://github.com/simonw/sqlite-utils/blob/1491b66dd7439dd87cd5cd4c4684f46eb3c5751b/docs/python-api.rst#registering-custom-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1320243134,Support custom names for registered functions, https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1190578687,https://api.github.com/repos/simonw/sqlite-utils/issues/456,1190578687,IC_kwDOCGYnMM5G9sn_,9599,simonw,2022-07-20T17:50:50Z,2022-07-20T17:50:50Z,OWNER,Wow https://tidyr.tidyverse.org/articles/pivot.html is _fascinating_.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1310243385,feature request: pivot command, https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1190449764,https://api.github.com/repos/simonw/sqlite-utils/issues/456,1190449764,IC_kwDOCGYnMM5G9NJk,45919695,jcmkk3,2022-07-20T15:45:54Z,2022-07-20T15:45:54Z,NONE,"> hadley wickham's melt and reshape could be good inspo: http://had.co.nz/reshape/introduction.pdf Note that Hadley has since implemented `pivot_longer` and `pivot_wider` instead of the previous verbs/functions that he used. Those can be found in the tidyr package and are probably the best reference which includes all of the learnings from years of user feedback. https://tidyr.tidyverse.org/articles/pivot.html","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1310243385,feature request: pivot command, https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1190277829,https://api.github.com/repos/simonw/sqlite-utils/issues/456,1190277829,IC_kwDOCGYnMM5G8jLF,536941,fgregg,2022-07-20T13:19:15Z,2022-07-20T13:19:15Z,CONTRIBUTOR,hadley wickham's melt and reshape could be good inspo: http://had.co.nz/reshape/introduction.pdf,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1310243385,feature request: pivot command, https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1190272780,https://api.github.com/repos/simonw/sqlite-utils/issues/456,1190272780,IC_kwDOCGYnMM5G8h8M,536941,fgregg,2022-07-20T13:14:54Z,2022-07-20T13:14:54Z,CONTRIBUTOR,"for example, i have data on votes that look like this: | ballot_id | option_id | choice | |-|-|-| | 1 | 1 | 0 | | 1 | 2 | 1 | | 1 | 3 | 0 | | 1 | 4 | 1 | | 2 | 1 | 1 | | 2 | 2 | 0 | | 2 | 3 | 1 | | 2 | 4 | 0 | and i want to reshape from this long form to this wide form: | ballot_id | option_id_1 | option_id_2 | option_id_3 | option_id_ 4| |-|-|-|-| -| | 1 | 0 | 1 | 0 | 1 | | 2 | 1 | 0 | 1| 0 | i could do such a think like this. ```sql select ballot_id, sum(choice) filter (where option_id = 1) as option_id_1, sum(choice) filter (where option_id = 2) as option_id_2, sum(choice) filter (where option_id = 3) as option_id_3, sum(choice) filter (where option_id = 4) as option_id_4 from vote group by ballot_id ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1310243385,feature request: pivot command, https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1189698289,https://api.github.com/repos/simonw/sqlite-utils/issues/456,1189698289,IC_kwDOCGYnMM5G6Vrx,9599,simonw,2022-07-20T01:13:47Z,2022-07-20T01:13:47Z,OWNER,"Really interesting idea! Can you flesh out what this might look like, maybe with a before and after example schema?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1310243385,feature request: pivot command, https://github.com/simonw/sqlite-utils/issues/454#issuecomment-1185971600,https://api.github.com/repos/simonw/sqlite-utils/issues/454,1185971600,IC_kwDOCGYnMM5GsH2Q,9599,simonw,2022-07-15T21:48:23Z,2022-07-15T21:48:23Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#duplicating-tables - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#duplicate","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1306548397,CLI command for duplicating tables, https://github.com/simonw/sqlite-utils/issues/453#issuecomment-1185974145,https://api.github.com/repos/simonw/sqlite-utils/issues/453,1185974145,IC_kwDOCGYnMM5GsIeB,9599,simonw,2022-07-15T21:52:18Z,2022-07-15T21:52:18Z,OWNER,"I should warn you that this isn't a supported API - I reserve the right to change how it works between release without a major version bump, because it's not part of the documented API surface. You'll be fine if you pin to exact versions of the library though! You may find this recently-documented function useful though: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file See: - #443 I'm going to close this issue for the moment, but if anyone wants to submit a PR that cleans up this I'll happily review it. ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1303169663,'unclosed file' warning when using insert_upsert_implementation from Python, https://github.com/simonw/sqlite-utils/issues/451#issuecomment-1185987117,https://api.github.com/repos/simonw/sqlite-utils/issues/451,1185987117,IC_kwDOCGYnMM5GsLot,9599,simonw,2022-07-15T22:09:33Z,2022-07-15T22:09:33Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-chunks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1298531653,Make sqlite_utils.utils.chunks a documented function, https://github.com/simonw/sqlite-utils/issues/451#issuecomment-1185975132,https://api.github.com/repos/simonw/sqlite-utils/issues/451,1185975132,IC_kwDOCGYnMM5GsItc,9599,simonw,2022-07-15T21:53:42Z,2022-07-15T21:53:42Z,OWNER,Documentation can go here: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite-utils-utils,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1298531653,Make sqlite_utils.utils.chunks a documented function, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1186002019,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1186002019,IC_kwDOCGYnMM5GsPRj,9599,simonw,2022-07-15T22:39:09Z,2022-07-15T22:39:09Z,OWNER,Here are all of the changes I made in this issue: https://github.com/simonw/sqlite-utils/compare/e10536c7f59abbb785f092bf83c4ab94c00e31a3...b9a89a0f2c3559989efe65f25a6e1f8fa76fe8b0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1186001408,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1186001408,IC_kwDOCGYnMM5GsPIA,9599,simonw,2022-07-15T22:37:40Z,2022-07-15T22:37:40Z,OWNER,I'm going to skip `add-geometry-column` and `create-spatial-index` as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185997860,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185997860,IC_kwDOCGYnMM5GsOQk,9599,simonw,2022-07-15T22:32:51Z,2022-07-15T22:32:51Z,OWNER,`add-foreign-keys` is enough of a power-feature that I'm happy not to add this there.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185994282,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185994282,IC_kwDOCGYnMM5GsNYq,9599,simonw,2022-07-15T22:24:12Z,2022-07-15T22:24:12Z,OWNER,"I can do this for `--create-index` instead: ``` Options: --name TEXT Explicit name for the new index --unique Make this a unique index --if-not-exists, --ignore Ignore if index already exists --analyze Run ANALYZE after creating the index --load-extension TEXT SQLite extensions to load -h, --help Show this message and exit. ``` That supports both names for the option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185993791,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185993791,IC_kwDOCGYnMM5GsNQ_,9599,simonw,2022-07-15T22:23:02Z,2022-07-15T22:23:02Z,OWNER,"`create-spatial-index` is tricky. For consistency with `create-index` it should really use `--if-not-exists` - but under the hood it's not using `IF NOT EXISTS` because those indexes are created using `select CreateSpatialIndex(...)`. Really this highlights that `--if-not-exists` is a bad name for that option on `create-index`. I'm tempted to add `--ignore` to `create-index` and mark `--if-not-exists` as deprecated - though it will still work because I don't want to release a major version bump.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185988480,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185988480,IC_kwDOCGYnMM5GsL-A,9599,simonw,2022-07-15T22:12:20Z,2022-07-15T22:12:20Z,OWNER,`disable-fts` already fails silently if the table doesn't have FTS setup.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185988277,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185988277,IC_kwDOCGYnMM5GsL61,9599,simonw,2022-07-15T22:11:52Z,2022-07-15T22:11:52Z,OWNER,"For `enable-fts` I should use `--replace` instead, since that matches how the Python API works: https://github.com/simonw/sqlite-utils/blob/9dd4cf891d9f4565019e030ddc712507ac87b998/sqlite_utils/db.py#L2088-L2106","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185983894,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185983894,IC_kwDOCGYnMM5GsK2W,9599,simonw,2022-07-15T22:06:29Z,2022-07-15T22:37:20Z,OWNER,"Commands that could have `--ignore` added to them: - [x] `enable-fts` - [x] `disable-fts` (not doing) - [x] `add-column` - [x] `add-foreign-keys` (decided not to do this) - [x] `duplicate` - [x] `add-geometry-column` (decided not to do this) - [x] `create-spatial-index` (decided not to do this)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185982012,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185982012,IC_kwDOCGYnMM5GsKY8,9599,simonw,2022-07-15T22:02:59Z,2022-07-15T22:03:49Z,OWNER,"From https://sqlite-utils.datasette.io/en/latest/cli-reference.html Commands that currently have an `--ignore` flag, and their descriptions: - `insert`: Ignore records if pk already exists - `create-table`: If table already exists, do nothing - `add-foreign-key`: If foreign key already exists, do nothing - `drop-table`: *description is missing* - `create-view`: If view already exists, do nothing - `drop-view`: *description is missing* And `create-index` has: - `--if-not-exists`: Ignore if index already exists","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1185970114,https://api.github.com/repos/simonw/sqlite-utils/issues/449,1185970114,IC_kwDOCGYnMM5GsHfC,9599,simonw,2022-07-15T21:46:13Z,2022-07-15T21:46:13Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#duplicating-tables - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.duplicate","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279863844,Utilities for duplicating tables and creating a table with the results of a query, https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1179579878,https://api.github.com/repos/simonw/sqlite-utils/issues/449,1179579878,IC_kwDOCGYnMM5GTvXm,1690072,davidleejy,2022-07-09T17:41:32Z,2022-07-09T17:41:50Z,CONTRIBUTOR,Learnt that the types in Sqlite-utils differ somewhat from those in Sqlite. I've changed my test to account for this difference and the test has passed successfully. I will submit a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279863844,Utilities for duplicating tables and creating a table with the results of a query, https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1174027079,https://api.github.com/repos/simonw/sqlite-utils/issues/449,1174027079,IC_kwDOCGYnMM5F-jtH,1690072,davidleejy,2022-07-04T17:33:04Z,2022-07-04T17:48:43Z,CONTRIBUTOR,"I've written the code and test. Would you be able to advise how to compare table columns in a pytest function properly? Experiencing a challenge when comparing columns. Test: ```python def test_duplicate(fresh_db): table = fresh_db.create_table( ""table1"", { ""text_col"": str, ""float_col"": float, ""int_col"": int, ""bool_col"": bool, ""bytes_col"": bytes, ""datetime_col"": datetime.datetime, }, ) dt = datetime.datetime.now() b = bytes('hello world', 'utf-8') data = {""text_col"": ""Cleo"", ""float_col"": 3.14, ""int_col"": -2, ""bool_col"": True, ""bytes_col"": b, ""datetime_col"": str(dt)} table1 = fresh_db[""table1""] row_id = table1.insert(data).last_rowid table1.duplicate('table2') table2 = fresh_db[""table2""] assert data == table2.get(row_id) assert table1.columns == table2.columns # FAILS HERE ``` Result: ![Screenshot 2022-07-05 at 1 31 55 AM](https://user-images.githubusercontent.com/1690072/177198814-daac48c9-5746-49d0-a14a-14fe181c5a2f.png) Failure is due to column types being named differently -- e.g. 'FLOAT' vs 'REAL', 'INTEGER' vs 'INT'. How should I go about comparing columns while accounting for equivalent types? Or did I miss out something in my duplication code correctly? Here's how I did it: in `db.py`, I've added the following code: ```python class Table(Queryable): [...] def duplicate( self, name_new: str ) -> ""Table"": """""" Duplicate this table in this database. :param name_new: Name of new table. """""" assert self.exists() with self.db.conn: sql = ""CREATE TABLE [{new_table}] AS SELECT * FROM [{table}];"".format( new_table = name_new, table = self.name, ) self.db.execute(sql) return self.db[name_new] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279863844,Utilities for duplicating tables and creating a table with the results of a query, https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1172902594,https://api.github.com/repos/simonw/sqlite-utils/issues/449,1172902594,IC_kwDOCGYnMM5F6RLC,9599,simonw,2022-07-02T13:55:51Z,2022-07-02T13:55:51Z,OWNER,"I like the idea of this as a feature - design could look like this: db[""my_table""].duplicate(""new_table"") Then for the CLI tool: sqlite-utils duplicate data.db my_table new_table Additional features that might be useful: - Duplicate across connection aliases - so you can copy a table to another database using the `duplicate` method or command - Ability to execute an arbitrary SQL query and save the results to a new table - basically a wrapper around `CREATE TABLE t AS SELECT ...`, again with alias support for copying to another database","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279863844,Utilities for duplicating tables and creating a table with the results of a query, https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1539109816,https://api.github.com/repos/simonw/sqlite-utils/issues/448,1539109816,IC_kwDOCGYnMM5bvPO4,9599,simonw,2023-05-08T22:01:00Z,2023-05-08T22:01:00Z,OWNER,"This is being handled in: - #520","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279144769,Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto', https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1297703307,https://api.github.com/repos/simonw/sqlite-utils/issues/448,1297703307,IC_kwDOCGYnMM5NWWGL,167893,mcarpenter,2022-10-31T21:23:51Z,2022-10-31T21:27:32Z,CONTRIBUTOR,"The Windows aspect is a red herring: OP's sample above produces the same error on Linux. (Though I don't know what's going on with the CI). The same error can also be obtained by passing an `io` from a file opened in non-binary mode (`'r'` as opposed to `'rb'`) to `rows_from_file()`. This is how I got here. The fix for my case is easy: open the file in mode `'rb'`. The analagous fix for OP's problem also works: use `BytesIO` in place of `StringIO`. Minimal test case (derived from [utils.py](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/utils.py#L304)): ``` python import io from typing import cast #fp = io.StringIO(""id,name\n1,Cleo"") # error fp = io.BytesIO(bytes(""id,name\n1,Cleo"", encoding='utf-8')) # okay reader = io.BufferedReader(cast(io.RawIOBase, fp)) reader.peek(1) # exception thrown here ``` I see the signature of `rows_from_file()` correctly has `fp: BinaryIO` but I guess you'd need either a runtime type check for that (not all `io`s have `mode()`), or to catch the `AttributeError` on `peek()` to produce a better error for users. Neither option is ideal. Some thoughts on testing binary-ness of `io`s in this SO question: https://stackoverflow.com/questions/44584829/how-to-determine-if-file-is-opened-in-binary-or-text-mode","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279144769,Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto', https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1186002560,https://api.github.com/repos/simonw/sqlite-utils/issues/448,1186002560,IC_kwDOCGYnMM5GsPaA,9599,simonw,2022-07-15T22:40:40Z,2022-07-15T22:40:40Z,OWNER,"This is very strange. GitHub Actions CI here runs against Windows and installs OK. Marking this as ""Help wanted"" to see if anyone can figure out what's going on here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279144769,Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto', https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1162500525,https://api.github.com/repos/simonw/sqlite-utils/issues/448,1162500525,IC_kwDOCGYnMM5FSlmt,236907,mungewell,2022-06-22T00:46:43Z,2022-06-22T00:46:43Z,NONE,"[log.txt](https://github.com/simonw/sqlite-utils/files/8953589/log.txt) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279144769,Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto', https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1162498734,https://api.github.com/repos/simonw/sqlite-utils/issues/448,1162498734,IC_kwDOCGYnMM5FSlKu,236907,mungewell,2022-06-22T00:43:45Z,2022-06-22T00:43:45Z,NONE,"Attempted to test on a machine with a new version of Python, but install failed with an error message for the 'click' package. ``` C:\WINDOWS\system32>""c:\Program Files\Python310\python.exe"" Python 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)] on win32 Type ""help"", ""copyright"", ""credits"" or ""license"" for more information. >>> quit() C:\WINDOWS\system32>cd C:\Users\swood\Downloads\sqlite-utils-main-20220621\sqlite-utils-main C:\Users\swood\Downloads\sqlite-utils-main-20220621\sqlite-utils-main>""c:\Program Files\Python310\python.exe"" setup.py install running install running bdist_egg running egg_info ... Installed c:\program files\python310\lib\site-packages\click_default_group_wheel-1.2.2-py3.10.egg Searching for click Downloading https://files.pythonhosted.org/packages/3d/da/f3bbf30f7e71d881585d598f67f4424b2cc4c68f39849542e81183218017/click-default-group-wheel-1.2.2.tar.gz#sha256=e90da42d92c03e88a12ed0c0b69c8a29afb5d36e3dc8d29c423ba4219e6d7747 Best match: click default-group-wheel-1.2.2 Processing click-default-group-wheel-1.2.2.tar.gz Writing C:\Users\swood\AppData\Local\Temp\easy_install-aiaj0_eh\click-default-group-wheel-1.2.2\setup.cfg Running click-default-group-wheel-1.2.2\setup.py -q bdist_egg --dist-dir C:\Users\swood\AppData\Local\Temp\easy_install-aiaj0_eh\click-default-group-wheel-1.2.2\egg-dist-tmp-z61a4h8n zip_safe flag not set; analyzing archive contents... removing 'c:\program files\python310\lib\site-packages\click_default_group_wheel-1.2.2-py3.10.egg' (and everything under it) Copying click_default_group_wheel-1.2.2-py3.10.egg to c:\program files\python310\lib\site-packages click-default-group-wheel 1.2.2 is already the active version in easy-install.pth Installed c:\program files\python310\lib\site-packages\click_default_group_wheel-1.2.2-py3.10.egg error: The 'click' distribution was not found and is required by click-default-group-wheel, sqlite-utils ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279144769,Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto', https://github.com/simonw/sqlite-utils/issues/447#issuecomment-1162186856,https://api.github.com/repos/simonw/sqlite-utils/issues/447,1162186856,IC_kwDOCGYnMM5FRZBo,9599,simonw,2022-06-21T18:48:46Z,2022-06-21T18:48:46Z,OWNER,"That fixed it: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1278571700,Incorrect syntax highlighting in docs CLI reference, https://github.com/simonw/sqlite-utils/issues/447#issuecomment-1161869859,https://api.github.com/repos/simonw/sqlite-utils/issues/447,1161869859,IC_kwDOCGYnMM5FQLoj,9599,simonw,2022-06-21T15:00:42Z,2022-06-21T15:00:42Z,OWNER,Deploying that to https://sqlite-utils.datasette.io/en/latest/cli-reference.html#insert,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1278571700,Incorrect syntax highlighting in docs CLI reference, https://github.com/simonw/sqlite-utils/issues/447#issuecomment-1161857806,https://api.github.com/repos/simonw/sqlite-utils/issues/447,1161857806,IC_kwDOCGYnMM5FQIsO,9599,simonw,2022-06-21T14:55:51Z,2022-06-21T14:58:14Z,OWNER,"https://stackoverflow.com/a/44379513 suggests that the fix is: .. code-block:: text Or set this in `conf.py`: highlight_language = ""none"" I like that better - I don't like that all `::` blocks default to being treated as Python code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1278571700,Incorrect syntax highlighting in docs CLI reference, https://github.com/simonw/sqlite-utils/issues/446#issuecomment-1162234441,https://api.github.com/repos/simonw/sqlite-utils/issues/446,1162234441,IC_kwDOCGYnMM5FRkpJ,9599,simonw,2022-06-21T19:28:35Z,2022-06-21T19:28:35Z,OWNER,"`just -l` now does this: ``` % just -l Available recipes: black # Apply Black cog # Rebuild docs with cog default # Run tests and linters lint # Run linters: black, flake8, mypy, cog test *options # Run pytest with supplied options ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1277328147,Use Just to automate running tests and linters locally, https://github.com/simonw/sqlite-utils/issues/446#issuecomment-1160798645,https://api.github.com/repos/simonw/sqlite-utils/issues/446,1160798645,IC_kwDOCGYnMM5FMGG1,9599,simonw,2022-06-20T19:55:34Z,2022-06-20T19:55:34Z,OWNER,"`just` now defaults to running the tests and linters. `just test` runs the tests - it can take arguments, e.g. `just test -k transform` `just lint` runs all of the linters. `just black` applies Black. In all case it assumes you are using `pipenv`, at least for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1277328147,Use Just to automate running tests and linters locally, https://github.com/simonw/sqlite-utils/issues/445#issuecomment-1160794175,https://api.github.com/repos/simonw/sqlite-utils/issues/445,1160794175,IC_kwDOCGYnMM5FMFA_,9599,simonw,2022-06-20T19:49:02Z,2022-06-20T19:49:02Z,OWNER,"New documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#detecting-column-types-using-typetracker - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-typetracker","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1277295119,`sqlite_utils.utils.TypeTracker` should be a documented API, https://github.com/simonw/sqlite-utils/issues/445#issuecomment-1160793114,https://api.github.com/repos/simonw/sqlite-utils/issues/445,1160793114,IC_kwDOCGYnMM5FMEwa,9599,simonw,2022-06-20T19:47:36Z,2022-06-20T19:47:36Z,OWNER,I also added inline documentation and types: https://github.com/simonw/sqlite-utils/blob/773f2b6b20622bb986984a1c3161d5b3aaa1046b/sqlite_utils/utils.py#L318-L360,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1277295119,`sqlite_utils.utils.TypeTracker` should be a documented API, https://github.com/simonw/sqlite-utils/issues/445#issuecomment-1160763268,https://api.github.com/repos/simonw/sqlite-utils/issues/445,1160763268,IC_kwDOCGYnMM5FL9eE,9599,simonw,2022-06-20T19:09:21Z,2022-06-20T19:09:21Z,OWNER,Code to document: https://github.com/simonw/sqlite-utils/blob/3fbe8a784cc2f3fa0bfa8612fec9752ff9068a2b/sqlite_utils/utils.py#L318-L331,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1277295119,`sqlite_utils.utils.TypeTracker` should be a documented API, https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155966234,https://api.github.com/repos/simonw/sqlite-utils/issues/444,1155966234,IC_kwDOCGYnMM5E5qUa,9599,simonw,2022-06-15T04:18:05Z,2022-06-15T04:18:05Z,OWNER,I'm going to push a branch with my not-yet-working code (which does at least include a test).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1271426387,CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool, https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155815956,https://api.github.com/repos/simonw/sqlite-utils/issues/444,1155815956,IC_kwDOCGYnMM5E5FoU,9599,simonw,2022-06-14T23:49:56Z,2022-07-07T16:39:18Z,OWNER,"Yeah my initial implementation there makes no sense: ```python csv_reader_args = {""dialect"": dialect} if delimiter: csv_reader_args[""delimiter""] = delimiter if quotechar: csv_reader_args[""quotechar""] = quotechar reader = _extra_key_strategy( csv_std.reader(decoded, **csv_reader_args), ignore_extras, extras_key ) first_row = next(reader) if no_headers: headers = [""untitled_{}"".format(i + 1) for i in range(len(first_row))] reader = itertools.chain([first_row], reader) else: headers = first_row docs = (dict(zip(headers, row)) for row in reader) ``` Because my `_extra_key_strategy()` helper function is designed to work against `csv.DictReader` - not against `csv.reader()` which returns a sequence of lists, not a sequence of dictionaries. In fact, what's happening here is that `dict(zip(headers, row))` is ignoring anything in the row that doesn't correspond to a header: ```pycon >>> list(zip([""a"", ""b""], [1, 2, 3])) [('a', 1), ('b', 2)] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1271426387,CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool, https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155815186,https://api.github.com/repos/simonw/sqlite-utils/issues/444,1155815186,IC_kwDOCGYnMM5E5FcS,9599,simonw,2022-06-14T23:48:16Z,2022-06-14T23:48:16Z,OWNER,"This is tricky to implement because of this code: https://github.com/simonw/sqlite-utils/blob/b8af3b96f5c72317cc8783dc296a94f6719987d9/sqlite_utils/cli.py#L938-L945 It's reconstructing each document using the known headers here: `docs = (dict(zip(headers, row)) for row in reader)` So my first attempt at this - the diff here - did not have the desired result: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 86eddfb..00b920b 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -6,7 +6,7 @@ import hashlib import pathlib import sqlite_utils from sqlite_utils.db import AlterError, BadMultiValues, DescIndex -from sqlite_utils.utils import maximize_csv_field_size_limit +from sqlite_utils.utils import maximize_csv_field_size_limit, _extra_key_strategy from sqlite_utils import recipes import textwrap import inspect @@ -797,6 +797,15 @@ _import_options = ( ""--encoding"", help=""Character encoding for input, defaults to utf-8"", ), + click.option( + ""--ignore-extras"", + is_flag=True, + help=""If a CSV line has more than the expected number of values, ignore the extras"", + ), + click.option( + ""--extras-key"", + help=""If a CSV line has more than the expected number of values put them in a list in this column"", + ), ) @@ -885,6 +894,8 @@ def insert_upsert_implementation( sniff, no_headers, encoding, + ignore_extras, + extras_key, batch_size, alter, upsert, @@ -909,6 +920,10 @@ def insert_upsert_implementation( raise click.ClickException(""--flatten cannot be used with --csv or --tsv"") if encoding and not (csv or tsv): raise click.ClickException(""--encoding must be used with --csv or --tsv"") + if ignore_extras and extras_key: + raise click.ClickException( + ""--ignore-extras and --extras-key cannot be used together"" + ) if pk and len(pk) == 1: pk = pk[0] encoding = encoding or ""utf-8-sig"" @@ -935,7 +950,9 @@ def insert_upsert_implementation( csv_reader_args[""delimiter""] = delimiter if quotechar: csv_reader_args[""quotechar""] = quotechar - reader = csv_std.reader(decoded, **csv_reader_args) + reader = _extra_key_strategy( + csv_std.reader(decoded, **csv_reader_args), ignore_extras, extras_key + ) first_row = next(reader) if no_headers: headers = [""untitled_{}"".format(i + 1) for i in range(len(first_row))] @@ -1101,6 +1118,8 @@ def insert( sniff, no_headers, encoding, + ignore_extras, + extras_key, batch_size, alter, detect_types, @@ -1176,6 +1195,8 @@ def insert( sniff, no_headers, encoding, + ignore_extras, + extras_key, batch_size, alter=alter, upsert=False, @@ -1214,6 +1235,8 @@ def upsert( sniff, no_headers, encoding, + ignore_extras, + extras_key, alter, not_null, default, @@ -1254,6 +1277,8 @@ def upsert( sniff, no_headers, encoding, + ignore_extras, + extras_key, batch_size, alter=alter, upsert=True, @@ -1297,6 +1322,8 @@ def bulk( sniff, no_headers, encoding, + ignore_extras, + extras_key, load_extension, ): """""" @@ -1331,6 +1358,8 @@ def bulk( sniff=sniff, no_headers=no_headers, encoding=encoding, + ignore_extras=ignore_extras, + extras_key=extras_key, batch_size=batch_size, alter=False, upsert=False, ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1271426387,CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool, https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155804591,https://api.github.com/repos/simonw/sqlite-utils/issues/444,1155804591,IC_kwDOCGYnMM5E5C2v,9599,simonw,2022-06-14T23:28:36Z,2022-06-14T23:28:36Z,OWNER,I'm going with `--extras-key` and `--ignore-extras` as the two new options.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1271426387,CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool, https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155804459,https://api.github.com/repos/simonw/sqlite-utils/issues/444,1155804459,IC_kwDOCGYnMM5E5C0r,9599,simonw,2022-06-14T23:28:18Z,2022-06-14T23:28:18Z,OWNER,"I think these become part of the `_import_options` list which is used in a few places: https://github.com/simonw/sqlite-utils/blob/b8af3b96f5c72317cc8783dc296a94f6719987d9/sqlite_utils/cli.py#L765-L800","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1271426387,CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool, https://github.com/simonw/sqlite-utils/issues/443#issuecomment-1160794604,https://api.github.com/repos/simonw/sqlite-utils/issues/443,1160794604,IC_kwDOCGYnMM5FMFHs,9599,simonw,2022-06-20T19:49:37Z,2022-06-20T19:49:37Z,OWNER,Also now shows up here: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-rows-from-file,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1269998342,Make `utils.rows_from_file()` a documented API, https://github.com/simonw/sqlite-utils/issues/443#issuecomment-1155672522,https://api.github.com/repos/simonw/sqlite-utils/issues/443,1155672522,IC_kwDOCGYnMM5E4inK,9599,simonw,2022-06-14T20:18:58Z,2022-06-14T20:18:58Z,OWNER,New documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1269998342,Make `utils.rows_from_file()` a documented API, https://github.com/simonw/sqlite-utils/issues/442#issuecomment-1155748444,https://api.github.com/repos/simonw/sqlite-utils/issues/442,1155748444,IC_kwDOCGYnMM5E41Jc,9599,simonw,2022-06-14T21:55:15Z,2022-06-14T21:55:15Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#setting-the-maximum-csv-field-size-limit,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1269886084,`maximize_csv_field_size_limit()` utility function, https://github.com/simonw/sqlite-utils/issues/442#issuecomment-1155714131,https://api.github.com/repos/simonw/sqlite-utils/issues/442,1155714131,IC_kwDOCGYnMM5E4sxT,9599,simonw,2022-06-14T21:07:50Z,2022-06-14T21:07:50Z,OWNER,"Here's the commit where I added that originally, including a test: https://github.com/simonw/sqlite-utils/commit/1a93b72ba710ea2271eaabc204685a27d2469374","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1269886084,`maximize_csv_field_size_limit()` utility function, https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1155750270,https://api.github.com/repos/simonw/sqlite-utils/issues/441,1155750270,IC_kwDOCGYnMM5E41l-,9599,simonw,2022-06-14T21:57:57Z,2022-06-14T21:57:57Z,OWNER,I added `where=` and `where_args=` parameters to that `.search()` method - updated documentation is here: https://sqlite-utils.datasette.io/en/latest/python-api.html#searching-with-table-search,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1257724585,Combining `rows_where()` and `search()` to limit which rows are searched, https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1155515426,https://api.github.com/repos/simonw/sqlite-utils/issues/441,1155515426,IC_kwDOCGYnMM5E38Qi,1448859,betatim,2022-06-14T17:53:43Z,2022-06-14T17:53:43Z,NONE,"That would be handy (additional where filters) but I think the trick with the `with` statement is already an order of magnitude better than what I had thought of, so my problem is solved by it (plus I got to learn about `with` today!)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1257724585,Combining `rows_where()` and `search()` to limit which rows are searched, https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1155421299,https://api.github.com/repos/simonw/sqlite-utils/issues/441,1155421299,IC_kwDOCGYnMM5E3lRz,9599,simonw,2022-06-14T16:23:52Z,2022-06-14T16:23:52Z,OWNER,Actually I have a thought for something that could help here: I could add a mechanism for inserting additional where filters and parameters into that `.search()` method.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1257724585,Combining `rows_where()` and `search()` to limit which rows are searched, https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1154373361,https://api.github.com/repos/simonw/sqlite-utils/issues/441,1154373361,IC_kwDOCGYnMM5Ezlbx,9599,simonw,2022-06-13T20:01:25Z,2022-06-13T20:01:25Z,OWNER,"Yeah, at the moment the best way to do this is with `search_sql()`, but you're right it really isn't very intuitive. Here's how I would do this, using a CTE trick to combine the queries: ```python search_sql = db[""articles""].search_sql(columns=[""title"", ""author""])) sql = f"""""" with search_results as ({search_sql}) select * from search_results where owner = :owner """""" results = db.query(sql, {""query"": ""my search query"", ""owner"": ""my owner""}) ``` I'm not sure if `sqlite-utils` should ever evolve to provide a better way of doing this kind of thing to be honest - if it did, it would turn into more of an ORM. Something like [PeeWee](http://docs.peewee-orm.com/en/latest/) may be a better option here.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1257724585,Combining `rows_where()` and `search()` to limit which rows are searched, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155767915,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155767915,IC_kwDOCGYnMM5E455r,9599,simonw,2022-06-14T22:22:27Z,2022-06-14T22:22:27Z,OWNER,I forgot to add equivalents of `extras_key=` and `ignore_extras=` to the CLI tool - will do that in a separate issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155672675,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155672675,IC_kwDOCGYnMM5E4ipj,9599,simonw,2022-06-14T20:19:07Z,2022-06-14T20:19:07Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155666672,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155666672,IC_kwDOCGYnMM5E4hLw,9599,simonw,2022-06-14T20:11:52Z,2022-06-14T20:11:52Z,OWNER,I'm going to rename `restkey` to `extras_key` for consistency with `ignore_extras`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155389614,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155389614,IC_kwDOCGYnMM5E3diu,9599,simonw,2022-06-14T15:54:03Z,2022-06-14T15:54:03Z,OWNER,"Filed an issue against `python/typeshed`: - https://github.com/python/typeshed/issues/8075","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155358637,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155358637,IC_kwDOCGYnMM5E3V-t,9599,simonw,2022-06-14T15:31:34Z,2022-06-14T15:31:34Z,OWNER,"Getting this past `mypy` is really hard! ``` % mypy sqlite_utils sqlite_utils/utils.py:189: error: No overload variant of ""pop"" of ""MutableMapping"" matches argument type ""None"" sqlite_utils/utils.py:189: note: Possible overload variants: sqlite_utils/utils.py:189: note: def pop(self, key: str) -> str sqlite_utils/utils.py:189: note: def [_T] pop(self, key: str, default: Union[str, _T] = ...) -> Union[str, _T] ``` That's because of this line: row.pop(key=None) Which is legit here - we have a dictionary where one of the keys is `None` and we want to remove that key. But the baked in type is apparently `def pop(self, key: str) -> str`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155350755,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155350755,IC_kwDOCGYnMM5E3UDj,9599,simonw,2022-06-14T15:25:18Z,2022-06-14T15:25:18Z,OWNER,"That broke `mypy`: `sqlite_utils/utils.py:229: error: Incompatible types in assignment (expression has type ""Iterable[Dict[Any, Any]]"", variable has type ""DictReader[str]"")`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155317293,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155317293,IC_kwDOCGYnMM5E3L4t,9599,simonw,2022-06-14T15:04:01Z,2022-06-14T15:04:01Z,OWNER,"I think that's unavoidable: it looks like `csv.Sniffer` only works if you feed it a CSV file with an equal number of values in each row, which is understandable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155310521,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155310521,IC_kwDOCGYnMM5E3KO5,9599,simonw,2022-06-14T14:58:50Z,2022-06-14T14:58:50Z,OWNER,"Interesting challenge in writing tests for this: if you give `csv.Sniffer` a short example with an invalid row in it sometimes it picks the wrong delimiter! id,name\r\n1,Cleo,oops It decided the delimiter there was `e`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154475454,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154475454,IC_kwDOCGYnMM5Ez-W-,9599,simonw,2022-06-13T21:52:03Z,2022-06-13T21:52:03Z,OWNER,The exception will be called `RowError`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154474482,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154474482,IC_kwDOCGYnMM5Ez-Hy,9599,simonw,2022-06-13T21:50:59Z,2022-06-13T21:51:24Z,OWNER,"Decision: I'm going to default to raising an exception if a row has too many values in it. You'll be able to pass `ignore_extras=True` to ignore those extra values, or pass `restkey=""the_rest""` to stick them in a list in the `restkey` column.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154457893,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154457893,IC_kwDOCGYnMM5Ez6El,9599,simonw,2022-06-13T21:29:02Z,2022-06-13T21:29:02Z,OWNER,"Here's the current function signature for `rows_from_file()`: https://github.com/simonw/sqlite-utils/blob/26e6d2622c57460a24ffdd0128bbaac051d51a5f/sqlite_utils/utils.py#L174-L179","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154457028,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154457028,IC_kwDOCGYnMM5Ez53E,9599,simonw,2022-06-13T21:28:03Z,2022-06-13T21:28:03Z,OWNER,"Whatever I decide, I can implement it in `rows_from_file()`, maybe as an optional parameter - then decide how to call it from the `sqlite-utils insert` CLI (perhaps with a new option there too).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154456183,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154456183,IC_kwDOCGYnMM5Ez5p3,9599,simonw,2022-06-13T21:26:55Z,2022-06-13T21:26:55Z,OWNER,"So I need to make a design decision here: what should `sqlite-utils` do with CSV files that have rows with more values than there are headings? Some options: - Ignore those extra fields entirely - silently drop that data. I'm not keen on this. - Throw an error. The library does this already, but the error is incomprehensible - it could turn into a useful, human-readable error instead. - Put the data in a JSON list in a column with a known name (`None` is not a valid column name, so not that). This could be something like `_restkey` or `_values_with_no_heading`. This feels like a better option, but I'd need to carefully pick a name for it - and come up with an answer for the question of what to do if the CSV file being important already uses that heading name for something else.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154454127,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154454127,IC_kwDOCGYnMM5Ez5Jv,9599,simonw,2022-06-13T21:24:18Z,2022-06-13T21:24:18Z,OWNER,"That weird behaviour is documented here: https://docs.python.org/3/library/csv.html#csv.DictReader > If a row has more fields than fieldnames, the remaining data is put in a list and stored with the fieldname specified by *restkey* (which defaults to `None`). If a non-blank row has fewer fields than fieldnames, the missing values are filled-in with the value of *restval* (which defaults to `None`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154453319,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154453319,IC_kwDOCGYnMM5Ez49H,9599,simonw,2022-06-13T21:23:16Z,2022-06-13T21:23:16Z,OWNER,"Aha! I think I see what's happening here. Here's what `DictReader` does if one of the lines has too many items in it: ```pycon >>> import csv, io >>> list(csv.DictReader(io.StringIO(""id,name\n1,Cleo,nohead\n2,Barry""))) [{'id': '1', 'name': 'Cleo', None: ['nohead']}, {'id': '2', 'name': 'Barry'}] ``` See how that row with too many items gets this: `[{'id': '1', 'name': 'Cleo', None: ['nohead']}` That's a `None` for the key and (weirdly) a list containing the single item for the value! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154449442,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154449442,IC_kwDOCGYnMM5Ez4Ai,9599,simonw,2022-06-13T21:18:26Z,2022-06-13T21:20:12Z,OWNER,"Here are full steps to replicate the bug: ```python from urllib.request import urlopen import sqlite_utils db = sqlite_utils.Database(memory=True) with urlopen(""https://artsdatabanken.no/Fab2018/api/export/csv"") as fab: reader, other = sqlite_utils.utils.rows_from_file(fab, encoding=""utf-16le"") db[""fab2018""].insert_all(reader, pk=""Id"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154396400,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154396400,IC_kwDOCGYnMM5EzrDw,9599,simonw,2022-06-13T20:28:25Z,2022-06-13T20:28:25Z,OWNER,"Fixing that `key` thing (to ignore any key that is `None`) revealed a new bug: ``` File ~/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py:376, in hash_record(record, keys) 373 if keys is not None: 374 to_hash = {key: record[key] for key in keys} 375 return hashlib.sha1( --> 376 json.dumps(to_hash, separators=("","", "":""), sort_keys=True, default=repr).encode( 377 ""utf8"" 378 ) 379 ).hexdigest() File ~/.pyenv/versions/3.8.2/lib/python3.8/json/__init__.py:234, in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw) 232 if cls is None: 233 cls = JSONEncoder --> 234 return cls( 235 skipkeys=skipkeys, ensure_ascii=ensure_ascii, 236 check_circular=check_circular, allow_nan=allow_nan, indent=indent, 237 separators=separators, default=default, sort_keys=sort_keys, 238 **kw).encode(obj) File ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:199, in JSONEncoder.encode(self, o) 195 return encode_basestring(o) 196 # This doesn't pass the iterator directly to ''.join() because the 197 # exceptions aren't as detailed. The list call should be roughly 198 # equivalent to the PySequence_Fast that ''.join() would do. --> 199 chunks = self.iterencode(o, _one_shot=True) 200 if not isinstance(chunks, (list, tuple)): 201 chunks = list(chunks) File ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:257, in JSONEncoder.iterencode(self, o, _one_shot) 252 else: 253 _iterencode = _make_iterencode( 254 markers, self.default, _encoder, self.indent, floatstr, 255 self.key_separator, self.item_separator, self.sort_keys, 256 self.skipkeys, _one_shot) --> 257 return _iterencode(o, 0) TypeError: '<' not supported between instances of 'NoneType' and 'str' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154387591,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154387591,IC_kwDOCGYnMM5Ezo6H,9599,simonw,2022-06-13T20:17:51Z,2022-06-13T20:17:51Z,OWNER,I don't understand why that works but calling `insert_all()` does not.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154386795,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154386795,IC_kwDOCGYnMM5Ezotr,9599,simonw,2022-06-13T20:16:53Z,2022-06-13T20:16:53Z,OWNER,"Steps to demonstrate that `sqlite-utils insert` is not affected: ```bash curl -o artsdatabanken.csv https://artsdatabanken.no/Fab2018/api/export/csv sqlite-utils insert arts.db artsdatabanken artsdatabanken.csv --sniff --csv --encoding utf-16le ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154385916,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154385916,IC_kwDOCGYnMM5Ezof8,9599,simonw,2022-06-13T20:15:49Z,2022-06-13T20:15:49Z,OWNER,`rows_from_file()` isn't part of the documented API but maybe it should be!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/44#issuecomment-514162930,https://api.github.com/repos/simonw/sqlite-utils/issues/44,514162930,MDEyOklzc3VlQ29tbWVudDUxNDE2MjkzMA==,9599,simonw,2019-07-23T11:03:03Z,2019-07-23T11:03:16Z,OWNER,"I considered keyword arguments for this, but I am going with a dictionary instead - for two reasons: * leaves the option to add extra keyword arguments for further options later * supports column names that are not valid keyword arguments ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471628483,Utilities for building lookup tables, https://github.com/simonw/sqlite-utils/issues/44#issuecomment-514162499,https://api.github.com/repos/simonw/sqlite-utils/issues/44,514162499,MDEyOklzc3VlQ29tbWVudDUxNDE2MjQ5OQ==,9599,simonw,2019-07-23T11:01:45Z,2019-07-23T11:01:45Z,OWNER,"`id = table.lookup({""name"":""Cleo""})` - If table does not exist, create it with id, name where name is unique - If table does exist, add unique name column - If table and column exist, add unique constraint - throw error if impossible - now either insert the new row or return the existing ID","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471628483,Utilities for building lookup tables, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155953345,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155953345,IC_kwDOCGYnMM5E5nLB,9599,simonw,2022-06-15T03:53:43Z,2022-06-15T03:53:43Z,OWNER,"I tried fixing this by using `.tell()` to read the file position as I was iterating through it: ```diff diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py index d2ccc5f..29ad12e 100644 --- a/sqlite_utils/utils.py +++ b/sqlite_utils/utils.py @@ -149,10 +149,13 @@ class UpdateWrapper: def __init__(self, wrapped, update): self._wrapped = wrapped self._update = update + self._tell = wrapped.tell() def __iter__(self): for line in self._wrapped: - self._update(len(line)) + tell = self._wrapped.tell() + self._update(self._tell - tell) + self._tell = tell yield line ``` This did not work - I get this error: ``` File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 206, in _extra_key_strategy for row in reader: File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 156, in __iter__ tell = self._wrapped.tell() OSError: telling position disabled by next() call ``` It looks like you can't use `.tell()` during iteration: https://stackoverflow.com/questions/29618936/how-to-solve-oserror-telling-position-disabled-by-next-call","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155789101,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155789101,IC_kwDOCGYnMM5E4_Et,9599,simonw,2022-06-14T23:00:45Z,2022-06-14T23:00:45Z,OWNER,"I'm going to mark this as ""help wanted"" and leave it open. I'm glad that it's not actually a bug where errors get swallowed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155788944,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155788944,IC_kwDOCGYnMM5E4_CQ,9599,simonw,2022-06-14T23:00:24Z,2022-06-14T23:00:24Z,OWNER,"The progress bar only works if the file-like object passed to it has a `fp.fileno()` that isn't 0 (for stdin) - that's how it detects that the file is something which it can measure the size of in order to show progress. If we know the file size in bytes AND we know the character encoding, can we change `UpdateWrapper` to update the number of bytes-per-character instead? I don't think so: I can't see a way of definitively saying ""for this encoding the number of bytes per character is X"" - and in fact I'm pretty sure that question doesn't even make sense since variable-length encodings exist. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155784284,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155784284,IC_kwDOCGYnMM5E495c,9599,simonw,2022-06-14T22:51:03Z,2022-06-14T22:52:13Z,OWNER,"Yes, this is the problem. The progress bar length is set to the length in bytes of the file - `os.path.getsize(file.name)` - but it's then incremented by the length of each DECODED line in turn. So if the file is in `utf-16-le` (twice the size of `utf-8`) the progress bar will finish at 50%!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155782835,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155782835,IC_kwDOCGYnMM5E49iz,9599,simonw,2022-06-14T22:48:22Z,2022-06-14T22:49:53Z,OWNER,"Here's the code that implements the progress bar in question: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/cli.py#L918-L932 It calls `file_progress()` which looks like this: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/utils.py#L159-L175 Which uses this: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/utils.py#L148-L156","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155781399,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155781399,IC_kwDOCGYnMM5E49MX,9599,simonw,2022-06-14T22:45:41Z,2022-06-14T22:45:41Z,OWNER,TIL how to use `iconv`: https://til.simonwillison.net/linux/iconv,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155776023,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155776023,IC_kwDOCGYnMM5E474X,9599,simonw,2022-06-14T22:36:07Z,2022-06-14T22:36:07Z,OWNER,"Wait! The arguments in that are the wrong way round. This is correct: sqlite-utils insert --csv --delimiter "";"" --encoding ""utf-16-le"" test.db test csv It still outputs the following: [------------------------------------] 0% [#################-------------------] 49% 00:00:02% But it creates a `test.db` file that is 6.2MB. That database has 3141 rows in it: ``` % sqlite-utils tables test.db --counts -t table count ------- ------- test 3142 ``` I converted that `csv` file to utf-8 like so: iconv -f UTF-16LE -t UTF-8 csv > utf8.csv And it contains 3142 lines: ``` % wc -l utf8.csv 3142 utf8.csv ``` So my hunch here is that the problem is actually that the progress bar doesn't know how to correctly measure files in `utf-16-le` encoding!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155772244,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155772244,IC_kwDOCGYnMM5E469U,9599,simonw,2022-06-14T22:30:03Z,2022-06-14T22:30:03Z,OWNER,"Tried this: ``` % python -i $(which sqlite-utils) insert --csv --delimiter "";"" --encoding ""utf-16-le"" test test.db csv [------------------------------------] 0% [#################-------------------] 49% 00:00:01Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1072, in main ctx.exit() File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 692, in exit raise Exit(code) click.exceptions.Exit: 0 During handling of the above exception, another exception occurred: Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils"", line 33, in sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1137, in __call__ return self.main(*args, **kwargs) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1090, in main sys.exit(e.exit_code) SystemExit: 0 >>> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155771462,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155771462,IC_kwDOCGYnMM5E46xG,9599,simonw,2022-06-14T22:28:38Z,2022-06-14T22:28:38Z,OWNER,"Maybe this isn't a CSV field value problem - I tried this patch and didn't seem to hit the new breakpoints: ```diff diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py index d2ccc5f..f1b823a 100644 --- a/sqlite_utils/utils.py +++ b/sqlite_utils/utils.py @@ -204,13 +204,17 @@ def _extra_key_strategy( # DictReader adds a 'None' key with extra row values if None not in row: yield row - elif ignore_extras: + continue + else: + breakpoint() + if ignore_extras: # ignoring row.pop(none) because of this issue: # https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155358637 row.pop(None) # type: ignore yield row elif not extras_key: extras = row.pop(None) # type: ignore + breakpoint() raise RowError( ""Row {} contained these extra values: {}"".format(row, extras) ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155769216,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155769216,IC_kwDOCGYnMM5E46OA,9599,simonw,2022-06-14T22:24:49Z,2022-06-14T22:25:06Z,OWNER,"I have a hunch that this crash may be caused by a CSV value which is too long, as addressed at the library level in: - #440 But not yet addressed in the CLI tool, see: - #444 Either way though, I really don't like that errors like this are swallowed!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155767202,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155767202,IC_kwDOCGYnMM5E45ui,9599,simonw,2022-06-14T22:21:10Z,2022-06-14T22:21:10Z,OWNER,"I can't figure out why that error is being swallowed like that. The most likely culprit was this code: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/cli.py#L1021-L1043 But I tried changing it like this: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 86eddfb..ed26fdd 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -1023,6 +1023,7 @@ def insert_upsert_implementation( docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs ) except Exception as e: + raise if ( isinstance(e, sqlite3.OperationalError) and e.args ``` And your steps to reproduce still got to 49% and then failed silently.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1139426398,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1139426398,IC_kwDOCGYnMM5D6kRe,4068,frafra,2022-05-27T09:04:05Z,2022-05-27T10:44:54Z,NONE,"This code works: ```python import csv import sqlite_utils db = sqlite_utils.Database(""test.db"") reader = csv.DictReader(open(""csv"", encoding=""utf-16-le"").read().split(""\r\n""), delimiter="";"") db[""test""].insert_all(reader, pk=""Id"") ``` I used `iconv` to change the encoding; sqlite-utils can import the resulting file, even if it stops at 98 %: ``` sqlite-utils insert --csv test test.db clean [------------------------------------] 0% [###################################-] 98% 00:00:00 ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/438#issuecomment-1139392769,https://api.github.com/repos/simonw/sqlite-utils/issues/438,1139392769,IC_kwDOCGYnMM5D6cEB,4068,frafra,2022-05-27T08:21:53Z,2022-05-27T08:21:53Z,NONE,Argument were specified in the wrong order. `PATH TABLE FILE` can be misleading :),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250161887,illegal UTF-16 surrogate, https://github.com/simonw/sqlite-utils/issues/438#issuecomment-1139379923,https://api.github.com/repos/simonw/sqlite-utils/issues/438,1139379923,IC_kwDOCGYnMM5D6Y7T,4068,frafra,2022-05-27T08:05:01Z,2022-05-27T08:05:01Z,NONE,"I tried to debug it using `pdb`, but it looks `sqlite-utils` catches the exception, so it is not quick to figure out where the failure is happening.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250161887,illegal UTF-16 surrogate, https://github.com/simonw/sqlite-utils/issues/435#issuecomment-1133417432,https://api.github.com/repos/simonw/sqlite-utils/issues/435,1133417432,IC_kwDOCGYnMM5DjpPY,9599,simonw,2022-05-20T21:56:10Z,2022-05-20T21:56:10Z,OWNER,"Before: ![sqlite-utils-datasette-io-en-stable-reference-html](https://user-images.githubusercontent.com/9599/169617623-457b4c01-3713-4e5c-b3c3-8574a2214238.png) After: ![sqlite-utils-datasette-io-en-latest-reference-html](https://user-images.githubusercontent.com/9599/169617666-ba388167-36b8-4a3e-be35-931ee2ab6b3b.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243704847,Switch to Furo documentation theme, https://github.com/simonw/sqlite-utils/issues/435#issuecomment-1133416698,https://api.github.com/repos/simonw/sqlite-utils/issues/435,1133416698,IC_kwDOCGYnMM5DjpD6,9599,simonw,2022-05-20T21:54:43Z,2022-05-20T21:54:43Z,OWNER,Done: https://sqlite-utils.datasette.io/en/latest/reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243704847,Switch to Furo documentation theme, https://github.com/simonw/sqlite-utils/issues/434#issuecomment-1155801812,https://api.github.com/repos/simonw/sqlite-utils/issues/434,1155801812,IC_kwDOCGYnMM5E5CLU,9599,simonw,2022-06-14T23:23:32Z,2022-06-14T23:23:32Z,OWNER,"Since table names can be quoted like this: ```sql CREATE VIRTUAL TABLE ""searchable_fts"" USING FTS4 (text1, text2, [name with . and spaces], content=""searchable"") ``` OR like this: ```sql CREATE VIRTUAL TABLE ""searchable_fts"" USING FTS4 (text1, text2, [name with . and spaces], content=[searchable]) ``` This fix looks to be correct to me (copying from the updated `test_with_trace()` test): ```python ( ""SELECT name FROM sqlite_master\n"" "" WHERE rootpage = 0\n"" "" AND (\n"" "" sql LIKE :like\n"" "" OR sql LIKE :like2\n"" "" OR (\n"" "" tbl_name = :table\n"" "" AND sql LIKE '%VIRTUAL TABLE%USING FTS%'\n"" "" )\n"" "" )"", { ""like"": ""%VIRTUAL TABLE%USING FTS%content=[dogs]%"", ""like2"": '%VIRTUAL TABLE%USING FTS%content=""dogs""%', ""table"": ""dogs"", }, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243151184,`detect_fts()` identifies the wrong table if tables have names that are subsets of each other, https://github.com/simonw/sqlite-utils/issues/434#issuecomment-1155794149,https://api.github.com/repos/simonw/sqlite-utils/issues/434,1155794149,IC_kwDOCGYnMM5E5ATl,9599,simonw,2022-06-14T23:09:54Z,2022-06-14T23:09:54Z,OWNER,"A test that demonstrates the problem: ```python @pytest.mark.parametrize(""reverse_order"", (True, False)) def test_detect_fts_similar_tables(fresh_db, reverse_order): # https://github.com/simonw/sqlite-utils/issues/434 table1, table2 = (""demo"", ""demo2"") if reverse_order: table1, table2 = table2, table1 fresh_db[table1].insert({""title"": ""Hello""}).enable_fts( [""title""], fts_version=""FTS4"" ) fresh_db[table2].insert({""title"": ""Hello""}).enable_fts( [""title""], fts_version=""FTS4"" ) assert fresh_db[table1].detect_fts() == ""{}_fts"".format(table1) assert fresh_db[table2].detect_fts() == ""{}_fts"".format(table2) ``` The order matters - so this test currently passes in one direction and fails in the other: ``` > assert fresh_db[table2].detect_fts() == ""{}_fts"".format(table2) E AssertionError: assert 'demo2_fts' == 'demo_fts' E - demo_fts E + demo2_fts E ? + tests/test_introspect.py:53: AssertionError ========================================================================================= short test summary info ========================================================================================= FAILED tests/test_introspect.py::test_detect_fts_similar_tables[True] - AssertionError: assert 'demo2_fts' == 'demo_fts' =============================================================================== 1 failed, 1 passed, 855 deselected in 1.00s =============================================================================== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243151184,`detect_fts()` identifies the wrong table if tables have names that are subsets of each other, https://github.com/simonw/sqlite-utils/issues/434#issuecomment-1155791109,https://api.github.com/repos/simonw/sqlite-utils/issues/434,1155791109,IC_kwDOCGYnMM5E4_kF,9599,simonw,2022-06-14T23:04:40Z,2022-06-14T23:04:40Z,OWNER,"Definitely a bug - thanks for the detailed write-up! You're right, the code at fault is here: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L2213-L2231","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243151184,`detect_fts()` identifies the wrong table if tables have names that are subsets of each other, https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1640826795,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1640826795,IC_kwDOCGYnMM5hzQer,76528036,J450n-4-W,2023-07-18T19:08:50Z,2023-07-18T19:08:50Z,NONE,"Came here to report this, but instead I'll confirm the issue across two terminal emulators (Gnome Terminal and Alacritty) on Pop_OS! 22.04 (currently based on Ubuntu/Gnome). Also messes up the formatting of the terminal. Can also confirm that reset fixes it until the next sqlite-utils command. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor, https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1578840450,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1578840450,IC_kwDOCGYnMM5eGzGC,392720,jonafato,2023-06-06T14:09:04Z,2023-06-06T14:09:04Z,NONE,"I also ran into this recently. See below for a patch for one possible solution (tested via ""it works on my machine"", but I don't expect that this behavior would vary a whole lot across terminal emulators and shells). Another possible solution might be to subclass click's `ProgressBar` to keep the logic within the original context manager. Happy to send a PR or for this patch to serve as the basis for a fix that someone else authors. ```patch diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py index 06c1a4c..530a3a3 100644 --- a/sqlite_utils/utils.py +++ b/sqlite_utils/utils.py @@ -147,14 +147,23 @@ def decode_base64_values(doc): class UpdateWrapper: - def __init__(self, wrapped, update): + def __init__(self, wrapped, update, render_finish): self._wrapped = wrapped self._update = update + self._render_finish = render_finish def __iter__(self): - for line in self._wrapped: - self._update(len(line)) - yield line + return self + + def __next__(self): + try: + line = next(self._wrapped) + except StopIteration as e: + self._render_finish() + raise + + self._update(len(line)) + return line def read(self, size=-1): data = self._wrapped.read(size) @@ -178,7 +187,7 @@ def file_progress(file, silent=False, **kwargs): else: file_length = os.path.getsize(file.name) with click.progressbar(length=file_length, **kwargs) as bar: - yield UpdateWrapper(file, bar.update) + yield UpdateWrapper(file, bar.update, bar.render_finish) class Format(enum.Enum): ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",1239034903,CLI eats my cursor, https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1444474487,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1444474487,IC_kwDOCGYnMM5WGO53,167893,mcarpenter,2023-02-24T20:57:43Z,2023-02-24T22:22:18Z,CONTRIBUTOR,"I think I see what is happening here, although I haven't quite work out a fix yet. Usually: * `click.progressbar.render_progress()` renders the cursor invisible on each invocation (update of the bar) * When the progress bar goes out of scope, the `__exit()__` method is invoked, which calls `render_finish()` to make the cursor re-appear. (See terminal escape sequences `BEFORE_BAR` and `AFTER_BAR` in click). However the sqlite-utils `utils.file_progress` context manager wraps `click.progressbar` and yields an instance of a helper class: ``` python @contextlib.contextmanager def file_progress(file, silent=False, **kwargs): ... with click.progressbar(length=file_length, **kwargs) as bar: yield UpdateWrapper(file, bar.update) ``` The yielded `UpdateWrapper` goes out of scope quickly and `click.progressbar.__exit__()` is called. The cursor is made un-invisible. Hoewever `bar` is still live and so when the caller iterates on the yielded wrapper this invokes the bar's update method, calling `render_progress()`, each time printing the ""make cursor invisible"" escape code. The `progressbar.__exit__` function is not called again, so the cursor doesn't re-appear. ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor, https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1416486796,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1416486796,IC_kwDOCGYnMM5Ubd-M,16236421,alecstein,2023-02-03T22:32:10Z,2023-02-03T22:32:10Z,NONE,Came here to say that I also have this issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor, https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1252898131,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1252898131,IC_kwDOCGYnMM5KrbVT,7908073,chapmanjacobd,2022-09-20T20:51:21Z,2022-09-20T20:56:07Z,CONTRIBUTOR,"When I run `reset` it fixes my terminal. I suspect it is related to the progress bar https://linux.die.net/man/1/reset ``` 950 1s /m/d/03_Downloads 🐑 echo $TERM xterm-kitty ▓░▒░ /m/d/03_Downloads 🌏 kitty -v kitty 0.26.2 created by Kovid Goyal $ sqlite-utils insert test.db facility facility-boundary-us-all.csv --csv blah blah blah (no offense) $ $ reset $ ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor, https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1155749696,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1155749696,IC_kwDOCGYnMM5E41dA,9599,simonw,2022-06-14T21:57:05Z,2022-06-14T21:57:05Z,OWNER,Marking this as help wanted because I can't figure out how to replicate it!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor, https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1139484453,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1139484453,IC_kwDOCGYnMM5D6ycl,4068,frafra,2022-05-27T10:20:08Z,2022-05-27T10:20:08Z,NONE,I can confirm. This only happens with sqlite-utils. I am using gnome-terminal with bash.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor, https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155764428,https://api.github.com/repos/simonw/sqlite-utils/issues/432,1155764428,IC_kwDOCGYnMM5E45DM,9599,simonw,2022-06-14T22:16:21Z,2022-06-14T22:16:21Z,OWNER,"Initial idea of how the `.table()` method would change: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 7a06304..3ecb40b 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -474,11 +474,12 @@ class Database: self._tracer(sql, None) return self.conn.executescript(sql) - def table(self, table_name: str, **kwargs) -> Union[""Table"", ""View""]: + def table(self, table_name: str, alias: Optional[str] = None, **kwargs) -> Union[""Table"", ""View""]: """""" Return a table object, optionally configured with default options. :param table_name: Name of the table + :param alias: The database alias to use, if referring to a table in another connected database """""" klass = View if table_name in self.view_names() else Table return klass(self, table_name, **kwargs) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1236693079,"Support `rows_where()`, `delete_where()` etc for attached alias databases", https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155764064,https://api.github.com/repos/simonw/sqlite-utils/issues/432,1155764064,IC_kwDOCGYnMM5E449g,9599,simonw,2022-06-14T22:15:44Z,2022-06-14T22:15:44Z,OWNER,"Implementing this would be a pretty big change - initial instinct is that I'd need to introduce a `self.alias` property to `Queryable` (the subclass of `Table` and `View`) and a new `self.name_with_alias` getter which returns `alias.tablename` if `alias` is set to a not-None value. Then I'd need to rewrite every piece of code like this: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L1161 To look like this instead: ```python sql = ""select {} from [{}]"".format(select, self.name_with_alias) ``` But some parts would be harder - for example: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L1227-L1231 Would have to know to query `alias.sqlite_master` instead. The cached table counts logic like this would need a bunch of changes too: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L644-L657","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1236693079,"Support `rows_where()`, `delete_where()` etc for attached alias databases", https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155759857,https://api.github.com/repos/simonw/sqlite-utils/issues/432,1155759857,IC_kwDOCGYnMM5E437x,9599,simonw,2022-06-14T22:09:07Z,2022-06-14T22:09:07Z,OWNER,"Third option, and I think the one I like the best: ```python rows = db.table(""tablename"", alias=""otherdb"").rows_where(alias=""otherdb"") ``` The `db.table(tablename)` method already exists as an alternative to `db[tablename]`: https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-table-configuration ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1236693079,"Support `rows_where()`, `delete_where()` etc for attached alias databases", https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155758664,https://api.github.com/repos/simonw/sqlite-utils/issues/432,1155758664,IC_kwDOCGYnMM5E43pI,9599,simonw,2022-06-14T22:07:50Z,2022-06-14T22:07:50Z,OWNER,"Another potential fix: add a `alias=` parameter to `rows_where()` and other similar methods. Then you could do this: ```python rows = db[""tablename""].rows_where(alias=""otherdb"") ``` This feels wrong to me: `db[""tablename""]` is the bit that is supposed to return a table object. Having part of what that table object is exist as a parameter to other methods is confusing. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1236693079,"Support `rows_where()`, `delete_where()` etc for attached alias databases", https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155756742,https://api.github.com/repos/simonw/sqlite-utils/issues/432,1155756742,IC_kwDOCGYnMM5E43LG,9599,simonw,2022-06-14T22:05:38Z,2022-06-14T22:05:49Z,OWNER,"I don't like the idea of `table_names()` returning names of tables from connected databases as well, because it feels like it could lead to surprising behaviour - especially if those connected databases turn to have table names that are duplicated in the main connected database. It would be neat if functions like `.rows_where()` worked though. One thought would be to support something like this: ```python rows = db[""otherdb.tablename""].rows_where() ``` But... `.` is a valid character in a SQLite table name. So `""otherdb.tablename""` might ambiguously refer to a table called `tablename` in a connected database with the alias `otherdb`, OR a table in the current database with the name `otherdb.tablename`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1236693079,"Support `rows_where()`, `delete_where()` etc for attached alias databases", https://github.com/simonw/sqlite-utils/issues/431#issuecomment-1164460052,https://api.github.com/repos/simonw/sqlite-utils/issues/431,1164460052,IC_kwDOCGYnMM5FaEAU,738408,rafguns,2022-06-23T14:12:51Z,2022-06-23T14:12:51Z,NONE,"Yeah, I think I prefer your suggestion: it seems cleaner than my initial `left_name=`/`right_name=` idea. Perhaps one downside is that it's less obvious what the role of each field is: in this example, is `people_id_1` a reference to parent or child?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1227571375,Allow making m2m relation of a table to itself, https://github.com/simonw/sqlite-utils/issues/431#issuecomment-1155753397,https://api.github.com/repos/simonw/sqlite-utils/issues/431,1155753397,IC_kwDOCGYnMM5E42W1,9599,simonw,2022-06-14T22:01:38Z,2022-06-14T22:01:38Z,OWNER,"Yeah, I think it would be neat if the library could support self-referential many-to-many in a nice way. I'm not sure about the `left_name/right_name` design though. Would it be possible to have this work as the user intends, by spotting that the other table name `""people""` matches the name of the current table? ```python db[""people""].insert({""name"": ""Mary""}, pk=""name"").m2m( ""people"", [{""name"": ""Michael""}, {""name"": ""Suzy""}], m2m_table=""parent_child"", pk=""name"" ) ``` The created table could look like this: ```sql CREATE TABLE [parent_child] ( [people_id_1] TEXT REFERENCES [people]([name]), [people_id_2] TEXT REFERENCES [people]([name]), PRIMARY KEY ([people_id_1], [people_id_2]) ) ``` I've not thought very hard about this, so the design I'm proposing here might not work. Are there other reasons people might wan the `left_name=` and `right_name=` parameters? If so then I'm much happier with those.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1227571375,Allow making m2m relation of a table to itself, https://github.com/simonw/sqlite-utils/issues/431#issuecomment-1126295407,https://api.github.com/repos/simonw/sqlite-utils/issues/431,1126295407,IC_kwDOCGYnMM5DIedv,738408,rafguns,2022-05-13T17:47:32Z,2022-05-13T17:47:32Z,NONE,"I'd be happy to write a PR for this, if you think it's worth having.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1227571375,Allow making m2m relation of a table to itself, https://github.com/simonw/sqlite-utils/issues/430#issuecomment-1155803262,https://api.github.com/repos/simonw/sqlite-utils/issues/430,1155803262,IC_kwDOCGYnMM5E5Ch-,9599,simonw,2022-06-14T23:26:11Z,2022-06-14T23:26:11Z,OWNER,"It looks like `PRAGMA temp_store` was the right option to use here: https://www.sqlite.org/pragma.html#pragma_temp_store `temp_store_directory` is listed as deprecated here: https://www.sqlite.org/pragma.html#pragma_temp_store_directory I'm going to turn this into a help-wanted documentation issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1224112817,Document how to use `PRAGMA temp_store` to avoid errors when running VACUUM against huge databases, https://github.com/simonw/sqlite-utils/issues/430#issuecomment-1116336340,https://api.github.com/repos/simonw/sqlite-utils/issues/430,1116336340,IC_kwDOCGYnMM5CifDU,9308268,rayvoelker,2022-05-03T17:03:31Z,2022-05-03T17:03:31Z,NONE,"So, the good news is that it appears that setting one of those PRAGMA statements fixed the issue of `table.extract()` method call on this large database completing (that I described above.) The bad news is that I'm not sure which one! I wonder if it's something system / environment specific about SQLite, or maybe something else going on.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1224112817,Document how to use `PRAGMA temp_store` to avoid errors when running VACUUM against huge databases, https://github.com/simonw/sqlite-utils/issues/43#issuecomment-514202279,https://api.github.com/repos/simonw/sqlite-utils/issues/43,514202279,MDEyOklzc3VlQ29tbWVudDUxNDIwMjI3OQ==,9599,simonw,2019-07-23T13:09:05Z,2019-07-23T13:09:05Z,OWNER,"Won't fix, I don't think this can be done in an elegant way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470691999,.add_column() doesn't match indentation of initial creation, https://github.com/simonw/sqlite-utils/issues/43#issuecomment-513481607,https://api.github.com/repos/simonw/sqlite-utils/issues/43,513481607,MDEyOklzc3VlQ29tbWVudDUxMzQ4MTYwNw==,9599,simonw,2019-07-20T16:36:34Z,2019-07-20T16:36:34Z,OWNER,"Hmm... is it even possible to fix this within sqlite-utils? Maybe this is a SQLite implementation detail. It's possible to update the schema itself - that's how we add missing foreign key constraints - but it seems a little bit of overkill to use the writable schema when adding a new column JUST to get prettier indentation: https://github.com/simonw/sqlite-utils/blob/c88f0a4d4617ff832aa728aac1da3500548c3137/sqlite_utils/db.py#L316-L326","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470691999,.add_column() doesn't match indentation of initial creation, https://github.com/simonw/sqlite-utils/issues/43#issuecomment-513481493,https://api.github.com/repos/simonw/sqlite-utils/issues/43,513481493,MDEyOklzc3VlQ29tbWVudDUxMzQ4MTQ5Mw==,9599,simonw,2019-07-20T16:34:54Z,2019-07-20T16:34:54Z,OWNER,"Here's the implementation in `create_table()`: https://github.com/simonw/sqlite-utils/blob/c88f0a4d4617ff832aa728aac1da3500548c3137/sqlite_utils/db.py#L235-L249 And here's where `add_column()` does its thing: https://github.com/simonw/sqlite-utils/blob/c88f0a4d4617ff832aa728aac1da3500548c3137/sqlite_utils/db.py#L547-L552","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470691999,.add_column() doesn't match indentation of initial creation, https://github.com/simonw/sqlite-utils/issues/428#issuecomment-1109190401,https://api.github.com/repos/simonw/sqlite-utils/issues/428,1109190401,IC_kwDOCGYnMM5CHOcB,9599,simonw,2022-04-26T01:05:29Z,2022-04-26T01:05:29Z,OWNER,Django makes extensive use of savepoints for nested transactions: https://docs.djangoproject.com/en/4.0/topics/db/transactions/#savepoints,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215216249,Research adding support for savepoints, https://github.com/simonw/sqlite-utils/issues/427#issuecomment-1172903355,https://api.github.com/repos/simonw/sqlite-utils/issues/427,1172903355,IC_kwDOCGYnMM5F6RW7,9599,simonw,2022-07-02T13:59:30Z,2022-07-02T13:59:30Z,OWNER,"This should work to avoid that: sqlite-utils convert \ ${dbfile} details dob \ 'r.parsedate(value)' \ --where ""dob != '*'""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212701569,"sqlite-utils convert date parsing recipe complains about trying to parse ""*""", https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1509951952,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1509951952,IC_kwDOCGYnMM5aAAnQ,89400147,Dhyanesh97,2023-04-15T20:14:58Z,2023-04-15T20:14:58Z,NONE,is this change released ? Because when we run docker containers issue still persists for production deployments.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher, https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1129332959,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1129332959,IC_kwDOCGYnMM5DUEDf,102771161,McEazy2700,2022-05-17T21:27:02Z,2022-05-17T21:27:02Z,NONE,"Hi, I'm trying to deploy my site using elasticbeanstalk and I keep getting this same error : deterministic=True requires SQLite 3.8.3 or higher I saw your previous solution that involves editing sqlite-utils/sqlite_utils/db.py file, but I'm curious as to how that will work in production.","{""total_count"": 5, ""+1"": 5, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher, https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1101594549,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1101594549,IC_kwDOCGYnMM5BqP-1,9599,simonw,2022-04-18T17:36:14Z,2022-04-18T17:36:14Z,OWNER,"Releated: - #408","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher, https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1098545390,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1098545390,IC_kwDOCGYnMM5Benju,9599,simonw,2022-04-13T22:34:52Z,2022-04-13T22:34:52Z,OWNER,"That broke Python 3.7 because it doesn't support `deterministic=True` even being passed: > function takes at most 3 arguments (4 given)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher, https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1098537000,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1098537000,IC_kwDOCGYnMM5Belgo,9599,simonw,2022-04-13T22:18:22Z,2022-04-13T22:18:22Z,OWNER,"I figured out a workaround in https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531 The current `register(fn)` method looks like this: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L403 This alternative implementation worked in the environment where that failed: ```python def register(fn): name = fn.__name__ arity = len(inspect.signature(fn).parameters) if not replace and (name, arity) in self._registered_functions: return fn kwargs = {} done = False if deterministic: # Try this, but fall back if sqlite3.NotSupportedError try: self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True)) done = True except sqlite3.NotSupportedError: pass if not done: self.conn.create_function(name, arity, fn, **kwargs) self._registered_functions.add((name, arity)) return fn ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher, https://github.com/simonw/sqlite-utils/issues/424#issuecomment-1098548090,https://api.github.com/repos/simonw/sqlite-utils/issues/424,1098548090,IC_kwDOCGYnMM5BeoN6,9599,simonw,2022-04-13T22:40:15Z,2022-04-13T22:40:15Z,OWNER,"New error: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""foo""].create({}) Traceback (most recent call last): File """", line 1, in File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1465, in create self.db.create_table( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 885, in create_table sql = self.create_table_sql( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 771, in create_table_sql assert columns, ""Tables must have at least one column"" AssertionError: Tables must have at least one column ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200866134,Better error message if you try to create a table with no columns, https://github.com/simonw/sqlite-utils/issues/423#issuecomment-1189010812,https://api.github.com/repos/simonw/sqlite-utils/issues/423,1189010812,IC_kwDOCGYnMM5G3t18,536941,fgregg,2022-07-19T12:47:39Z,2022-07-19T12:47:39Z,CONTRIBUTOR,just ran into this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1199158210,.extract() doesn't set foreign key when extracted columns contain NULL value, https://github.com/simonw/sqlite-utils/issues/422#issuecomment-1079406708,https://api.github.com/repos/simonw/sqlite-utils/issues/422,1079406708,IC_kwDOCGYnMM5AVnB0,9599,simonw,2022-03-25T20:23:21Z,2022-03-25T20:23:21Z,OWNER,"Fixing this would require a bump to 4.0 because it would break existing code. The alternative would be to introduce a new `ignore_nulls=True` parameter which users can change to `ignore_nulls=False`. Or come up with better wording for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1181236173,Reconsider not running convert functions against null values, https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098548931,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098548931,IC_kwDOCGYnMM5BeobD,9599,simonw,2022-04-13T22:41:59Z,2022-04-13T22:41:59Z,OWNER,"I'm going to close this ticket since it looks like this is a bug in the way the Dockerfile builds Python, but I'm going to ship a fix for that issue I found so the `LD_PRELOAD` workaround above should work OK with the next release of `sqlite-utils`. Thanks for the detailed bug report!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098535531,IC_kwDOCGYnMM5BelJr,9599,simonw,2022-04-13T22:15:48Z,2022-04-13T22:15:48Z,OWNER,"Trying this alternative implementation of the `register()` method: ```python def register(fn): name = fn.__name__ arity = len(inspect.signature(fn).parameters) if not replace and (name, arity) in self._registered_functions: return fn kwargs = {} done = False if deterministic: # Try this, but fall back if sqlite3.NotSupportedError try: self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True)) done = True except sqlite3.NotSupportedError: pass if not done: self.conn.create_function(name, arity, fn, **kwargs) self._registered_functions.add((name, arity)) return fn ``` With that fix, the following worked! ``` LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db --table table index_name seqno cid name desc coll key --------- -------------------------- ------- ----- ------- ------ ------ ----- countries idx_countries_country_name 0 1 country 0 BINARY 1 countries idx_countries_country_name 1 2 name 0 BINARY 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098532220,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098532220,IC_kwDOCGYnMM5BekV8,9599,simonw,2022-04-13T22:09:52Z,2022-04-13T22:09:52Z,OWNER,That error is weird - it's not supposed to happen according to this code here: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L400,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098531354,IC_kwDOCGYnMM5BekIa,9599,simonw,2022-04-13T22:08:20Z,2022-04-13T22:08:20Z,OWNER,"OK I figured out what's going on here. First I added an extra `print(sql)` statement to the `indexes` command to see what SQL it was running: ``` (app-root) sqlite-utils indexes global.db --table select sqlite_master.name as ""table"", indexes.name as index_name, xinfo.* from sqlite_master join pragma_index_list(sqlite_master.name) indexes join pragma_index_xinfo(index_name) xinfo where sqlite_master.type = 'table' and xinfo.key = 1 Error: near ""("": syntax error ``` This made me suspicious that the SQLite version being used here didn't support joining against the `pragma_index_list(...)` table-valued functions in that way. So I checked the version: ``` (app-root) sqlite3 SQLite version 3.36.0 2021-06-18 18:36:39 ``` That version should be fine - it's the one you compiled in the Dockerfile. Then I checked the version that `sqlite-utils` itself was using: ``` (app-root) sqlite-utils memory 'select sqlite_version()' [{""sqlite_version()"": ""3.7.17""}] ``` It's running SQLite 3.7.17! So the problem here is that the Python in that Docker image is running a very old version of SQLite. I tried using the trick in https://til.simonwillison.net/sqlite/ld-preload as a workaround, and it almost worked: ``` (app-root) python3 -c 'import sqlite3; print(sqlite3.connect("":memory"").execute(""select sqlite_version()"").fetchone())' ('3.7.17',) (app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so python3 -c 'import sqlite3; print(sqlite3.connect("":memory"").execute(""select sqlite_version()"").fetchone())' ('3.36.0',) ``` But when I try to run `sqlite-utils` like that I get an error: ``` (app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db ... File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py"", line 1624, in query db.register_fts4_bm25() File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 412, in register_fts4_bm25 self.register_function(rank_bm25, deterministic=True) File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 408, in register_function register(fn) File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 401, in register self.conn.create_function(name, arity, fn, **kwargs) sqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098295517,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098295517,IC_kwDOCGYnMM5Bdqjd,9599,simonw,2022-04-13T17:16:20Z,2022-04-13T17:16:20Z,OWNER,"Aha! I was able to replicate the bug using your `Dockerfile` - thanks very much for providing that. ``` (app-root) sqlite-utils indexes global.db --table Error: near ""("": syntax error ``` (That wa sbefore I even ran the `extract` command.) To build your `Dockerfile` I copied it into an empty folder and ran the following: ``` wget https://www.sqlite.org/2021/sqlite-autoconf-3360000.tar.gz docker build . -t centos-sqlite-utils docker run -it centos-sqlite-utils /bin/bash ``` This gave me a shell in which I could replicate the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098288158,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098288158,IC_kwDOCGYnMM5Bdowe,9599,simonw,2022-04-13T17:07:53Z,2022-04-13T17:07:53Z,OWNER,"I can't replicate the bug I'm afraid: ``` % wget ""https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true"" ... 2022-04-13 10:06:29 (8.97 MB/s) - ‘global_power_plant_database.csv?raw=true’ saved [8856038/8856038] % sqlite-utils insert global.db power_plants \ 'global_power_plant_database.csv?raw=true' --csv [------------------------------------] 0% [###################################-] 99% 00:00:00% % sqlite-utils indexes global.db --table table index_name seqno cid name desc coll key ------- ------------ ------- ----- ------ ------ ------ ----- % sqlite-utils extract global.db power_plants country country_long \ --table countries \ --fk-column country_id \ --rename country_long name % sqlite-utils indexes global.db --table table index_name seqno cid name desc coll key --------- -------------------------- ------- ----- ------- ------ ------ ----- countries idx_countries_country_name 0 1 country 0 BINARY 1 countries idx_countries_country_name 1 2 name 0 BINARY 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1081079506,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1081079506,IC_kwDOCGYnMM5Ab_bS,24938923,learning4life,2022-03-28T19:58:55Z,2022-03-28T20:05:57Z,NONE,"Sure, it is from the documentation example: [Extracting columns into a separate table](https://sqlite-utils.datasette.io/en/stable/cli.html#extracting-columns-into-a-separate-table) ``` wget ""https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true"" sqlite-utils insert global.db power_plants \ 'global_power_plant_database.csv?raw=true' --csv # Extract those columns: sqlite-utils extract global.db power_plants country country_long \ --table countries \ --fk-column country_id \ --rename country_long name ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1079407962,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1079407962,IC_kwDOCGYnMM5AVnVa,9599,simonw,2022-03-25T20:25:10Z,2022-03-25T20:25:18Z,OWNER,"Can you share either your whole `global.db` table or a shrunk down example that illustrates the bug? My hunch is that you may have a table or column with a name that triggers the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1229438242,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1229438242,IC_kwDOCGYnMM5JR70i,9599,simonw,2022-08-28T11:34:21Z,2022-08-28T11:34:37Z,OWNER,"I found a fix that makes that `global` workaround unnecessary: - #472","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1082476727,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1082476727,IC_kwDOCGYnMM5AhUi3,770231,strada,2022-03-29T23:52:38Z,2022-03-29T23:52:38Z,NONE,"@simonw Thanks for looking into it and documenting the solution! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1081047053,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1081047053,IC_kwDOCGYnMM5Ab3gN,9599,simonw,2022-03-28T19:22:37Z,2022-03-28T19:22:37Z,OWNER,Wrote about this in my weeknotes: https://simonwillison.net/2022/Mar/28/datasette-auth0/#new-features-as-documentation,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1080141111,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1080141111,IC_kwDOCGYnMM5AYaU3,9599,simonw,2022-03-28T03:25:57Z,2022-03-28T03:54:37Z,OWNER,"So now this should solve your problem: ``` echo '[{""name"": ""notaword""}, {""name"": ""word""}] ' | python3 -m sqlite_utils insert listings.db listings - --convert ' import enchant d = enchant.Dict(""en_US"") def convert(row): global d row[""is_dictionary_word""] = d.check(row[""name""]) ' ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079404281,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079404281,IC_kwDOCGYnMM5AVmb5,9599,simonw,2022-03-25T20:19:50Z,2022-03-25T20:19:50Z,OWNER,Now documented here: https://sqlite-utils.datasette.io/en/latest/cli.html#using-a-convert-function-to-execute-initialization,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079384771,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079384771,IC_kwDOCGYnMM5AVhrD,9599,simonw,2022-03-25T19:51:34Z,2022-03-25T19:53:01Z,OWNER,"This works: ``` % sqlite-utils insert dogs.db dogs dogs.json --convert ' import random print(""seeding"") random.seed(10) print(random.random()) def convert(row): global random print(row) row[""random_score""] = random.random() ' seeding 0.5714025946899135 {'id': 1, 'name': 'Cleo'} {'id': 2, 'name': 'Pancakes'} {'id': 3, 'name': 'New dog'} (sqlite-utils) sqlite-utils % sqlite-utils rows dogs.db dogs [{""id"": 1, ""name"": ""Cleo"", ""random_score"": 0.4288890546751146}, {""id"": 2, ""name"": ""Pancakes"", ""random_score"": 0.5780913011344704}, {""id"": 3, ""name"": ""New dog"", ""random_score"": 0.20609823213950174}] ``` Having to use `global random` inside the function is frustrating but apparently necessary. https://stackoverflow.com/a/56552138/6083","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079376283,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079376283,IC_kwDOCGYnMM5AVfmb,9599,simonw,2022-03-25T19:39:30Z,2022-03-25T19:43:35Z,OWNER,"Actually this doesn't work as I thought. This demo shows that the initialization code is run once per item, not a single time at the start of the run: ``` % sqlite-utils insert dogs.db dogs dogs.json --convert ' import random print(""seeding"") random.seed(10) print(random.random()) def convert(row): print(row) row[""random_score""] = random.random() ' seeding 0.5714025946899135 seeding 0.5714025946899135 seeding 0.5714025946899135 seeding 0.5714025946899135 ``` Also that `print(row)` line is not being printed anywhere that gets to the console for some reason. ... my mistake, that happened because I changed this line in order to try to get local imports to work: ```python try: exec(code, globals, locals) return globals[""convert""] except (AttributeError, SyntaxError, NameError, KeyError, TypeError): ``` It should be `locals[""convert""]`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079243535,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079243535,IC_kwDOCGYnMM5AU_MP,9599,simonw,2022-03-25T17:25:12Z,2022-03-25T17:25:12Z,OWNER,"That documentation is split across a few places. This is the only bit that talks about `def convert()` pattern right now: - https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns But that's for `sqlite-utils convert` - the documentation for `sqlite-utils insert --convert` at https://sqlite-utils.datasette.io/en/stable/cli.html#applying-conversions-while-inserting-data doesn't mention it. Since both `sqlite-utils convert` and `sqlite-utils insert --convert` apply the same rules to the code, they should link to a shared explanation in the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078343231,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078343231,IC_kwDOCGYnMM5ARjY_,9599,simonw,2022-03-24T21:16:10Z,2022-03-24T21:17:20Z,OWNER,"Aha! This may be possible already: https://github.com/simonw/sqlite-utils/blob/396f80fcc60da8dd844577114f7920830a2e5403/sqlite_utils/utils.py#L311-L316 And yes, this does indeed work - you can do something like this: ``` echo '{""name"": ""harry""}' | sqlite-utils insert db.db people - --convert ' import time # Simulate something expensive time.sleep(1) def convert(row): row[""upper""] = row[""name""].upper() ' ``` And after running that: ``` sqlite-utils dump db.db BEGIN TRANSACTION; CREATE TABLE [people] ( [name] TEXT, [upper] TEXT ); INSERT INTO ""people"" VALUES('harry','HARRY'); COMMIT; ``` So this is a documentation issue - there's a trick for it but I didn't know what the trick was!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078328774,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078328774,IC_kwDOCGYnMM5ARf3G,9599,simonw,2022-03-24T21:12:33Z,2022-03-24T21:12:33Z,OWNER,"Here's how the `_compile_code()` mechanism works at the moment: https://github.com/simonw/sqlite-utils/blob/396f80fcc60da8dd844577114f7920830a2e5403/sqlite_utils/utils.py#L308-L342 At the end it does this: ```python return locals[""fn""] ``` So it's already building and then returning a function. The question is if there's a sensible way to allow people to further customize that function by executing some code first, in a way that's easy to explain.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078322301,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078322301,IC_kwDOCGYnMM5AReR9,9599,simonw,2022-03-24T21:10:52Z,2022-03-24T21:10:52Z,OWNER,"I can think of three ways forward: - Figure out a pattern that gets that local file import workaround to work - Add another option such as `--convert-init` that lets you pass code that will be executed once at the start - Come up with a pattern where the `--convert` code can run some initialization code and then return a function which will be called against each value I quite like the idea of that third option - I'm going to prototype it and see if I can work something out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078315922,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078315922,IC_kwDOCGYnMM5ARcuS,9599,simonw,2022-03-24T21:09:27Z,2022-03-24T21:09:27Z,OWNER,"Yeah, this is WAY harder than it should be. There's a clumsy workaround you could use which looks something like this: create a file `my_enchant.py` containing: ```python import enchant d = enchant.Dict(""en_US"") def check(word): return d.check(word) ``` Then run `sqlite-utils` like this: ``` PYTHONPATH=. cat items.json | jq '.data' | sqlite-utils insert listings.db listings - --convert 'my_enchant.check(value)' --import my_enchant ``` Except I tried that and it doesn't work! I don't know the right pattern for getting `--import` to work with modules in the same directory. So yeah, this is definitely a big feature gap.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697037974,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697037974,MDEyOklzc3VlQ29tbWVudDY5NzAzNzk3NA==,9599,simonw,2020-09-22T23:39:31Z,2020-09-22T23:39:31Z,OWNER,Documentation for `sqlite-utils extract`: https://sqlite-utils.readthedocs.io/en/latest/cli.html#extracting-columns-into-a-separate-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697031174,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697031174,MDEyOklzc3VlQ29tbWVudDY5NzAzMTE3NA==,9599,simonw,2020-09-22T23:16:00Z,2020-09-22T23:16:00Z,OWNER,"Trying this demo again: ``` wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv sqlite-utils extract global.db power_plants country country_long --table countries --rename country_long name ``` It worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697025403,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697025403,MDEyOklzc3VlQ29tbWVudDY5NzAyNTQwMw==,9599,simonw,2020-09-22T22:57:53Z,2020-09-22T22:57:53Z,OWNER,The documentation for the `.extract()` method is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#extracting-columns-into-a-separate-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697019944,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697019944,MDEyOklzc3VlQ29tbWVudDY5NzAxOTk0NA==,9599,simonw,2020-09-22T22:40:00Z,2020-09-22T22:40:00Z,OWNER,"I tried out the prototype of the CLI on the Global Power Plants data: ``` wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv sqlite-utils extract global.db power_plants country country_long ``` This threw an error because `rowid` columns are not yet supported. I fixed that like so: ``` sqlite-utils transform global.db power_plants --rename rowid id sqlite-utils extract global.db power_plants country country_long ``` That worked! But it didn't play great with Datasette, because the resulting extracted table had columns `country` and `country_long` and neither of those are called `name` or `value` or `title`. Based on this I need to add `rowid` table support AND I need to implement the proposed `rename=` argument for renaming columns on their way into the new table. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697013681,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697013681,MDEyOklzc3VlQ29tbWVudDY5NzAxMzY4MQ==,9599,simonw,2020-09-22T22:22:49Z,2020-09-22T22:22:49Z,OWNER,"The command-line version of this needs to accept a table and one or more columns, then a `--table` and `--fk-column` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697012111,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697012111,MDEyOklzc3VlQ29tbWVudDY5NzAxMjExMQ==,9599,simonw,2020-09-22T22:18:13Z,2020-09-22T22:18:13Z,OWNER,"Here's how I'm generating the examples for the documentation: ``` In [2]: import sqlite_utils In [3]: db = sqlite_utils.Database(memory=True) In [4]: db[""Trees""].insert({""id"": 1, ""TreeAddress"": ""52 Vine St"", ""CommonName"": ...: ""Palm"", ""LatinName"": ""foo""}, pk=""id"") Out[4]:
In [5]: db[""Trees""].extract([""CommonName"", ""LatinName""], table=""Species"", fk_col ...: umn=""species_id"") In [6]: print(db[""Trees""].schema) CREATE TABLE ""Trees"" ( [id] INTEGER PRIMARY KEY, [TreeAddress] TEXT, [species_id] INTEGER, FOREIGN KEY(species_id) REFERENCES Species(id) ) In [7]: print(db[""Species""].schema) CREATE TABLE [Species] ( [id] INTEGER PRIMARY KEY, [CommonName] TEXT, [LatinName] TEXT ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987925,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696987925,MDEyOklzc3VlQ29tbWVudDY5Njk4NzkyNQ==,9599,simonw,2020-09-22T21:19:04Z,2020-09-22T21:19:04Z,OWNER,Need to make sure this works correctly for `rowid` tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987257,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696987257,MDEyOklzc3VlQ29tbWVudDY5Njk4NzI1Nw==,9599,simonw,2020-09-22T21:17:34Z,2020-09-22T21:17:34Z,OWNER,"What to do if the table already exists? The `.lookup()` function already knows how to modify an existing table to create the correct constraints etc, so I'll rely on that mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980709,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696980709,MDEyOklzc3VlQ29tbWVudDY5Njk4MDcwOQ==,9599,simonw,2020-09-22T21:05:07Z,2020-09-22T21:05:07Z,OWNER,"So `.extract()` probably takes a `batch_size=` argument too, which defaults to maybe 1000.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980503,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696980503,MDEyOklzc3VlQ29tbWVudDY5Njk4MDUwMw==,9599,simonw,2020-09-22T21:04:45Z,2020-09-22T21:04:45Z,OWNER,"`table.extract()` can take an optional `progress=` argument which is a callback which will be used to report progress - called after each batch with `(num_done, total)`. It will get called with `(0, total)` once at the start to allow progress bars to be initialized. The command-line progress bar will use this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979626,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696979626,MDEyOklzc3VlQ29tbWVudDY5Njk3OTYyNg==,9599,simonw,2020-09-22T21:03:11Z,2020-09-22T21:03:11Z,OWNER,"And if you want to rename some of the columns in the new table: ```python db[""trees""].extract([""common_name"", ""latin_name""], table=""species"", rename={""common_name"": ""name""}) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979168,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696979168,MDEyOklzc3VlQ29tbWVudDY5Njk3OTE2OA==,9599,simonw,2020-09-22T21:02:24Z,2020-09-22T21:02:24Z,OWNER,"In Python it looks like this: ```python # Simple case - species column species_id pointing to species table db[""trees""].extract(""species"") # Setting a custom table db[""trees""].extract(""species"", table=""Species"") # Custom foreign key column on trees db[""trees""].extract(""species"", fk_column=""species"") # Extracting multiple columns db[""trees""].extract([""common_name"", ""latin_name""]) # (this creates a lookup table called common_name_latin_name ref'd by common_name_latin_name_id) # Or with explicit table (fk_column here defaults to species_id because of the table name) db[""trees""].extract([""common_name"", ""latin_name""], table=""species"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696976678,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696976678,MDEyOklzc3VlQ29tbWVudDY5Njk3NjY3OA==,9599,simonw,2020-09-22T20:57:57Z,2020-09-22T20:57:57Z,OWNER,"I think I understand the shape of this feature now. It lets you specify one or more columns on the source table which will be extracted into another table. It uses the `.lookup()` mechanism to populate that other table, which means each unique column value / pair / triple will be assigned an integer ID. That integer ID gets written back into the first of the columns that are being transformed. A `.transform()` call then converts that column to an integer (and drops the additional columns). Finally we set up the new foreign key relationship.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893774,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696893774,MDEyOklzc3VlQ29tbWVudDY5Njg5Mzc3NA==,9599,simonw,2020-09-22T18:15:33Z,2020-09-22T18:15:33Z,OWNER,I think the new foreign key column is called `company_name_id` by default in this example but can be customized by passing `--fk-column=xxx`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893244,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696893244,MDEyOklzc3VlQ29tbWVudDY5Njg5MzI0NA==,9599,simonw,2020-09-22T18:14:33Z,2020-09-22T18:14:45Z,OWNER,"Thinking more about this one: ``` $ sqlite-utils extract my.db \ dea_sales company_name company_address \ --table companies ``` The goal here is to pull the company name and address pair out into a separate table. Some questions: - should this first verify that every company_name has just one company_address? I like the idea of a unique constraint on the created table for this. - what should the foreign key column that gets added to the `companies` table be called?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696567460,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696567460,MDEyOklzc3VlQ29tbWVudDY5NjU2NzQ2MA==,9599,simonw,2020-09-22T07:56:42Z,2020-09-22T07:56:42Z,OWNER,`.transform()` has landed now which should make this a lot easier to solve.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-695698227,https://api.github.com/repos/simonw/sqlite-utils/issues/42,695698227,MDEyOklzc3VlQ29tbWVudDY5NTY5ODIyNw==,9599,simonw,2020-09-20T04:27:26Z,2020-09-20T04:28:26Z,OWNER,This is going to need #114 (the `transform_table()` method) in order to convert string columns into integer foreign key columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513262013,https://api.github.com/repos/simonw/sqlite-utils/issues/42,513262013,MDEyOklzc3VlQ29tbWVudDUxMzI2MjAxMw==,9599,simonw,2019-07-19T14:58:23Z,2020-09-22T18:12:11Z,OWNER,"CLI design idea: $ sqlite-utils extract my.db \ dea_sales company_name Here we just specify the original table and column - the new extracted table will automatically be called ""company_name"" and will have ""id"" and ""value"" columns, by default. To set a custom extract table: $ sqlite-utils extract my.db \ dea_sales company_name \ --table companies And for extracting multiple columns and renaming them on the created table, maybe something like this: $ sqlite-utils extract my.db \ dea_sales company_name company_address \ --table companies \ --column company_name name \ --column company_address address ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513246831,https://api.github.com/repos/simonw/sqlite-utils/issues/42,513246831,MDEyOklzc3VlQ29tbWVudDUxMzI0NjgzMQ==,9599,simonw,2019-07-19T14:20:15Z,2019-07-19T14:20:49Z,OWNER,"Since these operations could take a long time against large tables, it would be neat if there was a progress bar option for the CLI command. The operations are full table scans so calculating progress shouldn't be too difficult.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513246124,https://api.github.com/repos/simonw/sqlite-utils/issues/42,513246124,MDEyOklzc3VlQ29tbWVudDUxMzI0NjEyNA==,9599,simonw,2019-07-19T14:18:35Z,2019-07-19T14:19:40Z,OWNER,"How about the Python version? That should be easier to design. ```python db[""dea_sales""].extract( columns=[""company_name"", ""company_address""], to_table=""companies"" ) ``` If we want to transform the extracted data (e.g. rename those columns) maybe support a `transform=` argument? ```python db[""dea_sales""].extract( columns=[""company_name"", ""company_address""], to_table=""companies"", transform = lambda extracted: { ""name"": extracted[""company_name""], ""address"": extracted[""company_address""], } ) ``` This would create a new ""companies"" table with three columns: id, name and address. Would also be nice if there was a syntax for saying ""... and use the value from this column as the primary key column in the newly created table"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513244121,https://api.github.com/repos/simonw/sqlite-utils/issues/42,513244121,MDEyOklzc3VlQ29tbWVudDUxMzI0NDEyMQ==,9599,simonw,2019-07-19T14:13:33Z,2019-07-19T14:13:33Z,OWNER,"So what could the interface to this look like? Especially for the CLI? One option: sqlite-utils extract dea_sales company_name companies name Tricky thing here is that it's quite a large number of positional arguments: sqlite-utils extract dea_sales company_name companies name Table column New table New column (maybe optional?) It would be great if this could supported multiple columns - for if a spreadsheet has e.g. a “Company Name”, “Company Address” pair of fields that always match each other and areduplicated many times. This could be handled by creating the new table with two columns that are indexed as a unique compound key. Then you can easily get-or-create on the pairs (or triples or whatever) from the original table. Challenge here is what does the CLI syntax look like. Something like this? $ sqlite-utils extract dea_sales -c company_name -c company_address \ --to companies --to-col name --to-col address Perhaps the columns in the new table are FORCED to be the same as the old ones, hence avoiding some options? Bit restrictive… maybe they default to the same but you can customize? $ sqlite-utils extract dea_sales -c company_name -c company_address -t companies","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1079441621,https://api.github.com/repos/simonw/sqlite-utils/issues/417,1079441621,IC_kwDOCGYnMM5AVvjV,9599,simonw,2022-03-25T21:18:37Z,2022-03-25T21:18:37Z,OWNER,Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-newline-delimited-json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175744654,insert fails on JSONL with whitespace, https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074256603,https://api.github.com/repos/simonw/sqlite-utils/issues/417,1074256603,IC_kwDOCGYnMM5AB9rb,9954,blaine,2022-03-21T18:19:41Z,2022-03-21T18:19:41Z,NONE,"That makes sense; just a little hint that points folks towards doing the right thing might be helpful! fwiw, the reason I was using jq in the first place was just a quick way to extract one attribute from an actual JSON array. When I initially imported it, I got a table with a bunch of embedded JSON values, rather than a native table, because each array entry had two attributes, one with the data I _actually_ wanted. Not sure how common a use-case this is, though (and easily fixed, aside from the jq weirdness!)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175744654,insert fails on JSONL with whitespace, https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074243540,https://api.github.com/repos/simonw/sqlite-utils/issues/417,1074243540,IC_kwDOCGYnMM5AB6fU,9599,simonw,2022-03-21T18:08:03Z,2022-03-21T18:08:03Z,OWNER,"I've not really thought about standards as much here as I should. It looks like there are two competing specs for newline-delimited JSON! http://ndjson.org/ is the one I've been using in `sqlite-utils` - and https://github.com/ndjson/ndjson-spec#31-serialization says: > The JSON texts MUST NOT contain newlines or carriage returns. https://jsonlines.org/ is the other one. It is slightly less clear, but it does say this: > 2. Each Line is a Valid JSON Value > > The most common values will be objects or arrays, but any JSON value is permitted. My interpretation of both of these is that newlines in the middle of a JSON object shouldn't be allowed. So what's `jq` doing here? It looks to me like that `jq` format is its own thing - it's not actually compatible with either of those two loose specs described above. The `jq` docs seem to call this ""whitespace-separated JSON"": https://stedolan.github.io/jq/manual/v1.6/#Invokingjq The thing I like about newline-delimited JSON is that it's really trivial to parse - loop through each line, run it through `json.loads()` and that's it. No need to try and unwrap JSON objects that might span multiple lines. Unless someone has written a robust Python implementation of a `jq`-compatible whitespace-separated JSON parser, I'm inclined to leave this as is. I'd be fine adding some documentation that helps point people towards `jq -c` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175744654,insert fails on JSONL with whitespace, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1116684581,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1116684581,IC_kwDOCGYnMM5Cj0El,638427,mattkiefer,2022-05-03T21:36:49Z,2022-05-03T21:36:49Z,NONE,"Thanks for addressing this @simonw! However, I just reinstalled sqlite-utils 3.26.1 and get an `ParserError: Unknown string format: None`: ``` sqlite-utils --version sqlite-utils, version 3.26.1 ``` ``` sqlite-utils convert idfpr.db license ""Original Issue Date"" ""r.parsedate(value)"" Traceback (most recent call last): File ""/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2514, in convert_value return fn(v) File """", line 2, in fn File ""/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/recipes.py"", line 19, in parsedate parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst) File ""/usr/lib/python3/dist-packages/dateutil/parser/_parser.py"", line 1374, in parse return DEFAULTPARSER.parse(timestr, **kwargs) File ""/usr/lib/python3/dist-packages/dateutil/parser/_parser.py"", line 649, in parse raise ParserError(""Unknown string format: %s"", timestr) dateutil.parser._parser.ParserError: Unknown string format: None Traceback (most recent call last): File ""/home/matt/.local/bin/sqlite-utils"", line 8, in sys.exit(cli()) File ""/usr/lib/python3/dist-packages/click/core.py"", line 829, in __call__ return self.main(*args, **kwargs) File ""/usr/lib/python3/dist-packages/click/core.py"", line 782, in main rv = self.invoke(ctx) File ""/usr/lib/python3/dist-packages/click/core.py"", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/usr/lib/python3/dist-packages/click/core.py"", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/usr/lib/python3/dist-packages/click/core.py"", line 610, in invoke return callback(*args, **kwargs) File ""/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/cli.py"", line 2707, in convert db[table].convert( File ""/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2530, in convert self.db.execute(sql, where_args or []) File ""/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/db.py"", line 463, in execute return self.conn.execute(sql, parameters) sqlite3.OperationalError: user-defined function raised exception ``` I definitely have some invalid data in the db. Happy to send a copy if it's helpful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456222,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073456222,IC_kwDOCGYnMM4_-6Re,9599,simonw,2022-03-21T03:45:52Z,2022-03-21T03:45:52Z,OWNER,Needs tests and documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456155,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073456155,IC_kwDOCGYnMM4_-6Qb,9599,simonw,2022-03-21T03:45:37Z,2022-03-21T03:45:37Z,OWNER,"Prototype: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 8255b56..0a3693e 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -2583,7 +2583,11 @@ def _generate_convert_help(): """""" ).strip() recipe_names = [ - n for n in dir(recipes) if not n.startswith(""_"") and n not in (""json"", ""parser"") + n + for n in dir(recipes) + if not n.startswith(""_"") + and n not in (""json"", ""parser"") + and callable(getattr(recipes, n)) ] for name in recipe_names: fn = getattr(recipes, name) diff --git a/sqlite_utils/recipes.py b/sqlite_utils/recipes.py index 6918661..569c30d 100644 --- a/sqlite_utils/recipes.py +++ b/sqlite_utils/recipes.py @@ -1,17 +1,38 @@ from dateutil import parser import json +IGNORE = object() +SET_NULL = object() -def parsedate(value, dayfirst=False, yearfirst=False): + +def parsedate(value, dayfirst=False, yearfirst=False, errors=None): ""Parse a date and convert it to ISO date format: yyyy-mm-dd"" - return ( - parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() - ) + try: + return ( + parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst) + .date() + .isoformat() + ) + except parser.ParserError: + if errors is IGNORE: + return value + elif errors is SET_NULL: + return None + else: + raise -def parsedatetime(value, dayfirst=False, yearfirst=False): +def parsedatetime(value, dayfirst=False, yearfirst=False, errors=None): ""Parse a datetime and convert it to ISO datetime format: yyyy-mm-ddTHH:MM:SS"" - return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + try: + return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + except parser.ParserError: + if errors is IGNORE: + return value + elif errors is SET_NULL: + return None + else: + raise def jsonsplit(value, delimiter="","", type=str): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073455905,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073455905,IC_kwDOCGYnMM4_-6Mh,9599,simonw,2022-03-21T03:44:47Z,2022-03-21T03:45:00Z,OWNER,"This is quite nice: ``` % sqlite-utils convert test-dates.db dates date ""r.parsedate(value, errors=r.IGNORE)"" [####################################] 100% % sqlite-utils rows test-dates.db dates [{""id"": 1, ""date"": ""2016-03-15""}, {""id"": 2, ""date"": ""2016-03-16""}, {""id"": 3, ""date"": ""2016-03-17""}, {""id"": 4, ""date"": ""2016-03-18""}, {""id"": 5, ""date"": ""2016-03-19""}, {""id"": 6, ""date"": ""2016-03-20""}, {""id"": 7, ""date"": ""2016-03-21""}, {""id"": 8, ""date"": ""2016-03-22""}, {""id"": 9, ""date"": ""2016-03-23""}, {""id"": 10, ""date"": ""//""}, {""id"": 11, ""date"": ""2016-03-25""}, {""id"": 12, ""date"": ""2016-03-26""}, {""id"": 13, ""date"": ""2016-03-27""}, {""id"": 14, ""date"": ""2016-03-28""}, {""id"": 15, ""date"": ""2016-03-29""}, {""id"": 16, ""date"": ""2016-03-30""}, {""id"": 17, ""date"": ""2016-03-31""}, {""id"": 18, ""date"": ""2016-04-01""}] % sqlite-utils convert test-dates.db dates date ""r.parsedate(value, errors=r.SET_NULL)"" [####################################] 100% % sqlite-utils rows test-dates.db dates [{""id"": 1, ""date"": ""2016-03-15""}, {""id"": 2, ""date"": ""2016-03-16""}, {""id"": 3, ""date"": ""2016-03-17""}, {""id"": 4, ""date"": ""2016-03-18""}, {""id"": 5, ""date"": ""2016-03-19""}, {""id"": 6, ""date"": ""2016-03-20""}, {""id"": 7, ""date"": ""2016-03-21""}, {""id"": 8, ""date"": ""2016-03-22""}, {""id"": 9, ""date"": ""2016-03-23""}, {""id"": 10, ""date"": null}, {""id"": 11, ""date"": ""2016-03-25""}, {""id"": 12, ""date"": ""2016-03-26""}, {""id"": 13, ""date"": ""2016-03-27""}, {""id"": 14, ""date"": ""2016-03-28""}, {""id"": 15, ""date"": ""2016-03-29""}, {""id"": 16, ""date"": ""2016-03-30""}, {""id"": 17, ""date"": ""2016-03-31""}, {""id"": 18, ""date"": ""2016-04-01""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453370,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073453370,IC_kwDOCGYnMM4_-5k6,9599,simonw,2022-03-21T03:41:06Z,2022-03-21T03:41:06Z,OWNER,I'm going to try the `errors=r.IGNORE` option and see what that looks like once implemented.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453230,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073453230,IC_kwDOCGYnMM4_-5iu,9599,simonw,2022-03-21T03:40:37Z,2022-03-21T03:40:37Z,OWNER,"I think the options here should be: - On error, raise an exception and revert the transaction (the current default) - On error, leave the value as-is - On error, set the value to `None` These need to be indicated by parameters to the `r.parsedate()` function. Some design options: - `ignore=True` to ignore errors - but how does it know if it should leave the value or set it to `None`? This is similar to other `ignore=True` parameters elsewhere in the Python API. - `errors=""ignore""`, `errors=""set-null""` - I don't like magic string values very much, but this is similar to Python's `str.encode(errors=)` mechanism - `errors=r.IGNORE` - using constants, which at least avoids magic strings. The other one could be `errors=r.SET_NULL` - `error=lambda v: None` or `error=lambda v: v` - this is a bit confusing though, introducing another callback that gets to have a go at converting the error if the first callback failed? And what happens if that lambda itself raises an error?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073451659,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073451659,IC_kwDOCGYnMM4_-5KL,9599,simonw,2022-03-21T03:35:01Z,2022-03-21T03:35:01Z,OWNER,"I confirmed that if it fails for any value ALL values are left alone, since it runs in a transaction. Here's the code that does that: https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L2523-L2526","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073450588,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073450588,IC_kwDOCGYnMM4_-45c,9599,simonw,2022-03-21T03:32:58Z,2022-03-21T03:32:58Z,OWNER,"Then I ran this to convert `2016-03-27` etc to `2016/03/27` so I could see which ones were later converted: sqlite-utils convert test-dates.db dates date 'value.replace(""-"", ""/"")' ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073448904,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073448904,IC_kwDOCGYnMM4_-4fI,9599,simonw,2022-03-21T03:28:12Z,2022-03-21T03:30:37Z,OWNER,"Generating a test database using a pattern from https://www.geekytidbits.com/date-range-table-sqlite/ ``` sqlite-utils create-database test-dates.db sqlite-utils create-table test-dates.db dates id integer date text --pk id sqlite-utils test-dates.db ""WITH RECURSIVE cnt(x) AS ( SELECT 0 UNION ALL SELECT x+1 FROM cnt LIMIT (SELECT ((julianday('2016-04-01') - julianday('2016-03-15'))) + 1) ) insert into dates (date) select date(julianday('2016-03-15'), '+' || x || ' days') as date FROM cnt;"" ``` After running that: ``` % sqlite-utils rows test-dates.db dates [{""id"": 1, ""date"": ""2016-03-15""}, {""id"": 2, ""date"": ""2016-03-16""}, {""id"": 3, ""date"": ""2016-03-17""}, {""id"": 4, ""date"": ""2016-03-18""}, {""id"": 5, ""date"": ""2016-03-19""}, {""id"": 6, ""date"": ""2016-03-20""}, {""id"": 7, ""date"": ""2016-03-21""}, {""id"": 8, ""date"": ""2016-03-22""}, {""id"": 9, ""date"": ""2016-03-23""}, {""id"": 10, ""date"": ""2016-03-24""}, {""id"": 11, ""date"": ""2016-03-25""}, {""id"": 12, ""date"": ""2016-03-26""}, {""id"": 13, ""date"": ""2016-03-27""}, {""id"": 14, ""date"": ""2016-03-28""}, {""id"": 15, ""date"": ""2016-03-29""}, {""id"": 16, ""date"": ""2016-03-30""}, {""id"": 17, ""date"": ""2016-03-31""}, {""id"": 18, ""date"": ""2016-04-01""}] ``` Then to make one of them invalid: sqlite-utils test-dates.db ""update dates set date = '//' where id = 10""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072834273,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1072834273,IC_kwDOCGYnMM4_8ibh,9599,simonw,2022-03-18T21:36:05Z,2022-03-18T21:36:05Z,OWNER,"Python's `str.encode()` method has a `errors=` parameter that does something along these lines: https://docs.python.org/3/library/stdtypes.html#str.encode > *errors* may be given to set a different error handling scheme. The default for *errors* is `'strict'`, meaning that encoding errors raise a [`UnicodeError`](https://docs.python.org/3/library/exceptions.html#UnicodeError ""UnicodeError""). Other possible values are `'ignore'`, `'replace'`, `'xmlcharrefreplace'`, `'backslashreplace'` and any other name registered via [`codecs.register_error()`](https://docs.python.org/3/library/codecs.html#codecs.register_error ""codecs.register_error""), Imitating this might be the way to go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072833174,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1072833174,IC_kwDOCGYnMM4_8iKW,9599,simonw,2022-03-18T21:34:06Z,2022-03-18T21:34:06Z,OWNER,"Good call-out: right now the `parsedate()` and `parsedatetime()` functions both terminate with an exception if they hit something invalid: https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes It would be better if this was configurable by the user (and properly documented) - options could include ""set null if date is invalid"" and ""leave the value as it is if invalid"" in addition to throwing an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073468996,https://api.github.com/repos/simonw/sqlite-utils/issues/415,1073468996,IC_kwDOCGYnMM4_-9ZE,9599,simonw,2022-03-21T04:14:42Z,2022-03-21T04:14:42Z,OWNER,"I can fix this like so: ``` % sqlite-utils convert demo.db demo foo '{""foo"": ""bar""}' --multi --dry-run abc --- becomes: {""foo"": ""bar""} Would affect 1 row ``` Diff is this: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 0cf0468..b2a0440 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -2676,7 +2676,10 @@ def convert( raise click.ClickException(str(e)) if dry_run: # Pull first 20 values for first column and preview them - db.conn.create_function(""preview_transform"", 1, lambda v: fn(v) if v else v) + preview = lambda v: fn(v) if v else v + if multi: + preview = lambda v: json.dumps(fn(v), default=repr) if v else v + db.conn.create_function(""preview_transform"", 1, preview) sql = """""" select [{column}] as value, ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1171599874,Convert with `--multi` and `--dry-run` flag does not work, https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073463375,https://api.github.com/repos/simonw/sqlite-utils/issues/415,1073463375,IC_kwDOCGYnMM4_-8BP,9599,simonw,2022-03-21T04:02:36Z,2022-03-21T04:02:36Z,OWNER,Thanks for the really clear steps to reproduce!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1171599874,Convert with `--multi` and `--dry-run` flag does not work, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065384183,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065384183,IC_kwDOCGYnMM4_gHj3,9599,simonw,2022-03-11T18:40:39Z,2022-03-11T18:40:39Z,OWNER,"That fixed it: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065382145,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065382145,IC_kwDOCGYnMM4_gHEB,9599,simonw,2022-03-11T18:39:05Z,2022-03-11T18:39:05Z,OWNER,"https://sqlite-utils.datasette.io/en/3.25.1/changelog.html is currently wrong: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065381047,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065381047,IC_kwDOCGYnMM4_gGy3,9599,simonw,2022-03-11T18:38:27Z,2022-03-11T18:38:27Z,OWNER,"OK that fixed it here: https://sqlite-utils.datasette.io/en/stable/changelog.html I'm going to trigger a rebuild of `3.25.1` too: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065380286,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065380286,IC_kwDOCGYnMM4_gGm-,9599,simonw,2022-03-11T18:37:23Z,2022-03-11T18:37:23Z,OWNER,"On ReadTheDocs that triggered a new `stable` build but it didn't seem to trigger a new build of `3.25.1`: https://readthedocs.org/projects/sqlite-utils/builds/ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065379528,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065379528,IC_kwDOCGYnMM4_gGbI,9599,simonw,2022-03-11T18:36:17Z,2022-03-11T18:36:17Z,OWNER,"I created a new tag and release: https://github.com/simonw/sqlite-utils/releases/tag/3.25.1 And I cancelled the publish workflow: https://github.com/simonw/sqlite-utils/actions/runs/1970415399","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065378902,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065378902,IC_kwDOCGYnMM4_gGRW,9599,simonw,2022-03-11T18:35:26Z,2022-03-11T18:35:26Z,OWNER,I deleted both the release and the tag from GitHub using the web interface.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065377926,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065377926,IC_kwDOCGYnMM4_gGCG,9599,simonw,2022-03-11T18:34:05Z,2022-03-11T18:34:05Z,OWNER,"Two options: - Delete and recreate the release on GitHub, triggering it to be fixed on Read The Docs (as the `stable` version) - but cancel the push to PyPI, since that platform doesn't allow package versions to be over-written and in this case since the changelog file isn't included in the PyPI package there should be no change at all - Push a 3.25.2 release. I'm going to try and do that first option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065357081,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065357081,IC_kwDOCGYnMM4_gA8Z,9599,simonw,2022-03-11T18:07:10Z,2022-03-11T18:07:10Z,OWNER,I'm really happy with this improvement.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065345515,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065345515,IC_kwDOCGYnMM4_f-Hr,9599,simonw,2022-03-11T17:52:22Z,2022-03-11T17:52:22Z,OWNER,"Well this is a huge improvement! https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.insert I'm not crazy about the `extracts=` thing though - I wonder if there's a neat way to customize that to be less verbose?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065249883,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065249883,IC_kwDOCGYnMM4_fmxb,9599,simonw,2022-03-11T16:03:35Z,2022-03-11T16:03:35Z,OWNER,"Applying this change fixes that: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 3bc528f..2a79711 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -2293,18 +2293,18 @@ class Table(Queryable): """""" Apply conversion function ``fn`` to every value in the specified columns. - - ``columns`` - a single column or list of string column names to convert. - - ``fn`` - a callable that takes a single argument, ``value``, and returns it converted. - - ``output`` - optional string column name to write the results to (defaults to the input column). - - ``output_type`` - if the output column needs to be created, this is the type that will be used + :param columns: a single column or list of string column names to convert. + :param fn: a callable that takes a single argument, ``value``, and returns it converted. + :param output: optional string column name to write the results to (defaults to the input column). + :param output_type: if the output column needs to be created, this is the type that will be used for the new column. - - ``drop`` - boolean, should the original column be dropped once the conversion is complete? - - ``multi`` - boolean, if ``True`` the return value of ``fn(value)`` will be expected to be a + :param drop: boolean, should the original column be dropped once the conversion is complete? + :param multi: boolean, if ``True`` the return value of ``fn(value)`` will be expected to be a dictionary, and new columns will be created for each key of that dictionary. - - ``where`` - a SQL fragment to use as a ``WHERE`` clause to limit the rows to which the conversion + :param where: a SQL fragment to use as a ``WHERE`` clause to limit the rows to which the conversion is applied, for example ``age > ?`` or ``age > :age``. - - ``where_args`` - a list of arguments (if using ``?``) or a dictionary (if using ``:age``). - - ``show_progress`` - boolean, should a progress bar be displayed? + :param where_args: a list of arguments (if using ``?``) or a dictionary (if using ``:age``). + :param show_progress: boolean, should a progress bar be displayed? See :ref:`python_api_convert`. """""" ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065247619,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065247619,IC_kwDOCGYnMM4_fmOD,9599,simonw,2022-03-11T16:01:20Z,2022-03-11T16:01:20Z,OWNER,"Definitely an improvement! It does highlight that I'm not currently using the `:param XXX: description` syntax though, which should move my descriptions of each parameter into that generated list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065245831,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065245831,IC_kwDOCGYnMM4_flyH,9599,simonw,2022-03-11T15:59:14Z,2022-03-11T15:59:14Z,OWNER,"Hint from https://twitter.com/AdamChainz/status/1502311047612575745 > Try: > > `autodoc_typehints = 'description'` > > For a list-of-arguments format > > https://sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_typehints","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1155364367,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1155364367,IC_kwDOCGYnMM5E3XYP,9599,simonw,2022-06-14T15:36:28Z,2022-06-14T15:36:28Z,OWNER,"Here's as far as I got with my initial prototype, in `sqlite_utils/pandas.py`: ```python from .db import Database as _Database, Table as _Table, View as _View import pandas as pd from typing import ( Iterable, Union, Optional, ) class Database(_Database): def query( self, sql: str, params: Optional[Union[Iterable, dict]] = None ) -> pd.DataFrame: return pd.DataFrame(super().query(sql, params)) def table(self, table_name: str, **kwargs) -> Union[""Table"", ""View""]: ""Return a table object, optionally configured with default options."" klass = View if table_name in self.view_names() else Table return klass(self, table_name, **kwargs) class PandasQueryable: def rows_where( self, where: str = None, where_args: Optional[Union[Iterable, dict]] = None, order_by: str = None, select: str = ""*"", limit: int = None, offset: int = None, ) -> pd.DataFrame: return pd.DataFrame( super().rows_where( where, where_args, order_by=order_by, select=select, limit=limit, offset=offset, ) ) class Table(PandasQueryable, _Table): pass class View(PandasQueryable, _View): pass ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059652834,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059652834,IC_kwDOCGYnMM4_KQTi,596279,zaneselvans,2022-03-05T02:14:40Z,2022-03-05T02:14:40Z,NONE,"We do a lot of `df.to_sql()` to write into sqlite, mostly in [this moddule](https://github.com/catalyst-cooperative/pudl/blob/main/src/pudl/load.py#L25)","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059652538,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059652538,IC_kwDOCGYnMM4_KQO6,9599,simonw,2022-03-05T02:13:17Z,2022-03-05T02:13:17Z,OWNER,"> It looks like the existing `pd.read_sql_query()` method has an optional dependency on SQLAlchemy: > > ``` > ... > import pandas as pd > pd.read_sql_query(db.conn, ""select * from articles"") > # ImportError: Using URI string without sqlalchemy installed. > ``` Hah, no I was wrong about this: SQLAlchemy is not needed for SQLite to work, I just had the arguments the wrong way round: ```python pd.read_sql_query(""select * from articles"", db.conn) # Shows a DateFrame ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059651306,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059651306,IC_kwDOCGYnMM4_KP7q,9599,simonw,2022-03-05T02:10:49Z,2022-03-05T02:10:49Z,OWNER,"I could teach `.insert_all()` and `.upsert_all()` to optionally accept a DataFrame. A challenge there is `mypy` - if Pandas is an optional dependency, is it possibly to declare types that accept a Union that includes DataFrame?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059651056,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059651056,IC_kwDOCGYnMM4_KP3w,9599,simonw,2022-03-05T02:09:38Z,2022-03-05T02:09:38Z,OWNER,"OK, so reading results from existing `sqlite-utils` into a Pandas DataFrame turns out to be trivial. How about writing a DataFrame to a database table? That feels like it could a lot more useful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059650190,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059650190,IC_kwDOCGYnMM4_KPqO,9599,simonw,2022-03-05T02:04:43Z,2022-03-05T02:04:54Z,OWNER,"To be honest, I'm having second thoughts about this now mainly because the idiom for turning a generator of dicts into a DataFrame is SO simple: ```python df = pd.DataFrame(db.query(""select * from articles"")) ``` Given it's that simple, I'm questioning if there's any value to adding this to `sqlite-utils` at all. This likely becomes a documentation thing instead!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649803,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059649803,IC_kwDOCGYnMM4_KPkL,9599,simonw,2022-03-05T02:02:41Z,2022-03-05T02:02:41Z,OWNER,"It looks like the existing `pd.read_sql_query()` method has an optional dependency on SQLAlchemy: ``` ... import pandas as pd pd.read_sql_query(db.conn, ""select * from articles"") # ImportError: Using URI string without sqlalchemy installed. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649213,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059649213,IC_kwDOCGYnMM4_KPa9,9599,simonw,2022-03-05T02:00:10Z,2022-03-05T02:00:10Z,OWNER,Requested feedback on Twitter here :https://twitter.com/simonw/status/1499927075930578948,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649193,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059649193,IC_kwDOCGYnMM4_KPap,9599,simonw,2022-03-05T02:00:02Z,2022-03-05T02:00:02Z,OWNER,"Yeah, I imagine there are plenty of ways to do this with Pandas already - I'm opportunistically looking for a way to provide better integration with the rest of the Pandas situation from the work I've done in `sqlite-utils` already. Might be that this isn't worth doing at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059647114,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059647114,IC_kwDOCGYnMM4_KO6K,25778,eyeseast,2022-03-05T01:54:24Z,2022-03-05T01:54:24Z,CONTRIBUTOR,"I haven't tried this, but it looks like Pandas has a method for this: https://pandas.pydata.org/docs/reference/api/pandas.read_sql_query.html ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646645,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059646645,IC_kwDOCGYnMM4_KOy1,9599,simonw,2022-03-05T01:53:10Z,2022-03-05T01:53:10Z,OWNER,I'm not an experienced enough Pandas user to know if this design is right or not. I'm going to leave this open for a while and solicit some feedback.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646543,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059646543,IC_kwDOCGYnMM4_KOxP,9599,simonw,2022-03-05T01:52:47Z,2022-03-05T01:52:47Z,OWNER,"I built a prototype of that second option and it looks pretty good: Here's the `pandas.py` prototype: ```python from .db import Database as _Database, Table as _Table, View as _View import pandas as pd from typing import ( Iterable, Union, Optional, ) class Database(_Database): def query( self, sql: str, params: Optional[Union[Iterable, dict]] = None ) -> pd.DataFrame: return pd.DataFrame(super().query(sql, params)) def table(self, table_name: str, **kwargs) -> Union[""Table"", ""View""]: ""Return a table object, optionally configured with default options."" klass = View if table_name in self.view_names() else Table return klass(self, table_name, **kwargs) class PandasQueryable: def rows_where( self, where: str = None, where_args: Optional[Union[Iterable, dict]] = None, order_by: str = None, select: str = ""*"", limit: int = None, offset: int = None, ) -> pd.DataFrame: return pd.DataFrame( super().rows_where( where, where_args, order_by=order_by, select=select, limit=limit, offset=offset, ) ) class Table(PandasQueryable, _Table): pass class View(PandasQueryable, _View): pass ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646247,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059646247,IC_kwDOCGYnMM4_KOsn,9599,simonw,2022-03-05T01:51:03Z,2022-03-05T01:51:03Z,OWNER,"I considered two ways of doing this. First, have methods such as `db.query_df()` and `table.rows_df` which do the same as `.query()` and `table.rows` but return a DataFrame instead of a generator of dictionaries. Second, have a compatibility class that is imported separately such as: ```python from sqlite_utils.pandas import Database ``` Then have the `.query()` and `.rows` and other similar methods return dataframes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065597709,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065597709,IC_kwDOCGYnMM4_g7sN,9599,simonw,2022-03-11T22:32:43Z,2022-03-11T22:32:43Z,OWNER,"Trying to figure out what that extra field in `table_info` compared to `table_xinfo` is: ``` >>> list(db.query(""PRAGMA table_xinfo('t')"")) [{'cid': 0, 'name': 'body', 'type': 'TEXT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 0}, {'cid': 1, 'name': 'd', 'type': 'INT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 2}] `` Presumably `hidden` 0 v.s 2 v.s. other values has meaning.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065596417,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065596417,IC_kwDOCGYnMM4_g7YB,9599,simonw,2022-03-11T22:30:15Z,2022-03-11T22:30:15Z,OWNER,"I tried it out in Jupyter and it works as advertised: Introspection is a bit weird: there doesn't seem to be a way to introspect generated columns outside of parsing the stored SQL schema for the columns at the moment! And the `.columns` method doesn't return them at all: https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L1207-L1213 Here's why: ``` >>> db.execute(""PRAGMA table_info('t')"").fetchall() [(0, 'body', 'TEXT', 0, None, 0)] >>> db.execute(""PRAGMA table_xinfo('t')"").fetchall() [(0, 'body', 'TEXT', 0, None, 0, 0), (1, 'd', 'INT', 0, None, 0, 2)] ``` So `table_xinfo()` is needed to get back columns including generated columns: https://www.sqlite.org/pragma.html#pragma_table_xinfo > **PRAGMA** *schema.***table_xinfo(***table-name***);** > > This pragma returns one row for each column in the named table, including [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables. The output is the same as for [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that hidden columns are shown rather than being omitted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065477258,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065477258,IC_kwDOCGYnMM4_geSK,25778,eyeseast,2022-03-11T20:14:59Z,2022-03-11T20:14:59Z,CONTRIBUTOR,"Good call on adding this to `create-table`, especially for stored columns. Having the stored/virtual split might make this tricky to implement, but I haven't gone any farther than thinking about what the CLI looks like. I'm going to try making the SQL side work first and figure that'll tell me more about what it needs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065458729,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065458729,IC_kwDOCGYnMM4_gZwp,9599,simonw,2022-03-11T19:58:50Z,2022-03-11T20:00:25Z,OWNER,"I'm coming round to your suggestion to have this as extra arguments to `sqlite-utils add-column` now, especially since you also need to pass a column type. I'd like to come up with syntax for `sqlite-utils create-table` as well. https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table Maybe extra `--generated-stored colname expression` (and `--generated`) options would work there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065440445,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065440445,IC_kwDOCGYnMM4_gVS9,9599,simonw,2022-03-11T19:52:15Z,2022-03-11T19:52:15Z,OWNER,"Two new parameters to `.create_table()` and friends: - `generated={...}` - generated column definitions - `generated_stored={...}` generated stored column definitions These columns will be added at the end of the table, but you can use the `column_order=` parameter to apply a different order.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065402557,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065402557,IC_kwDOCGYnMM4_gMC9,9599,simonw,2022-03-11T19:01:08Z,2022-03-11T21:42:25Z,OWNER,"Just spotted this in https://www.sqlite.org/gencol.html > The only functional difference is that one cannot add new STORED columns using the [ALTER TABLE ADD COLUMN](https://www.sqlite.org/lang_altertable.html#altertabaddcol) command. Only VIRTUAL columns can be added using ALTER TABLE. So to add stored columns to an existing table we would need to use the `.transform()` trick. Which implies that this should actually be a capability of the various `.create()` methods, since transform works by creating a new table with those and then copying across the old data. Here's where `.transform()` calls `.create_table_sql()` under the hood: https://github.com/simonw/sqlite-utils/blob/9388edf57aa15719095e3cf0952c1653cd070c9b/sqlite_utils/db.py#L1627-L1637","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065389386,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065389386,IC_kwDOCGYnMM4_gI1K,9599,simonw,2022-03-11T18:42:53Z,2022-03-11T21:40:51Z,OWNER,"The Python API could be: ```python db[table_name].add_generated_column(""field"", str, ""json_extract(data, '$.field')"", stored=True) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065386352,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065386352,IC_kwDOCGYnMM4_gIFw,9599,simonw,2022-03-11T18:41:37Z,2022-03-11T18:41:37Z,OWNER,"I like `add-generated-column` - feels very clear to me, and is a nice place for adding logic that checks if the DB version supports it or not and shows a useful error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1264223554,https://api.github.com/repos/simonw/sqlite-utils/issues/409,1264223554,IC_kwDOCGYnMM5LWoVC,7908073,chapmanjacobd,2022-10-01T03:42:50Z,2022-10-01T03:42:50Z,CONTRIBUTOR,oh weird. it inserts into db2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149661489,`with db:` for transactions, https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1264223363,https://api.github.com/repos/simonw/sqlite-utils/issues/409,1264223363,IC_kwDOCGYnMM5LWoSD,7908073,chapmanjacobd,2022-10-01T03:41:45Z,2022-10-01T03:41:45Z,CONTRIBUTOR,"``` pytest xklb/check.py --pdb xklb/check.py:11: in test_transaction assert list(db2[""t""].rows) == [] E AssertionError: assert [{'foo': 1}] == [] E + where [{'foo': 1}] = list() E + where =
.rows >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > /home/xk/github/xk/lb/xklb/check.py(11)test_transaction() 9 with db1.conn: 10 db1[""t""].insert({""foo"": 1}) ---> 11 assert list(db2[""t""].rows) == [] 12 assert list(db2[""t""].rows) == [{""foo"": 1}] ``` It fails because it is already inserted. btw if you put these two lines in you pyproject.toml you can get `ipdb` in pytest ``` [tool.pytest.ini_options] addopts = ""--pdbcls=IPython.terminal.debugger:TerminalPdb --ignore=tests/data --capture=tee-sys --log-cli-level=ERROR"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149661489,`with db:` for transactions, https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1055930639,https://api.github.com/repos/simonw/sqlite-utils/issues/409,1055930639,IC_kwDOCGYnMM4-8DkP,9599,simonw,2022-03-01T22:40:15Z,2022-03-01T22:40:15Z,OWNER,"This test fails and I don't understand why: ```python from sqlite_utils import Database def test_transaction(): db1 = Database(memory_name=""transaction_test"", tracer=print) db2 = Database(memory_name=""transaction_test"", tracer=print) with db1.conn: db1[""t""].insert({""foo"": 1}) assert list(db2[""t""].rows) == [] assert list(db2[""t""].rows) == [{""foo"": 1}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149661489,`with db:` for transactions, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1066139147,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1066139147,IC_kwDOCGYnMM4_i_4L,24938923,learning4life,2022-03-13T16:45:00Z,2022-03-13T16:54:09Z,NONE,"@simonw Now I get this: ``` (app-root) sqlite-utils indexes global.db --table Error: near ""("": syntax error (app-root) sqlite-utils --version sqlite-utils, version 3.25.1 (app-root) sqlite3 --version 3.36.0 2021-06-18 18:36:39 (app-root) python --version Python 3.8.11 ``` Dockerfile ``` FROM centos/python-38-centos7 USER root RUN yum update -y RUN yum upgrade -y # epel RUN yum -y install epel-release && yum clean all # SQLite RUN yum -y install zlib-devel geos geos-devel proj proj-devel freexl freexl-devel libxml2-devel WORKDIR /build/ COPY sqlite-autoconf-3360000.tar.gz ./ RUN tar -zxf sqlite-autoconf-3360000.tar.gz WORKDIR /build/sqlite-autoconf-3360000 RUN ./configure RUN make RUN make install # RUN /opt/app-root/bin/python3.8 -m pip install --upgrade pip RUN pip install sqlite-utils ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1059097969,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1059097969,IC_kwDOCGYnMM4_II1x,24938923,learning4life,2022-03-04T11:55:21Z,2022-03-04T11:55:21Z,NONE,"Thanks @simonw I will test it after my vacation 👍","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1056001414,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1056001414,IC_kwDOCGYnMM4-8U2G,9599,simonw,2022-03-02T00:20:26Z,2022-03-02T00:20:26Z,OWNER,I need a `db.sqlite_version` property to implement this check.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1055996626,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1055996626,IC_kwDOCGYnMM4-8TrS,9599,simonw,2022-03-02T00:12:21Z,2022-03-02T00:12:21Z,OWNER,Here's the SQLite changelog mentioning that it was added in 3.8.3: https://www.sqlite.org/changes.html#version_3_8_3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1055995100,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1055995100,IC_kwDOCGYnMM4-8TTc,9599,simonw,2022-03-02T00:10:41Z,2022-03-02T00:10:41Z,OWNER,"Here's the code in question: https://github.com/simonw/sqlite-utils/blob/521921b849003ed3742338f76f9d47ff3d95eaf3/sqlite_utils/db.py#L384-L394 It's checking for Python 3.8, because that's the version of Python that added the `deterministic=True` option: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.create_function But from your error message it looks like it should be checking the SQLite version too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1055993700,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1055993700,IC_kwDOCGYnMM4-8S9k,9599,simonw,2022-03-02T00:08:10Z,2022-03-02T00:08:10Z,OWNER,"I thought I'd made it so `deterministic=True` would be silently ignored in environments that don't support it, but clearly I missed a case here!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1248440137,https://api.github.com/repos/simonw/sqlite-utils/issues/406,1248440137,IC_kwDOCGYnMM5Kaa9J,82988,psychemedia,2022-09-15T18:13:50Z,2022-09-15T18:13:50Z,NONE,"I was wondering if you have any more thoughts on this? I have a tangible use case now: adding a ""vector"" column to a database to support semantic search using doc2vec embeddings ([example](https://psychemedia.github.io/storynotes/Lang_Doc2Vec.html); note that the `vtfunc` package may no longer be reliable...).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128466114,Creating tables with custom datatypes, https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1041363433,https://api.github.com/repos/simonw/sqlite-utils/issues/406,1041363433,IC_kwDOCGYnMM4-EfHp,82988,psychemedia,2022-02-16T10:57:03Z,2022-02-16T10:57:19Z,NONE,"Wondering if this actually relates to https://github.com/simonw/sqlite-utils/issues/402 ? I also wonder if this would be a sensible approach for eg registering `pint` based quantity conversions into and out of the db, perhaps storing the quantity as a serialised `magnitude measurement` single column string?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128466114,Creating tables with custom datatypes, https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1041313679,https://api.github.com/repos/simonw/sqlite-utils/issues/406,1041313679,IC_kwDOCGYnMM4-ES-P,82988,psychemedia,2022-02-16T09:59:51Z,2022-02-16T10:00:10Z,NONE,"The `CustomColumnType()` approach looks good. This pushes you into the mindspace that you are defining and working with a custom column type. When creating the table, you could then error, or at least warn, if someone wasn't setting a column on a `type` or a custom column type, which I guess is where `mypy` comes in?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128466114,Creating tables with custom datatypes, https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1040978032,https://api.github.com/repos/simonw/sqlite-utils/issues/406,1040978032,IC_kwDOCGYnMM4-DBBw,9599,simonw,2022-02-16T01:10:31Z,2022-02-16T01:10:31Z,OWNER,"Allowing custom strings in the `create()` method, as you suggest in your example, feels like a reasonable way to support this. ```python db[""dummy""].create({ ""title"": str, ""vector"": ""array"", }) ``` I'm slightly nervous about that just because people might accidentally use this without realizig what they are doing - passing `""column-name"": ""string""` for example when they should have used `""column-name"": str` in order to get a `TEXT` column. Alternatively, this could work: ```python db[""dummy""].create({ ""title"": str, ""vector"": CustomColumnType(""array"") }) ``` This would play better with `mypy` too I think.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128466114,Creating tables with custom datatypes, https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1040974519,https://api.github.com/repos/simonw/sqlite-utils/issues/406,1040974519,IC_kwDOCGYnMM4-DAK3,9599,simonw,2022-02-16T01:08:17Z,2022-02-16T01:08:17Z,OWNER,"I had no idea this was possible! I guess SQLite will allow any text string as the column type, defaulting to `TEXT` as the underlying default representation if it doesn't recognize the type.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128466114,Creating tables with custom datatypes, https://github.com/simonw/sqlite-utils/issues/405#issuecomment-1033425512,https://api.github.com/repos/simonw/sqlite-utils/issues/405,1033425512,IC_kwDOCGYnMM49mNJo,9599,simonw,2022-02-09T07:20:11Z,2022-02-09T07:20:11Z,OWNER,"Datasette's implementation: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/database.py#L73-L79 ```python if self.memory_name: uri = ""file:{}?mode=memory&cache=shared"".format(self.memory_name) conn = sqlite3.connect( uri, uri=True, check_same_thread=False, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128139375,"`Database(memory_name=""name"")` constructor argument", https://github.com/simonw/sqlite-utils/issues/405#issuecomment-1033424454,https://api.github.com/repos/simonw/sqlite-utils/issues/405,1033424454,IC_kwDOCGYnMM49mM5G,9599,simonw,2022-02-09T07:18:25Z,2022-02-09T07:18:25Z,OWNER,Writing tests against this is always a tiny bit fiddly since the created databases persist across the lifetime of the test run. Using randomly generated names helps.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128139375,"`Database(memory_name=""name"")` constructor argument", https://github.com/simonw/sqlite-utils/issues/404#issuecomment-1033410970,https://api.github.com/repos/simonw/sqlite-utils/issues/404,1033410970,IC_kwDOCGYnMM49mJma,9599,simonw,2022-02-09T06:56:35Z,2022-02-09T06:56:35Z,OWNER,https://sqlite-utils.datasette.io/en/latest/cli-reference.html#insert,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128120451,Add example of `--convert` to the help for `sqlite-utils insert`, https://github.com/simonw/sqlite-utils/issues/404#issuecomment-1033407778,https://api.github.com/repos/simonw/sqlite-utils/issues/404,1033407778,IC_kwDOCGYnMM49mI0i,9599,simonw,2022-02-09T06:50:26Z,2022-02-09T06:50:26Z,OWNER,"I'll use this: ``` sqlite-utils insert plants.db plants plants.csv --csv --convert ' return { ""name"": row[""name""].upper(), ""latitude"": float(row[""latitude""]), ""longitude"": float(row[""longitude""]), }' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128120451,Add example of `--convert` to the help for `sqlite-utils insert`, https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1033332570,https://api.github.com/repos/simonw/sqlite-utils/issues/403,1033332570,IC_kwDOCGYnMM49l2da,536941,fgregg,2022-02-09T04:22:43Z,2022-02-09T04:22:43Z,CONTRIBUTOR,dddoooope,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126692066,Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`, https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032987901,https://api.github.com/repos/simonw/sqlite-utils/issues/403,1032987901,IC_kwDOCGYnMM49kiT9,9599,simonw,2022-02-08T19:36:06Z,2022-02-08T19:36:06Z,OWNER,New documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#adding-a-primary-key-to-a-rowid-table,"{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126692066,Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`, https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032976720,https://api.github.com/repos/simonw/sqlite-utils/issues/403,1032976720,IC_kwDOCGYnMM49kflQ,9599,simonw,2022-02-08T19:23:05Z,2022-02-08T19:23:27Z,OWNER,"This is already possible using `sqlite-utils transform` like so: ``` % echo '[{""name"": ""Barry""}, {""name"": ""Sandra""}]' | sqlite-utils insert rowid.db records - % sqlite-utils schema rowid.db CREATE TABLE [records] ( [name] TEXT ); % sqlite-utils rows rowid.db records [{""name"": ""Barry""}, {""name"": ""Sandra""}] % sqlite-utils transform rowid.db records --pk id % sqlite-utils rows rowid.db records [{""id"": 1, ""name"": ""Barry""}, {""id"": 2, ""name"": ""Sandra""}] % sqlite-utils schema rowid.db CREATE TABLE ""records"" ( [id] INTEGER PRIMARY KEY, [name] TEXT ); % echo '[{""name"": ""Barry 2""}, {""name"": ""Sandra 2""}]' | sqlite-utils insert rowid.db records - % sqlite-utils rows rowid.db records [{""id"": 1, ""name"": ""Barry""}, {""id"": 2, ""name"": ""Sandra""}, {""id"": 3, ""name"": ""Barry 2""}, {""id"": 4, ""name"": ""Sandra 2""}] ``` It's not covered in the documentation though: https://sqlite-utils.datasette.io/en/3.23/cli.html#transforming-tables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126692066,Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`, https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032126353,https://api.github.com/repos/simonw/sqlite-utils/issues/403,1032126353,IC_kwDOCGYnMM49hP-R,536941,fgregg,2022-02-08T01:45:15Z,2022-02-08T01:45:31Z,CONTRIBUTOR,"you can hack something like this to achieve this result: `sqlite-utils convert my_database my_table rowid ""{'id': value}"" --multi`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126692066,Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1041325398,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1041325398,IC_kwDOCGYnMM4-EV1W,82988,psychemedia,2022-02-16T10:12:48Z,2022-02-16T10:18:55Z,NONE,"> My hunch is that the case where you want to consider input from more than one column will actually be pretty rare - the only case I can think of where I would want to do that is for latitude/longitude columns Other possible pairs: unconventional date/datetime and timezone pairs eg `2022-02-16::17.00, London`; or more generally, numerical value and unit of measurement pairs (eg if you want to cast into and out of different measurement units using packages like `pint`) or currencies etc. Actually, in that case, I guess you may be presenting things that are unit typed already, and so a conversion would need to parse things into an appropriate, possibly two column `value, unit` format. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1035057014,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1035057014,IC_kwDOCGYnMM49sbd2,25778,eyeseast,2022-02-10T15:30:28Z,2022-02-10T15:30:40Z,CONTRIBUTOR,"Yeah, the CLI experience is probably where any kind of multi-column, configured setup is going to fall apart. Sticking with GIS examples, one way I might think about this is using the [fiona CLI](https://fiona.readthedocs.io/en/latest/cli.html): ```sh # assuming a database is already created and has SpatiaLite fio cat boundary.shp | sqlite-utils insert boundaries --conversion geometry GeometryGeoJSON - ``` Anyway, very interested to see where you land here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1033428967,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1033428967,IC_kwDOCGYnMM49mN_n,9599,simonw,2022-02-09T07:25:44Z,2022-02-09T07:28:11Z,OWNER,"The CLI version of this could perhaps look like this: sqlite-utils insert a.db places places.json \ --conversion boundary GeometryGeoJSON This will treat the boundary key as GeoJSON. It's equivalent to passing `conversions={""boundary"": geometryGeoJSON}` The combined latitude/longitude case here can be handled by combining this with the existing `--convert` mechanism. Any `Conversion` subclass will be available to the CLI in this way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1033366312,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1033366312,IC_kwDOCGYnMM49l-so,9599,simonw,2022-02-09T05:28:11Z,2022-02-09T07:28:48Z,OWNER,"My hunch is that the case where you want to consider input from more than one column will actually be pretty rare - the only case I can think of where I would want to do that is for latitude/longitude columns - everything else that I'd want to use it for (which admittedly is still mostly SpatiaLite stuff) works against a single value. The reason I'm leaning towards using the constructor for the values is that I really like the look of this variant for common conversions: ```python db[""places""].insert( { ""name"": ""London"", ""boundary"": GeometryFromGeoJSON({...}) } ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032732242,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1032732242,IC_kwDOCGYnMM49jj5S,25778,eyeseast,2022-02-08T15:26:59Z,2022-02-08T15:26:59Z,CONTRIBUTOR,"What if you did something like this: ```python class Conversion: def __init__(self, *args, **kwargs): ""Put whatever settings you need here"" def python(self, row, column, value): # not sure on args here ""Python step to transform value"" return value def sql(self, row, column, value): ""Return the actual sql that goes in the insert/update step, and maybe params"" # value is the return of self.python() return value, [] ``` This way, you're always passing an instance, which has methods that do the conversion. (Or you're passing a SQL string, as you would now.) The `__init__` could take column names, or SRID, or whatever other setup state you need per row, but the row is getting processed with the `python` and `sql` methods (or whatever you want to call them). This is pretty rough, so do what you will with names and args and such. You'd then use it like this: ```python # subclass might be unneeded here, if methods are present class LngLatConversion(Conversion): def __init__(self, x=""longitude"", y=""latitude""): self.x = x self.y = y def python(self, row, column, value): x = row[self.x] y = row[self.y] return x, y def sql(self, row, column, value): # value is now a tuple, returned above s = ""GeomFromText(POINT(? ?))"" return s, value table.insert_all(rows, conversions={""point"": LngLatConversion(""lng"", ""lat""))} ``` I haven't thought through all the implementation details here, and it'll probably break in ways I haven't foreseen, but wanted to get this idea out of my head. Hope it helps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032296717,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1032296717,IC_kwDOCGYnMM49h5kN,9599,simonw,2022-02-08T07:35:46Z,2022-02-08T07:35:46Z,OWNER,"I'm going to write the documentation for this first, before the implementation, so I can see if it explains cleanly enough that the design appears to be sound.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032294365,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1032294365,IC_kwDOCGYnMM49h4_d,9599,simonw,2022-02-08T07:32:09Z,2022-02-08T07:34:41Z,OWNER,"I have an idea for how that third option could work - the one that creates a new column using values from the existing ones: ```python db[""places""].insert( { ""name"": ""London"", ""lng"": -0.118092, ""lat"": 51.509865, }, conversions={""point"": LongitudeLatitude(""lng"", ""lat"")}, ) ``` How about specifying that the values in that `conversion=` dictionary can be: - A SQL string fragment (as currently implemented) - A subclass of `Conversion` as described above - Or... a callable function that takes the row as an argument and returns either a `Conversion` subclass instance or a literal value to be jnserted into the database (a string, int or float) Then you could do this: ```python db[""places""].insert( { ""name"": ""London"", ""lng"": -0.118092, ""lat"": 51.509865, }, conversions={ ""point"": lambda row: LongitudeLatitude( row[""lng""], row[""lat""] ) } ) ``` Something I really like about this is that it expands the abilities of `conversions=` beyond the slightly obscure need to customize the SQL fragment into something that can solve other data insertion cleanup problems too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031791783,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1031791783,IC_kwDOCGYnMM49f-Sn,25778,eyeseast,2022-02-07T18:37:40Z,2022-02-07T18:37:40Z,CONTRIBUTOR,"I've never used it either, but it's interesting, right? Feel like I should try it for something. I'm trying to get my head around how this conversions feature might work, because I really like the idea of it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031787865,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1031787865,IC_kwDOCGYnMM49f9VZ,9599,simonw,2022-02-07T18:33:27Z,2022-02-07T18:33:27Z,OWNER,"Hah, that's interesting - I've never used that mechanism before so it wasn't something that came to mind. They seem to be using a pretty surprising trick there that takes advantage of SQLite allowing you to define a column ""type"" using a made-up type name, which you can then introspect later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031779460,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1031779460,IC_kwDOCGYnMM49f7SE,25778,eyeseast,2022-02-07T18:24:56Z,2022-02-07T18:24:56Z,CONTRIBUTOR,"I wonder if there's any overlap with the goals here and the `sqlite3` module's concept of adapters and converters: https://docs.python.org/3/library/sqlite3.html#sqlite-and-python-types I'm not sure that's _exactly_ what we're talking about here, but it might be a parallel with some useful ideas to borrow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030904948,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1030904948,IC_kwDOCGYnMM49clx0,9599,simonw,2022-02-06T20:09:42Z,2022-02-08T07:40:44Z,OWNER,"I think this is the code that needs to become aware of this system: https://github.com/simonw/sqlite-utils/blob/fea8c9bcc509bcae75e99ae8870f520103b9aa58/sqlite_utils/db.py#L2453-L2469 There's an earlier branch that runs for upserts which needs to be modified too: https://github.com/simonw/sqlite-utils/blob/fea8c9bcc509bcae75e99ae8870f520103b9aa58/sqlite_utils/db.py#L2417-L2440","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030902102,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1030902102,IC_kwDOCGYnMM49clFW,9599,simonw,2022-02-06T19:53:34Z,2022-02-08T07:40:34Z,OWNER,"I like the idea that the contract for `Conversion` (or rather for its subclasses) is that it can wrap a Python value and then return both the SQL fragment - e.g. `GeomFromText(?, 4326)` - and the values that should be used as the SQL parameters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030901853,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1030901853,IC_kwDOCGYnMM49clBd,9599,simonw,2022-02-06T19:52:10Z,2022-02-06T19:52:10Z,OWNER,"So the key idea here is to introduce a new abstract base class, `Conversion`, which has the following abilities: - Can wrap one or more Python values (if called using the constructor) such that the `.insert_all()` method knows how to transform those into a format that can be included in an insert - something like `GeomFromText(?, 4326)` with input `POINT(-0.118092 51.509865)` - Can be passed to `conversions={""point"": LongitudeLatitude}` in a way that then knows to apply that conversion to every value in the `""point""` key of the data being inserted. - Maybe also extend `conversions=` to allow the definition of additional keys that use as input other rows? That's the `conversions={""point"": LongitudeLatitude(""lng"", ""lat"")}` example above - it may not be possible to get this working with the rest of the design though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030901189,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1030901189,IC_kwDOCGYnMM49ck3F,9599,simonw,2022-02-06T19:48:36Z,2022-02-06T19:48:52Z,OWNER,"From [that thread](https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566), two extra ideas which it may be possible to support in a single implementation: ```python from sqlite_utils.conversions import LongitudeLatitude db[""places""].insert( { ""name"": ""London"", ""lng"": -0.118092, ""lat"": 51.509865, }, conversions={""point"": LongitudeLatitude(""lng"", ""lat"")}, ) ``` And ```python db[""places""].insert( { ""name"": ""London"", ""point"": LongitudeLatitude(-0.118092, 51.509865) } ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/401#issuecomment-1030735372,https://api.github.com/repos/simonw/sqlite-utils/issues/401,1030735372,IC_kwDOCGYnMM49b8YM,9599,simonw,2022-02-06T02:05:03Z,2022-02-06T02:05:03Z,OWNER,Improved version: https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-column-values-using-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125081640,Update SpatiaLite example in the documentation, https://github.com/simonw/sqlite-utils/issues/401#issuecomment-1030734937,https://api.github.com/repos/simonw/sqlite-utils/issues/401,1030734937,IC_kwDOCGYnMM49b8RZ,9599,simonw,2022-02-06T02:02:24Z,2022-02-06T02:02:24Z,OWNER,The example also doesn't work right now - the code that fetches from Who's On First gets a 403 forbidden error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125081640,Update SpatiaLite example in the documentation, https://github.com/simonw/sqlite-utils/issues/400#issuecomment-1030730748,https://api.github.com/repos/simonw/sqlite-utils/issues/400,1030730748,IC_kwDOCGYnMM49b7P8,9599,simonw,2022-02-06T01:34:46Z,2022-02-06T01:34:46Z,OWNER,"Actually this is not needed - there is already an option that does this, it's just called `--ignore` rather than `--if-not-exists`. The lack of consistency here is a little annoying, but not annoying enough to justify making a backwards incompatible change. ``` % sqlite-utils create-table --help Usage: sqlite-utils create-table [OPTIONS] PATH TABLE COLUMNS... Add a table with the specified columns. Columns should be specified using name, type pairs, for example: sqlite-utils create-table my.db people \ id integer \ name text \ height float \ photo blob --pk id Options: --pk TEXT Column to use as primary key --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --fk ... Column, other table, other column to set as a foreign key --ignore If table already exists, do nothing --replace If table already exists, replace it --load-extension TEXT SQLite extensions to load -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125077063,`sqlite-utils create-table` ... `--if-not-exists`, https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1548913065,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1548913065,IC_kwDOCGYnMM5cUomp,433780,chrislkeller,2023-05-16T03:11:03Z,2023-05-16T03:11:52Z,NONE,"Using this thread and some [other resources](https://sqlite-utils.datasette.io/en/stable/cli.html#spatialite-helpers) I managed to cobble together a couple of sqlite-utils lines to add a geometry column for a table that already has a lat/lng column. ``` # add a geometry column sqlite-utils add-geometry-column [db name] [table name] geometry --type POINT --srid 4326 # add a point for each row to geometry column sqlite-utils --load-extension=spatialite [db name] 'update [table name] SET Geometry=MakePoint(longitude, latitude, 4326);' ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1077671779,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1077671779,IC_kwDOCGYnMM5AO_dj,25778,eyeseast,2022-03-24T14:11:33Z,2022-03-24T14:11:43Z,CONTRIBUTOR,"Coming back to this. I was about to add a utility function to [datasette-geojson]() to convert lat/lng columns to geometries. Thankfully I googled first. There's a SpatiaLite function for this: [MakePoint](https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html#p0). ```sql select MakePoint(longitude, latitude) as geometry from places; ``` I'm not sure if that would work with `conversions`, since it needs two columns, but it's an option for tables that already have latitude, longitude columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030902158,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030902158,IC_kwDOCGYnMM49clGO,9599,simonw,2022-02-06T19:53:54Z,2022-02-06T19:53:54Z,OWNER,"Moving the design of this new `Conversion` subclass mechanism to: - https://github.com/simonw/sqlite-utils/issues/402","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030871591,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030871591,IC_kwDOCGYnMM49cdon,9599,simonw,2022-02-06T16:57:22Z,2022-02-06T16:57:22Z,OWNER,"I wonder if I could implement the above such that this *also* works: ```python db[""places""].insert( { ""name"": ""London"", ""point"": LongitudeLatitude(-0.118092, 51.509865) } ) ``` This feels like a very natural way to work with single inserts. The challenge is writing the code inside `.insert_all()` such that it can handle these special objects in the input column values in addition to them being passed in `conversions=`. I'm feeling very good about this direction in general though, it feels like it takes the existing but not particularly elegant `conversions=` mechanism and upgrades it to be far more useful, while maintaining backwards compatibility.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030807433,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030807433,IC_kwDOCGYnMM49cN-J,6025893,chris48s,2022-02-06T10:54:09Z,2022-02-06T10:54:09Z,NONE,"> Interesting that some accept an SRID and others do not - presumably GeomFromGeoJSON() always uses SRID=4326? The ewtk/ewkb ones don't accept an SRID is because ewkt encodes the SRID in the string, so you would do this with a wkt string: `GeomFromText('POINT(529090 179645)', 27700)` but for ewkt it would be `GeomFromEWKT('SRID=27700;POINT(529090 179645)')` The specs for KML and GeoJSON specify a Coordinate Reference System for the format - https://datatracker.ietf.org/doc/html/rfc7946#section-4 - https://docs.opengeospatial.org/is/12-007r2/12-007r2.html#1274 GML can specify the SRID in the XML at feature level e.g: ``` 529090, 179645 ``` There's a few more obscure formats in there, but broadly I think it is safe to assume an SRID param exists on the function for cases where the SRID is not implied by or specified in the input format.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030741289,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030741289,IC_kwDOCGYnMM49b90p,25778,eyeseast,2022-02-06T03:03:43Z,2022-02-06T03:03:43Z,CONTRIBUTOR,"> I wonder if there are any interesting non-geospatial canned conversions that it would be worth including? Off the top of my head: - Un-nesting JSON objects into columns - Splitting arrays - Normalizing dates and times - URL munging with `urlparse` - Converting strings to numbers Some of this is easy enough with SQL functions, some is easier in Python. Maybe that's where having pre-built classes gets really handy, because it saves you from thinking about which way it's implemented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740963,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740963,IC_kwDOCGYnMM49b9vj,9599,simonw,2022-02-06T03:00:33Z,2022-02-06T03:00:33Z,OWNER,"Yeah, having this be a general purpose mechanism which has a few canned examples for handling geospatial stuff is a lot neater than having a mechanism for this that's exclusive to SpatiaLite.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740846,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740846,IC_kwDOCGYnMM49b9tu,9599,simonw,2022-02-06T02:59:21Z,2022-02-06T02:59:21Z,OWNER,I wonder if there are any interesting non-geospatial canned conversions that it would be worth including?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740826,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740826,IC_kwDOCGYnMM49b9ta,25778,eyeseast,2022-02-06T02:59:10Z,2022-02-06T02:59:10Z,CONTRIBUTOR,"All this said, I don't think it's unreasonable to point people to dedicated tools like `geojson-to-sqlite`. If I'm dealing with a bunch of GeoJSON or Shapefiles, I need to something to read those anyway (or I need to figure out virtual tables). But something like this might make it easier to build those libraries, or standardize the underlying parts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740771,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740771,IC_kwDOCGYnMM49b9sj,9599,simonw,2022-02-06T02:58:29Z,2022-02-06T02:58:29Z,OWNER,That example you have there is really neat - I like the idea that they can also be used to populate completely new columns that are derived from the other column inputs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740653,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740653,IC_kwDOCGYnMM49b9qt,25778,eyeseast,2022-02-06T02:57:17Z,2022-02-06T02:57:17Z,CONTRIBUTOR,"I like the idea of having stock conversions you could import. I'd actually move them to a dedicated module (call it `sqlite_utils.conversions` or something), because it's different from other utilities. Maybe they even take configuration, or they're composable. ```python from sqlite_utils.conversions import LongitudeLatitude db[""places""].insert( { ""name"": ""London"", ""lng"": -0.118092, ""lat"": 51.509865, }, conversions={""point"": LongitudeLatitude(""lng"", ""lat"")}, ) ``` I would definitely use that for every CSV I get with lat/lng columns where I actually need GeoJSON.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740570,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740570,IC_kwDOCGYnMM49b9pa,9599,simonw,2022-02-06T02:56:17Z,2022-02-06T02:57:00Z,OWNER,"Thinking about types. The type of the `conversions` parameter right now is a bit lazy: ```python conversions: Optional[dict] = None, ``` That becomes: ```python Optional[Dict[str, Union[str, Conversion]]] ``` Where `Conversion` is an abstract base class which expects implementations to have a `.sql() -> str` and a `.convert(value) -> str` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030739566,IC_kwDOCGYnMM49b9Zu,9599,simonw,2022-02-06T02:45:25Z,2022-02-06T02:50:27Z,OWNER,"Another idea - my favourite option so far: ```python from sqlite_utils.utils import LongitudeLatitude db[""places""].insert( { ""name"": ""London"", ""point"": (-0.118092, 51.509865) }, conversions={""point"": LongitudeLatitude}, ) ``` Here `LongitudeLatitude` is a magical value which does TWO things: it sets up the `GeomFromText(?, 4326)` SQL function, and it handles converting the `(51.509865, -0.118092)` tuple into a `POINT({} {})` string. This would involve a change to the `conversions=` contract - where it usually expects a SQL string fragment, but it can also take an object which combines that SQL string fragment with a Python conversion function. Best of all... this resolves the `lat, lon` v.s. `lon, lat` dilemma because you can use `from sqlite_utils.utils import LongitudeLatitude` OR `from sqlite_utils.utils import LatitudeLongitude` depending on which you prefer!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030738023,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030738023,IC_kwDOCGYnMM49b9Bn,9599,simonw,2022-02-06T02:28:05Z,2022-02-06T02:29:24Z,OWNER,"Here's the definitive guide to `latitude, longitude` v.s. `longitude, latitude`: https://macwright.com/lonlat/ > Which is right? > > Neither. This is an opinion with no right answer. Geographical tradition favors lat, lon. Math and software prefer lon, lat. I asked on Twitter here: https://twitter.com/simonw/status/1490148001569906688","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736848,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030736848,IC_kwDOCGYnMM49b8vQ,9599,simonw,2022-02-06T02:17:35Z,2022-02-06T02:17:35Z,OWNER,"Note that GeoJSON itself uses `(longitude, latitude)` so I should probably stick to that order here too. https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736589,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030736589,IC_kwDOCGYnMM49b8rN,9599,simonw,2022-02-06T02:14:52Z,2022-02-06T02:14:52Z,OWNER,"Another idea: introduce a helper function transform pattern, something a bit like this: ```python transformer = make_transformer({ ""point"": lambda pair: ""POINT({} {})"".format(pair[1], pair[0]) }) db[""places""].insert_all( transformer([{""name"": ""London"", ""point"": (51.509865, -0.118092)}]) conversions={""point"": ""GeomFromText(?, 4326)""}, ) ``` The `make_transformer(...)` function builds an object that can work as a wrapping iterator, applying those transform functions to everything in the sequence that it wraps. So the above code would handle converting `(lat, lon)` to `POINT(lon lat)` - then the `conversions=` applies `GeomFromText`. Naming is a challenge here: `.transform()` and `.convert()` and `conversions=` all have existing meanings within the `sqlite-utils` Python library. It's also a bit of a messy way of solving this. It's not exactly a smooth API for inserting a bunch of lat/lon coordinate pairs!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736047,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030736047,IC_kwDOCGYnMM49b8iv,9599,simonw,2022-02-06T02:10:18Z,2022-02-06T02:10:18Z,OWNER,"So maybe back to that earlier idea where the code introspects the table, figures out that `""point""` is a geometry table of type POINT, then applies the necessary conversions to the raw Python data? That feels overly-complicated to me, especially since nothing else in the `.insert()` method currently relies on table introspection.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030735774,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030735774,IC_kwDOCGYnMM49b8ee,9599,simonw,2022-02-06T02:08:19Z,2022-02-06T02:08:59Z,OWNER,"Maybe I should leave this entirely up to documented patterns in the `conversions={}` dictionary? But even that's not ideal for the co-ordinate case. Consider the following: ```python db[""places""].insert( {""name"": ""London"", ""point"": (51.509865, -0.118092)}, conversions={""point"": ""GeomFromText(?, 4326)""}, ) ``` The challenge here is that the SpatiaLite function `GeomFromText()` expects a WKT string, which looks like this: POINT(-0.118092 51.509865) The existing `conversions=` mechanism doesn't support applying Python code to convert the `(lat, lon)` tuple to that value. It doesn't even support passing a Python tuple as a `?` parameter - so I don't think I could come up with a SQL string that would do the right thing here either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030732909,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030732909,IC_kwDOCGYnMM49b7xt,9599,simonw,2022-02-06T01:47:06Z,2022-02-06T01:47:06Z,OWNER,"Here's an idea for an API design: ```python geojson_geometry = {} # ... GeoJSON goes here db[""places""].insert( {""name"": ""Wales"", ""geometry"": geojson_geometry}, geojson=""geometry"" ) ``` That `geojson=` parameter takes either a single column name or an iterable of column names. Any column in that list is expected to be a compatible `geometry` and the correct conversion functions will be applied. That solves for GeoJSON, but it's a bit ugly. Should I add `wkt=` and maybe even `kml=` and `gml=` and so-on too? Definitely not, that's way too many ugly and inscrutable new parameters. More importantly: if I want to support the following how would I do it? ```python db[""places""].insert( {""name"": ""London"", ""point"": (51.509865, -0.118092)} ) ``` Here I want to provide a `(latitude, longitude)` pair and have it inserted correctly into a `point` column. Could do this, but again it's messy: ```python db[""places""].insert( {""name"": ""London"", ""point"": (51.509865, -0.118092)}, point=""point"" ) ``` And again, what about those `(longitude, latitude)` people?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030712129,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030712129,IC_kwDOCGYnMM49b2tB,9599,simonw,2022-02-05T23:08:45Z,2022-02-05T23:08:45Z,OWNER,"Useful thoughts on Twitter regarding making coordinate pairs easy and more complex shapes possible: https://twitter.com/dbreunig/status/1490099303888547843 > That is exactly where I was going: two modes. > > 1. Heuristics and assumptions to get coordinates as a pair (in tuple) or as columns (look for lat, lon, latitude, longitude, etc). > 2. GIS mode with projections, polys, etc > > Make it easy for people with csvs of coordinates. If you're using Geojson or shp files, you have to specify.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030468418,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030468418,IC_kwDOCGYnMM49a7NC,9599,simonw,2022-02-05T00:49:08Z,2022-02-05T22:59:06Z,OWNER,"I'm trying to think of ways to make this nicer from the perspective of someone calling the `.insert()` or `.insert_all()` methods against a table that has geometry columns. One option would be for the code to introspect the table (if it exists) before running the insert, looking for any geometry columns. This introspection isn't easy! The table schema just gives you `""name_of_column"" point` or similar - to figure out the SRID and suchlike you need to consult the `geometry_columns` table, I think - which throws a 500 error on https://calands.datasettes.com/calands/geometry_columns for some reason. Also does the shape of that table change between SpatiaLite versions? Assuming we can introspect the table, what would we do with that information? We could add code that detects if the user attempted to pass GeoJSON objects and automatically inserts a `GeomFromGeoJSON()` function call - but detecting GeoJSON is a bit weird, and GeoJSON also isn't necessarily the nicest format for populating e.g. latitude/longitude points. Maybe we just support the simplest possible case: a tuple of floats, which we assume is `latitude, longitude` (or should we expect `longitude, latitude`, the eternal debate?) - if those are used against a geometry table (especially a point table) we assume they are coordinates that need to be converted using `GeomFromText('POINT(...`. Not crazy about either of these ideas. Is there something better?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030466255,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030466255,IC_kwDOCGYnMM49a6rP,9599,simonw,2022-02-05T00:41:35Z,2022-02-05T00:42:23Z,OWNER,"Wow, it was the newlines that broke it! This works fine: ```sql select AsWKT(SetSRID(GeomFromGeoJSON('{""type"": ""Point"",""coordinates"": [-94.921875,45.460130637921004]}'), 4326)) ``` https://calands.datasettes.com/calands?sql=select+AsWKT%28SetSRID%28GeomFromGeoJSON%28%27%7B%22type%22%3A+%22Point%22%2C%22coordinates%22%3A+%5B-94.921875%2C45.460130637921004%5D%7D%27%29%2C+4326%29%29 And removing `SetSRID()` returns exactly the same result: https://calands.datasettes.com/calands?sql=select+AsWKT%28GeomFromGeoJSON%28%27%7B%22type%22%3A+%22Point%22%2C%22coordinates%22%3A+%5B-94.921875%2C45.460130637921004%5D%7D%27%29%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030465557,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030465557,IC_kwDOCGYnMM49a6gV,9599,simonw,2022-02-05T00:39:09Z,2022-02-05T00:39:09Z,OWNER,"I can't seem to get `GeomFromGeoJSON()` to work - example: https://calands.datasettes.com/calands?sql=select+IsValid%28SetSRID%28GeomFromGeoJSON%28%27%7B%0D%0A++++++++%22type%22%3A+%22Point%22%2C%0D%0A++++++++%22coordinates%22%3A+%5B%0D%0A++++++++++-94.921875%2C%0D%0A++++++++++45.460130637921004%0D%0A++++++++%5D%0D%0A++++++%7D%27%29%2C+4326%29%29 ```sql select IsValid(SetSRID(GeomFromGeoJSON('{ ""type"": ""Point"", ""coordinates"": [ -94.921875, 45.460130637921004 ] }'), 4326)) ``` Returns `-1` suggesting the geometry is not valid. Just doing this (with or without that `SetSRID()` function) returns null: ```sql select SetSRID(GeomFromGeoJSON('{ ""type"": ""Point"", ""coordinates"": [ -94.921875, 45.460130637921004 ] }'), 4326) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030461163,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030461163,IC_kwDOCGYnMM49a5br,9599,simonw,2022-02-05T00:30:18Z,2022-02-05T00:30:18Z,OWNER,"I wonder what the most developer-friendly way to insert geometry data into SpatiaLite is? From https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html it looks like these are the main options: - `GeomFromText( wkt String [ , SRID Integer] )` - `GeomFromWKB( wkbGeometry Binary [ , SRID Integer] )` - `GeomFromKml( KmlGeometry String )` - `GeomFromGML( gmlGeometry String )` - `GeomFromGeoJSON( geoJSONGeometry String )` - `GeomFromEWKB( ewkbGeometry String )` - `GeomFromEWKT( ewktGeometry String )` - `GeomFromFGF( fgfGeometry Binary [ , SRID Integer] )` - `GeomFromTWKB( twkbGeometry BLOB [ , SRID Integer] )` - `GeomFromGPB( geom GPKG Blob Geometry )` - GeoPackage format - `GeomFromExifGpsBlob( image BLOB )` Interesting that some accept an SRID and others do not - presumably `GeomFromGeoJSON()` always uses SRID=4326?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030455715,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030455715,IC_kwDOCGYnMM49a4Gj,9599,simonw,2022-02-05T00:15:28Z,2022-02-05T00:15:28Z,OWNER,"The `conversions=` argument to `.insert()` and friends is designed to handle this case, but I don't think it's very elegant: https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions ```python db[""places""].insert( {""name"": ""Wales"", ""geometry"": wkt}, conversions={""geometry"": ""GeomFromText(?, 4326)""}, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1040965672,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1040965672,IC_kwDOCGYnMM4-C-Ao,9599,simonw,2022-02-16T01:02:29Z,2022-02-16T01:02:29Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#create-database - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#add-geometry-column - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#create-spatial-index - https://sqlite-utils.datasette.io/en/latest/cli.html#spatialite-helpers","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1038336591,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1038336591,IC_kwDOCGYnMM4948JP,25778,eyeseast,2022-02-13T18:48:21Z,2022-02-13T18:49:49Z,CONTRIBUTOR,"Been chipping away at this between other things and realized `sqlite-utils init-spatialite` is probably unnecessary. Any of the other commands requires running `db.init_spatialite` to have the extension functions available, and that will do everything `init-spatialite` would do. I think it's probably worth keeping a SpatiaLite flag on `create-database` in case you wanted to create all the spatial metadata up front. Otherwise, it's going to get added the first time you run `add-geometry-column` or `create-spatial-index`, which is probably fine in most cases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030732222,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030732222,IC_kwDOCGYnMM49b7m-,9599,simonw,2022-02-06T01:42:19Z,2022-02-06T01:42:28Z,OWNER,"Adding some thoughts to: - #399 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030732093,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030732093,IC_kwDOCGYnMM49b7k9,9599,simonw,2022-02-06T01:41:37Z,2022-02-06T01:41:37Z,OWNER,Yeah I'd like to avoid adding any geo-dependencies to `sqlite-utils` if I can avoid it. I'm fine using stuff that's going to be available in SpatiaLite itself (provided it's available as a SQLite module) since then I don't need to add any extra Python dependencies.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030629879,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030629879,IC_kwDOCGYnMM49bin3,25778,eyeseast,2022-02-05T13:57:33Z,2022-02-05T19:49:38Z,CONTRIBUTOR,"I'm mostly using [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite) at the moment. Even with shapefiles, I'm usually converting to GeoJSON and projecting to EPSG:4326 (with [ogr2ogr](https://gdal.org/programs/ogr2ogr.html)) first. I think an open question here is how much you want to leave to external libraries and how much you want here. My thinking has been that adding Spatialite helpers here would make external stuff easier, but it would be nice to have some standard way to insert geometries. I'm in the middle of adding GeoJSON and Spatialite support to [geocode-sqlite](https://github.com/eyeseast/geocode-sqlite), and that will probably use WKT. Since that's all points, I think I can just make the string inline. But for polygons, I'd generally use Shapely, which probably isn't a dependency you want to add to sqlite-utils. I've also been trying to get some of the approaches [here](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=Supporting+GeoJSON) to work, but haven't had any success so far.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030534868,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030534868,IC_kwDOCGYnMM49bLbU,9599,simonw,2022-02-05T06:03:38Z,2022-02-05T06:03:38Z,OWNER,@eyeseast how do you usually insert geometries at the moment?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030521533,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030521533,IC_kwDOCGYnMM49bIK9,9599,simonw,2022-02-05T04:25:49Z,2022-02-05T04:25:49Z,OWNER,For ingesting geometry data from the command-line maybe GeoJSON would be the best route?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030456717,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030456717,IC_kwDOCGYnMM49a4WN,9599,simonw,2022-02-05T00:16:42Z,2022-02-05T00:16:42Z,OWNER,"> The one thing worth highlighting in docs is that geometry columns can only be added to existing tables. Trying to add a geometry column to a table that doesn't exist yet might mean you have a schema like `{""rowid"": int, ""geometry"": bytes}`. Might be worth nudging people to explicitly create a table first, then add geometry columns. That's a good call. I'm happy for `sqlite-utils add-geometry-column` to throw an error if the table doesn't exist yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030454114,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030454114,IC_kwDOCGYnMM49a3ti,9599,simonw,2022-02-05T00:14:47Z,2022-02-05T00:14:47Z,OWNER,"I like these designs a lot. I would suggest `sqlite-utils create database.db --init-spatialite` there for consistency with the `sqlite-utils init-spatialite database.db` command. The other part of this story is how we support actually inserting spatial data from the command-line. I opened an issue about the challenges in doing that for the Python API here - #399 - but we need a good answer for the CLI too. I don't yet have any good ideas here. The `conversions=` option in the Python library was designed to cover these kinds of cases but it's pretty clunky and I don't think it's very widely used: https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030730108,https://api.github.com/repos/simonw/sqlite-utils/issues/397,1030730108,IC_kwDOCGYnMM49b7F8,9599,simonw,2022-02-06T01:30:46Z,2022-02-06T01:30:46Z,OWNER,Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/python-api.html#explicitly-creating-a-table,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1123903919,Support IF NOT EXISTS for table creation, https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030727979,https://api.github.com/repos/simonw/sqlite-utils/issues/397,1030727979,IC_kwDOCGYnMM49b6kr,9599,simonw,2022-02-06T01:19:21Z,2022-02-06T01:19:21Z,OWNER,"Just noticed there's no explicit test coverage for the `db[""table""].create(...)` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1123903919,Support IF NOT EXISTS for table creation, https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030726991,https://api.github.com/repos/simonw/sqlite-utils/issues/397,1030726991,IC_kwDOCGYnMM49b6VP,9599,simonw,2022-02-06T01:13:58Z,2022-02-06T01:13:58Z,OWNER,This is a good idea. We already have that parameter for the `table.create_index()` method: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite_utils.db.Table.create_index,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1123903919,Support IF NOT EXISTS for table creation, https://github.com/simonw/sqlite-utils/issues/395#issuecomment-1029686150,https://api.github.com/repos/simonw/sqlite-utils/issues/395,1029686150,IC_kwDOCGYnMM49X8OG,9599,simonw,2022-02-04T06:03:51Z,2022-02-04T06:03:51Z,OWNER,I'm just going to run the SpatiaLite tests on Ubuntu.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1123849278,"""apt-get: command not found"" error on macOS", https://github.com/simonw/sqlite-utils/issues/394#issuecomment-1029402029,https://api.github.com/repos/simonw/sqlite-utils/issues/394,1029402029,IC_kwDOCGYnMM49W22t,9599,simonw,2022-02-03T21:06:35Z,2022-02-03T21:06:35Z,OWNER,"This broke on Windows: https://github.com/simonw/sqlite-utils/runs/5056912641 ``` if recreate and os.path.exists(filename_or_conn): > os.remove(filename_or_conn) E PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_recreate_False_True_0\\data.db' ``` I'm going to revert it from `main` for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122446693,Test against Python 3.11-dev, https://github.com/simonw/sqlite-utils/issues/393#issuecomment-1029450617,https://api.github.com/repos/simonw/sqlite-utils/issues/393,1029450617,IC_kwDOCGYnMM49XCt5,9599,simonw,2022-02-03T22:13:24Z,2022-02-03T22:13:24Z,OWNER,Much better: https://sqlite-utils.datasette.io/en/latest/python-api.html#insert-replacing-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1118585417,Better documentation for insert-replace, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1022466476,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1022466476,IC_kwDOCGYnMM488Zms,9599,simonw,2022-01-26T18:17:43Z,2022-01-26T18:17:43Z,OWNER,"Manually tested it like this: ``` # Create database with an empty ""lines"" table sqlite-utils create-table bulk-test.db lines line text # Stream records every 0.5s, commit every 5 records stream-delay docs/python-api.rst -d 500 | \ sqlite-utils bulk bulk-test.db 'insert into lines (line) values (:line)' - \ --lines --batch-size 5 ``` Running `datasette bulk-test.db` showed that records would show up about every 2.5s five at a time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101,`sqlite-utils bulk --batch-size` option, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021877769,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1021877769,IC_kwDOCGYnMM486J4J,9599,simonw,2022-01-26T05:19:48Z,2022-01-26T05:19:48Z,OWNER,Can use this utility function: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/utils.py#L322-L325,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101,`sqlite-utils bulk --batch-size` option, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021877058,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1021877058,IC_kwDOCGYnMM486JtC,9599,simonw,2022-01-26T05:18:12Z,2022-01-26T05:18:18Z,OWNER,"Help for `insert` says: ``` --batch-size INTEGER Commit every X records ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101,`sqlite-utils bulk --batch-size` option, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021876914,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1021876914,IC_kwDOCGYnMM486Jqy,9599,simonw,2022-01-26T05:17:49Z,2022-01-26T05:17:49Z,OWNER,Relevant code: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/cli.py#L1014-L1018,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101,`sqlite-utils bulk --batch-size` option, https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876463,https://api.github.com/repos/simonw/sqlite-utils/issues/391,1021876463,IC_kwDOCGYnMM486Jjv,9599,simonw,2022-01-26T05:16:51Z,2022-01-26T05:16:51Z,OWNER,"Actually adding a progress bar may not make sense here: it's designed to work with streaming input from stdin, in which case it's impossible for it to know the overall number of rows to be processed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114638930,`sqlite-utils bulk` progress bar, https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876055,https://api.github.com/repos/simonw/sqlite-utils/issues/391,1021876055,IC_kwDOCGYnMM486JdX,9599,simonw,2022-01-26T05:15:58Z,2022-01-26T05:15:58Z,OWNER,Could add support for `--batch-size` as seen in `insert`/`upsert` too - causing it to break the list up into batches and commit for each one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114638930,`sqlite-utils bulk` progress bar, https://github.com/simonw/sqlite-utils/issues/390#issuecomment-1021825322,https://api.github.com/repos/simonw/sqlite-utils/issues/390,1021825322,IC_kwDOCGYnMM4859Eq,9599,simonw,2022-01-26T03:20:12Z,2022-01-26T03:20:25Z,OWNER,"``` % sqlite-utils upsert trees.db blah - Usage: sqlite-utils upsert [OPTIONS] PATH TABLE FILE Try 'sqlite-utils upsert -h' for help. Error: Missing option '--pk'. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114557284,`sqlite-utils upsert` should require `--pk` more elegantly, https://github.com/simonw/sqlite-utils/issues/389#issuecomment-1021790136,https://api.github.com/repos/simonw/sqlite-utils/issues/389,1021790136,IC_kwDOCGYnMM4850e4,9599,simonw,2022-01-26T02:07:41Z,2022-01-26T02:07:41Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114544727,Plausible analytics for documentation, https://github.com/simonw/sqlite-utils/issues/389#issuecomment-1021789016,https://api.github.com/repos/simonw/sqlite-utils/issues/389,1021789016,IC_kwDOCGYnMM4850NY,9599,simonw,2022-01-26T02:05:41Z,2022-01-26T02:05:41Z,OWNER,"Oops, wrong domain in that - https://plausible.io/sqlite-utils.datasette.io is not yet showing data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114544727,Plausible analytics for documentation, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484277416,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1484277416,IC_kwDOCGYnMM5YeEao,9599,simonw,2023-03-26T23:43:12Z,2023-03-26T23:43:12Z,OWNER,Fixed: https://sqlite-utils.datasette.io/en/latest/reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484276302,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1484276302,IC_kwDOCGYnMM5YeEJO,9599,simonw,2023-03-26T23:39:27Z,2023-03-26T23:39:27Z,OWNER,"Actually final Datasette fix was: ```html+jinja {% block scripts %} {{ super() }} {% endblock %} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484269326,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1484269326,IC_kwDOCGYnMM5YeCcO,9599,simonw,2023-03-26T23:24:12Z,2023-03-26T23:28:14Z,OWNER,"This broke on the upgrade to Furo. I fixed it in Datasette like this: https://github.com/simonw/datasette/commit/5c1cfa451d78e3935193f5e10eba59bf741241a1 and https://github.com/simonw/datasette/commit/db8cf899e286fbaa0a40f3a9ae8d5aaa1478822e","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021788260,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021788260,IC_kwDOCGYnMM4850Bk,9599,simonw,2022-01-26T02:04:17Z,2022-01-26T02:04:17Z,OWNER,https://sqlite-utils.datasette.io/en/3.20/ now also shows the version warning banner.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021787822,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021787822,IC_kwDOCGYnMM485z6u,9599,simonw,2022-01-26T02:03:25Z,2022-01-26T02:03:25Z,OWNER,"https://sqlite-utils.datasette.io/en/latest/ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021785268,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021785268,IC_kwDOCGYnMM485zS0,9599,simonw,2022-01-26T01:58:19Z,2022-01-26T01:58:19Z,OWNER,"While I'm modifying that template I'm also going to add Plausible analytics: ```html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021784593,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021784593,IC_kwDOCGYnMM485zIR,9599,simonw,2022-01-26T01:56:48Z,2022-01-26T01:56:48Z,OWNER,"On https://readthedocs.org/dashboard/sqlite-utils/advanced/ I checked this box (previously unchecked): ![CleanShot 2022-01-25 at 17 56 22@2x](https://user-images.githubusercontent.com/9599/151090718-72e553a6-826a-43b6-98bd-f70b0cba8784.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/387#issuecomment-1021773056,https://api.github.com/repos/simonw/sqlite-utils/issues/387,1021773056,IC_kwDOCGYnMM485wUA,9599,simonw,2022-01-26T01:37:17Z,2022-01-26T01:37:17Z,OWNER,https://sqlite-utils.datasette.io/en/latest/python-api.html#getting-started,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1111293050,Python library docs should start with a self contained example, https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010440166,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010440166,IC_kwDOCGYnMM48Ohfm,9599,simonw,2022-01-11T22:55:05Z,2022-01-11T22:55:05Z,OWNER,Twitter thread about this: https://twitter.com/simonw/status/1481020195074293761,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`, https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010387223,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010387223,IC_kwDOCGYnMM48OUkX,9599,simonw,2022-01-11T21:45:32Z,2022-01-11T21:45:32Z,OWNER,The new page of documentation: https://sqlite-utils.datasette.io/en/latest/cli-reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`, https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010386802,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010386802,IC_kwDOCGYnMM48OUdy,9599,simonw,2022-01-11T21:44:53Z,2022-01-11T21:44:53Z,OWNER,Here's the `cog` code I used: https://github.com/simonw/sqlite-utils/blob/1d44b0cc2784c94aed1bcf350225cd86ee1aa7e5/docs/cli-reference.rst#L11-L76,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`, https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010333511,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010333511,IC_kwDOCGYnMM48OHdH,9599,simonw,2022-01-11T20:27:08Z,2022-01-11T20:27:08Z,OWNER,"I'll call the new page ""CLI reference"", for consistency with the API reference page here: https://sqlite-utils.datasette.io/en/stable/reference.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`, https://github.com/simonw/sqlite-utils/issues/382#issuecomment-1010461844,https://api.github.com/repos/simonw/sqlite-utils/issues/382,1010461844,IC_kwDOCGYnMM48OmyU,9599,simonw,2022-01-11T23:33:14Z,2022-01-11T23:33:14Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099585611,`--where` option for `sqlite-rows`, https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010462035,https://api.github.com/repos/simonw/sqlite-utils/issues/381,1010462035,IC_kwDOCGYnMM48Om1T,9599,simonw,2022-01-11T23:33:37Z,2022-01-11T23:33:37Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099584685,`sqlite-utils rows` options `--limit` and `--offset`, https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010441118,https://api.github.com/repos/simonw/sqlite-utils/issues/381,1010441118,IC_kwDOCGYnMM48Ohue,9599,simonw,2022-01-11T22:56:53Z,2022-01-11T22:57:09Z,OWNER,"`sqlite-utils search` has `--limit` already: https://sqlite-utils.datasette.io/en/latest/cli-reference.html#search ``` --limit INTEGER Number of rows to return - defaults to everything ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099584685,`sqlite-utils rows` options `--limit` and `--offset`, https://github.com/simonw/sqlite-utils/issues/380#issuecomment-1009544785,https://api.github.com/repos/simonw/sqlite-utils/issues/380,1009544785,IC_kwDOCGYnMM48LG5R,9599,simonw,2022-01-11T02:32:56Z,2022-01-11T02:32:56Z,OWNER,"CLI and Python library improvements to help run [ANALYZE](https://www.sqlite.org/lang_analyze.html) after creating indexes or inserting rows, to gain better performance from the SQLite query planner when it runs against indexes. Three new CLI commands: `create-database`, `analyze` and `bulk`. - New `sqlite-utils create-database` command for creating new empty database files. ([#348](https://github.com/simonw/sqlite-utils/issues/348)) - New Python methods for running `ANALYZE` against a database, table or index: `db.analyze()` and `table.analyze()`, see [Optimizing index usage with ANALYZE](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-analyze). ([#366](https://github.com/simonw/sqlite-utils/issues/366)) - New [sqlite-utils analyze command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-analyze) for running `ANALYZE` using the CLI. ([#379](https://github.com/simonw/sqlite-utils/issues/379)) - The `create-index`, `insert` and `update` commands now have a new `--analyze` option for running `ANALYZE` after the command has completed. ([#379](https://github.com/simonw/sqlite-utils/issues/379)) - New [sqlite-utils bulk command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-bulk) which can import records in the same way as `sqlite-utils insert` (from JSON, CSV or TSV) and use them to bulk execute a parametrized SQL query. ([#375](https://github.com/simonw/sqlite-utils/issues/375)) - The CLI tool can now also be run using `python -m sqlite_utils`. ([#368](https://github.com/simonw/sqlite-utils/issues/368)) - Using `--fmt` now implies `--table`, so you don't need to pass both options. ([#374](https://github.com/simonw/sqlite-utils/issues/374)) - The `--convert` function applied to rows can now modify the row in place. ([#371](https://github.com/simonw/sqlite-utils/issues/371)) - The [insert-files command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-insert-files) supports two new columns: `stem` and `suffix`. ([#372](https://github.com/simonw/sqlite-utils/issues/372)) - The `--nl` import option now ignores blank lines in the input. ([#376](https://github.com/simonw/sqlite-utils/issues/376)) - Fixed bug where streaming input to the `insert` command with `--batch-size 1` would appear to only commit after several rows had been ingested, due to unnecessary input buffering. ([#364](https://github.com/simonw/sqlite-utils/issues/364))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1098574572,Release notes for 3.21, https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1009536276,https://api.github.com/repos/simonw/sqlite-utils/issues/375,1009536276,IC_kwDOCGYnMM48LE0U,9599,simonw,2022-01-11T02:12:58Z,2022-01-11T02:12:58Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#executing-sql-in-bulk,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097251014,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008556706,https://api.github.com/repos/simonw/sqlite-utils/issues/375,1008556706,IC_kwDOCGYnMM48HVqi,9599,simonw,2022-01-10T05:33:41Z,2022-01-10T05:33:41Z,OWNER,"I tested the prototype like this: sqlite-utils blah.db 'create table blah (id integer primary key, name text)' echo 'id,name 1,Cleo 2,Chicken' > blah.csv sqlite-utils bulk blah.db 'insert into blah (id, name) values (:id, :name)' blah.csv --csv ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097251014,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008338186,https://api.github.com/repos/simonw/sqlite-utils/issues/375,1008338186,IC_kwDOCGYnMM48GgUK,9599,simonw,2022-01-09T17:13:33Z,2022-01-09T17:13:54Z,OWNER," cat blah.csv | sqlite-utils bulk blah.db - \ ""insert into blah (:foo, :bar)"" --csv ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097251014,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346841,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008346841,IC_kwDOCGYnMM48GibZ,9599,simonw,2022-01-09T18:06:50Z,2022-01-09T18:06:50Z,OWNER,"In addition to a unit test I manually tested all of the above, e.g. ``` % sqlite-utils indexes global-power-plants.db sqlite_master --fmt rst ======= ============ ======= ===== ====== ====== ====== ===== table index_name seqno cid name desc coll key ======= ============ ======= ===== ====== ====== ====== ===== ======= ============ ======= ===== ====== ====== ====== ===== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`, https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346338,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008346338,IC_kwDOCGYnMM48GiTi,9599,simonw,2022-01-09T18:03:22Z,2022-01-09T18:03:22Z,OWNER,"Commands that support `--fmt` (via the `@output_options` decorator) are: - `tables` - `views` - `query` - `memory` - `search` - `rows` - `triggers` - `indexes` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`, https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008345267,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008345267,IC_kwDOCGYnMM48GiCz,9599,simonw,2022-01-09T17:56:37Z,2022-01-09T17:56:37Z,OWNER,"Better: ```python if fmt: table = True ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`, https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008252732,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008252732,IC_kwDOCGYnMM48GLc8,9599,simonw,2022-01-09T08:25:30Z,2022-01-09T08:25:30Z,OWNER,Need to change `if table:` to `if table or fmt:` in a few places.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`, https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344980,https://api.github.com/repos/simonw/sqlite-utils/issues/373,1008344980,IC_kwDOCGYnMM48Gh-U,9599,simonw,2022-01-09T17:54:53Z,2022-01-09T17:54:53Z,OWNER,Updated TIL: https://til.simonwillison.net/python/cog-to-update-help-in-readme#user-content-cog-for-restructuredtext,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135732,List `--fmt` options in the docs , https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344525,https://api.github.com/repos/simonw/sqlite-utils/issues/373,1008344525,IC_kwDOCGYnMM48Gh3N,9599,simonw,2022-01-09T17:52:22Z,2022-01-09T17:52:22Z,OWNER,Updated docs: https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135732,List `--fmt` options in the docs , https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008341078,https://api.github.com/repos/simonw/sqlite-utils/issues/373,1008341078,IC_kwDOCGYnMM48GhBW,9599,simonw,2022-01-09T17:31:12Z,2022-01-09T17:31:12Z,OWNER,"Found an example of using `cog` in a rST file here: https://github.com/nedbat/coveragepy/blob/f3238eea7e403d13a217b30579b1a1c2cbff62e3/doc/dbschema.rst#L21 ``` .. [[[cog from coverage.sqldata import SCHEMA_VERSION print("".. code::"") print() print(f"" SCHEMA_VERSION = {SCHEMA_VERSION}"") print() .. ]]] .. code:: SCHEMA_VERSION = 7 .. [[[end]]] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135732,List `--fmt` options in the docs , https://github.com/simonw/sqlite-utils/issues/372#issuecomment-1008247370,https://api.github.com/repos/simonw/sqlite-utils/issues/372,1008247370,IC_kwDOCGYnMM48GKJK,9599,simonw,2022-01-09T07:51:18Z,2022-01-09T07:51:18Z,OWNER,"Pathlib says the stem of that would be `dogs.and.cats.jpg` - best stick with that for consistency. https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.suffix It calls the last bit `suffix` - maybe I should use that instead of `ext`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097129710,Idea: `suffix` and `stem` file columns, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008364701,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008364701,IC_kwDOCGYnMM48Gmyd,9599,simonw,2022-01-09T20:04:35Z,2022-01-09T20:04:35Z,OWNER,"The previous code for highlighting errors in syntax (which was already a bit confused thanks to the added `return`, see https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684 - isn't compatible with this approach at all. I'm going to ditch it and just show a generic `Error: Could not compile code` message.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008354207,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008354207,IC_kwDOCGYnMM48GkOf,9599,simonw,2022-01-09T18:54:54Z,2022-01-09T18:54:54Z,OWNER,"This seems to work: ```python def _compile_code(code, imports, variable=""value""): locals = {} globals = {""r"": recipes, ""recipes"": recipes} # If user defined a convert() function, return that try: exec(code, globals, locals) return locals[""convert""] except (AttributeError, SyntaxError, NameError, KeyError, TypeError): pass # Try compiling their code as a function instead body_variants = [code] # If single line and no 'return', try adding the return if ""\n"" not in code and not code.strip().startswith(""return ""): body_variants.insert(0, ""return {}"".format(code)) for variant in body_variants: new_code = [""def fn({}):"".format(variable)] for line in variant.split(""\n""): new_code.append("" {}"".format(line)) try: code_o = compile(""\n"".join(new_code), """", ""exec"") break except SyntaxError: # Try another variant, e.g. for 'return row[""column""] = 1' continue for import_ in imports: globals[import_.split(""."")[0]] = __import__(import_) exec(code_o, globals, locals) return locals[""fn""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008348032,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008348032,IC_kwDOCGYnMM48GiuA,9599,simonw,2022-01-09T18:14:02Z,2022-01-09T18:14:02Z,OWNER,Here's the code in question: https://github.com/simonw/sqlite-utils/blob/b8c134059e89f0fa040b84fb7d0bda25b9a52759/sqlite_utils/utils.py#L288-L299,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008347768,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008347768,IC_kwDOCGYnMM48Gip4,9599,simonw,2022-01-09T18:12:30Z,2022-01-09T18:12:30Z,OWNER,"Tried this test: ```python result = CliRunner().invoke( cli.cli, [ ""insert"", db_path, ""rows"", ""-"", ""--convert"", 'row[""is_chicken""] = True', ], input='{""name"": ""Azi""}', ) ``` And got this error: > `E + where 1 = ', 2, 30, ' return row[""is_chicken""] = True\n'))>.exit_code` The code snippet compilation isn't currently compatible with this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246366,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008246366,IC_kwDOCGYnMM48GJ5e,9599,simonw,2022-01-09T07:42:14Z,2022-01-09T07:42:14Z,OWNER,Also need to update relevant docs for that example.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246239,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008246239,IC_kwDOCGYnMM48GJ3f,9599,simonw,2022-01-09T07:41:24Z,2022-01-09T07:41:24Z,OWNER,"Might be a case of modifying this line: https://github.com/simonw/sqlite-utils/blob/e0c476bc380744680c8b7675c24fb0e9f5ec6dcd/sqlite_utils/cli.py#L828 To: ```python docs = (fn(doc) or doc for doc in docs) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-901452199,https://api.github.com/repos/simonw/sqlite-utils/issues/37,901452199,IC_kwDOCGYnMM41uxGn,9599,simonw,2021-08-18T21:48:57Z,2021-08-18T21:48:57Z,OWNER,"I did a bunch of work on this in #266. The library is now pretty thoroughly typed, and I even found a couple of bugs using `mypy` along the way: #313 and #315.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881190,https://api.github.com/repos/simonw/sqlite-utils/issues/37,868881190,MDEyOklzc3VlQ29tbWVudDg2ODg4MTE5MA==,9599,simonw,2021-06-25T23:24:28Z,2021-06-25T23:24:28Z,OWNER,Maybe I could release a separate Python package `types-sqlite-utils-numpy` which adds an over-ridden type definition that includes the `numpy` types?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881033,https://api.github.com/repos/simonw/sqlite-utils/issues/37,868881033,MDEyOklzc3VlQ29tbWVudDg2ODg4MTAzMw==,9599,simonw,2021-06-25T23:23:49Z,2021-06-25T23:23:49Z,OWNER,"Twitter conversation about how to add types to the `.create_table(columns=)` parameter: https://twitter.com/simonw/status/1408532867592818693 > Anyone know how to write a mypy type definition for this? > > {""id"": int, ""name"": str, ""image"": bytes, ""weight"": float} > > It's a dict where keys are strings and values are one of int/str/bytes/float (weird API design I know, but I designed this long before I was thinking about mypy) Looks like this could work: ```python def create_table( self, name, columns: Dict[str, Union[Type[int], Type[bytes], Type[str], Type[float]]], pk=None, foreign_keys=None, column_order=None, not_null=None, defaults=None, hash_id=None, extracts=None, ): ``` Except... that method can optionally also accept `numpy` types if `numpy` is installed. I don't know if it's possible to dynamically change a signature based on an import, since `mypy` is a static type analyzer and doesn't ever execute the code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-509686432,https://api.github.com/repos/simonw/sqlite-utils/issues/37,509686432,MDEyOklzc3VlQ29tbWVudDUwOTY4NjQzMg==,9599,simonw,2019-07-09T15:18:49Z,2019-07-09T15:18:49Z,OWNER,"Interestingly if I run that against my `monkeytype` branch I get the following errors (subset): ``` _______________________________________________________________________________________ mypy db.py ________________________________________________________________________________________ sqlite_utils/db.py:15: error: No library stub file for module 'numpy' sqlite_utils/db.py:15: note: (Stub files are from https://github.com/python/typeshed) sqlite_utils/db.py:197: error: Incompatible types in assignment (expression has type ""Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], Dict[, ]]"", variable has type ""Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]"") sqlite_utils/db.py:204: error: Item ""None"" of ""Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]"" has no attribute ""__iter__"" (not iterable) sqlite_utils/db.py:206: error: Argument 1 to ""set"" has incompatible type ""Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]""; expected ""Iterable[str]"" sqlite_utils/db.py:211: error: Item ""None"" of ""Optional[Tuple[str, str, str]]"" has no attribute ""index"" sqlite_utils/db.py:211: error: Unsupported right operand type for in (""Optional[Tuple[str, str, str]]"") sqlite_utils/db.py:232: error: Unsupported right operand type for in (""Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]"") sqlite_utils/db.py:234: error: Value of type ""Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]"" is not indexable sqlite_utils/db.py:307: error: Need type annotation for 'table_sql' (hint: ""table_sql: Dict[, ] = ..."") sqlite_utils/db.py:427: error: Item ""int"" of ""Union[int, None, str]"" has no attribute ""startswith"" sqlite_utils/db.py:427: error: Item ""None"" of ""Union[int, None, str]"" has no attribute ""startswith"" sqlite_utils/db.py:434: error: Incompatible types in assignment (expression has type ""List[Any]"", target has type ""int"") sqlite_utils/db.py:438: error: Incompatible types in assignment (expression has type ""object"", target has type ""int"") sqlite_utils/db.py:670: error: Need type annotation for 'all_column_types' (hint: ""all_column_types: Dict[, ] = ..."") sqlite_utils/db.py:757: error: Incompatible types in assignment (expression has type ""List[Any]"", variable has type ""chain[Any]"") sqlite_utils/db.py:761: error: Argument 1 to ""create"" of ""Table"" has incompatible type ""Dict[str, Union[Type[str], Type[int], Type[bool], Type[time], Type[float]]]""; expected ""Dict[str, Union[Type[str], Type[int], Type[bool], Type[time]]]"" sqlite_utils/db.py:770: error: Need type annotation for 'all_columns' (hint: ""all_columns: Set[] = ..."") sqlite_utils/db.py:773: error: Incompatible types in assignment (expression has type ""List[Any]"", variable has type ""Set[Any]"") sqlite_utils/db.py:775: error: ""Set[Any]"" has no attribute ""insert"" sqlite_utils/db.py:797: error: Need type annotation for 'values' (hint: ""values: List[] = ..."") sqlite_utils/db.py:811: error: Argument 1 to ""add_missing_columns"" of ""Table"" has incompatible type ""chain[Any]""; expected ""Union[List[Union[Dict[str, Union[int, str]], Dict[str, Union[int, str, List[str]]]]], List[Dict[str, Union[str, int, float]]], List[Dict[str, Union[int, str]]]]"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-509685610,https://api.github.com/repos/simonw/sqlite-utils/issues/37,509685610,MDEyOklzc3VlQ29tbWVudDUwOTY4NTYxMA==,9599,simonw,2019-07-09T15:16:52Z,2019-07-09T15:16:52Z,OWNER,"Also interesting: ``` pip install pytest-mypy pytest --mypy ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-509681590,https://api.github.com/repos/simonw/sqlite-utils/issues/37,509681590,MDEyOklzc3VlQ29tbWVudDUwOTY4MTU5MA==,9599,simonw,2019-07-09T15:07:12Z,2021-06-22T18:17:53Z,OWNER,"Here's a magic incantation for generating types detected through running the tests with https://github.com/Instagram/MonkeyType ``` pip install pytest-monkeytype pytest --monkeytype-output=./monkeytype.sqlite3 monkeytype list-modules monkeytype apply sqlite_utils.utils monkeytype apply sqlite_utils.cli monkeytype apply sqlite_utils.db ``` Here's the result: https://github.com/simonw/sqlite-utils/commit/d18c694fc25b7dd3d76e250c77ddf56d10ddf935","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1029402837,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1029402837,IC_kwDOCGYnMM49W3DV,9599,simonw,2022-02-03T21:07:35Z,2022-02-03T21:07:35Z,OWNER,"Closing this - it was something I was curious about, but evidently not curious enough to actually do the work!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008232075,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008232075,IC_kwDOCGYnMM48GGaL,9599,simonw,2022-01-09T05:13:15Z,2022-01-09T05:13:56Z,OWNER,"I think the query that will help solve this is: `explain query plan select * from ny_times_us_counties where state = 1 and county = 2` In this case, the query planner needs to decide if it should use the index for the `state` column or the index for the `county` column. That's where the statistics come into play. In particular: | tbl | idx | stat | |----------------------|---------------------------------|---------------| | ny_times_us_counties | idx_ny_times_us_counties_date | 2092871 2915 | | ny_times_us_counties | idx_ny_times_us_counties_fips | 2092871 651 | | ny_times_us_counties | idx_ny_times_us_counties_county | 2092871 1085 | | ny_times_us_counties | idx_ny_times_us_counties_state | 2092871 37373 | Those numbers are explained by this comment in the SQLite C code: https://github.com/sqlite/sqlite/blob/5622c7f97106314719740098cf0854e7eaa81802/src/analyze.c#L41-L55 ``` ** There is normally one row per index, with the index identified by the ** name in the idx column. The tbl column is the name of the table to ** which the index belongs. In each such row, the stat column will be ** a string consisting of a list of integers. The first integer in this ** list is the number of rows in the index. (This is the same as the ** number of rows in the table, except for partial indices.) The second ** integer is the average number of rows in the index that have the same ** value in the first column of the index. ``` So that table is telling us that using a value in the `county` column will filter down to an average of 1,085 rows, whereas filtering on the `state` column will filter down to an average of 37,373 - so clearly the `county` index is the better index to use here! Just one catch: against both my` covid.db` and my `covid-analyzed.db` databases the `county` index is picked for both of them - so SQLite is somehow guessing that `county` is a better index even though it doesn't have statistics for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008229341,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008229341,IC_kwDOCGYnMM48GFvd,9599,simonw,2022-01-09T04:45:38Z,2022-01-09T04:47:11Z,OWNER,"This is probably too fancy. I think maybe the way to do this is with `select * from [global-power-plants] where ""country_long"" = 'United Kingdom'` - then mess around with stats to see if I can get it to use the index or not based on them. Here's the explain for that: https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+*+from+[global-power-plants]+where+%22country_long%22+%3D+%27United+Kingdom%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008227625,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008227625,IC_kwDOCGYnMM48GFUp,9599,simonw,2022-01-09T04:25:38Z,2022-01-09T04:25:38Z,OWNER,"```sql EXPLAIN QUERY PLAN select country_long, count(*) from [global-power-plants] group by country_long ``` https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+country_long%2C+count%28*%29+from+%5Bglobal-power-plants%5D+group+by+country_long > SCAN TABLE global-power-plants USING COVERING INDEX ""global-power-plants_country_long""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226862,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008226862,IC_kwDOCGYnMM48GFIu,9599,simonw,2022-01-09T04:17:55Z,2022-01-09T04:17:55Z,OWNER,"There are some clues as to what effect ANALYZE has in https://www.sqlite.org/optoverview.html Some quotes: > SQLite might use a skip-scan on an index if it knows that the first one or more columns contain many duplication values. If there are too few duplicates in the left-most columns of the index, then it would be faster to simply step ahead to the next value, and thus do a full table scan, than to do a binary search on an index to locate the next left-column value. > > The only way that SQLite can know that there are many duplicates in the left-most columns of an index is if the ANALYZE command has been run on the database. Without the results of ANALYZE, SQLite has to guess at the ""shape"" of the data in the table, and the default guess is that there are an average of 10 duplicates for every value in the left-most column of the index. Skip-scan only becomes profitable (it only gets to be faster than a full table scan) when the number of duplicates is about 18 or more. Hence, a skip-scan is never used on a database that has not been analyzed. And > Join reordering is automatic and usually works well enough that programmers do not have to think about it, especially if ANALYZE has been used to gather statistics about the available indexes, though occasionally some hints from the programmer are needed. And > The various sqlite_statN tables contain information on how selective the various indexes are. For example, the sqlite_stat1 table might indicate that an equality constraint on column x reduces the search space to 10 rows on average, whereas an equality constraint on column y reduces the search space to 3 rows on average. In that case, SQLite would prefer to use index ex2i2 since that index is more selective. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226487,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008226487,IC_kwDOCGYnMM48GFC3,9599,simonw,2022-01-09T04:14:05Z,2022-01-09T04:14:05Z,OWNER,"Didn't manage to spot a meaningful difference with that database either: ``` analyze % python3 -m timeit '__import__(""sqlite3"").connect(""covid.db"").execute(""select fips, count(*) from [ny_times_us_counties] group by fips"").fetchall()' 2 loops, best of 5: 101 msec per loop analyze % python3 -m timeit '__import__(""sqlite3"").connect(""covid-analyzed.db"").execute(""select fips, count(*) from [ny_times_us_counties] group by fips"").fetchall()' 2 loops, best of 5: 103 msec per loop ``` Maybe `select fips, count(*) from [ny_times_us_counties] group by fips` isn't a good query for testing this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008220270,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008220270,IC_kwDOCGYnMM48GDhu,9599,simonw,2022-01-09T03:12:38Z,2022-01-09T03:13:15Z,OWNER,"Basically no difference using this very basic benchmark: ``` analyze % python3 -m timeit '__import__(""sqlite3"").connect(""global-power-plants.db"").execute(""select country_long, count(*) from [global-power-plants] group by country_long"").fetchall()' 100 loops, best of 5: 2.39 msec per loop analyze % python3 -m timeit '__import__(""sqlite3"").connect(""global-power-plants-analyzed.db"").execute(""select country_long, count(*) from [global-power-plants] group by country_long"").fetchall()' 100 loops, best of 5: 2.38 msec per loop ``` I should try this against a much larger database. https://covid-19.datasettes.com/covid.db is 879MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219844,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219844,IC_kwDOCGYnMM48GDbE,9599,simonw,2022-01-09T03:08:09Z,2022-01-09T03:08:09Z,OWNER,"``` analyze % sqlite-utils global-power-plants-analyzed.db 'analyze' [{""rows_affected"": -1}] analyze % sqlite-utils tables global-power-plants-analyzed.db [{""table"": ""global-power-plants""}, {""table"": ""global-power-plants_fts""}, {""table"": ""global-power-plants_fts_data""}, {""table"": ""global-power-plants_fts_idx""}, {""table"": ""global-power-plants_fts_docsize""}, {""table"": ""global-power-plants_fts_config""}, {""table"": ""sqlite_stat1""}] analyze % sqlite-utils rows global-power-plants-analyzed.db sqlite_stat1 -t tbl idx stat ------------------------------- ---------------------------------- --------- global-power-plants_fts_config global-power-plants_fts_config 1 1 global-power-plants_fts_docsize 33643 global-power-plants_fts_idx global-power-plants_fts_idx 199 40 1 global-power-plants_fts_data 136 global-power-plants ""global-power-plants_owner"" 33643 4 global-power-plants ""global-power-plants_country_long"" 33643 202 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219588,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219588,IC_kwDOCGYnMM48GDXE,9599,simonw,2022-01-09T03:06:42Z,2022-01-09T03:06:42Z,OWNER,"``` analyze % sqlite-utils indexes global-power-plants.db -t table index_name seqno cid name desc coll key ------------------------------ ------------------------------------------------- ------- ----- ------------ ------ ------ ----- global-power-plants ""global-power-plants_owner"" 0 12 owner 0 BINARY 1 global-power-plants ""global-power-plants_country_long"" 0 1 country_long 0 BINARY 1 global-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 0 0 segid 0 BINARY 1 global-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 1 1 term 0 BINARY 1 global-power-plants_fts_config sqlite_autoindex_global-power-plants_fts_config_1 0 0 k 0 BINARY 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219484,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219484,IC_kwDOCGYnMM48GDVc,9599,simonw,2022-01-09T03:05:44Z,2022-01-09T03:05:44Z,OWNER,I'll start by running some experiments against the 11MB database file from https://global-power-plants.datasettes.com/global-power-plants.db,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219191,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219191,IC_kwDOCGYnMM48GDQ3,9599,simonw,2022-01-09T03:03:53Z,2022-01-09T03:03:53Z,OWNER,"Refs: - #366 - #365","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216371,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008216371,IC_kwDOCGYnMM48GCkz,9599,simonw,2022-01-09T02:36:22Z,2022-01-09T02:36:22Z,OWNER,"In Python 3.6: https://docs.python.org/3.6/library/subprocess.html > This does not capture stdout or stderr by default. To do so, pass [`PIPE`](https://docs.python.org/3.6/library/subprocess.html#subprocess.PIPE ""subprocess.PIPE"") for the *stdout* and/or *stderr* arguments.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`, https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216271,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008216271,IC_kwDOCGYnMM48GCjP,9599,simonw,2022-01-09T02:35:09Z,2022-01-09T02:35:09Z,OWNER,"Test failure on Python 3.6: > `E TypeError: __init__() got an unexpected keyword argument 'capture_output'`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`, https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008215912,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008215912,IC_kwDOCGYnMM48GCdo,9599,simonw,2022-01-09T02:30:59Z,2022-01-09T02:30:59Z,OWNER,"Even better, inspired by `rich`, support `python -m sqlite_utils`. https://github.com/Textualize/rich/blob/master/rich/__main__.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009508865,IC_kwDOCGYnMM48K-IB,9599,simonw,2022-01-11T01:08:51Z,2022-01-11T01:08:51Z,OWNER,"The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009288898,IC_kwDOCGYnMM48KIbC,9599,simonw,2022-01-10T19:54:04Z,2022-01-10T19:54:04Z,OWNER,"Having browsed the API reference I think the methods that would benefit from an `analyze=True` parameter are: - `db.create_index` - `table.insert_all` - `table.upsert_all` - `table.delete_where`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009286373,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009286373,IC_kwDOCGYnMM48KHzl,9599,simonw,2022-01-10T19:50:22Z,2022-01-10T19:50:22Z,OWNER,"With respect to #365, I'm now thinking that having the ability to say ""... and then run ANALYZE"" could be useful for a bunch of Python methods. For example: ```python db[""dogs""].insert_all(list_of_dogs, analyze=True) db[""dogs""].create_index([""name""], analyze=True) ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009285627,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009285627,IC_kwDOCGYnMM48KHn7,9599,simonw,2022-01-10T19:49:19Z,2022-01-10T19:51:25Z,OWNER,Documentation for those two new methods: https://sqlite-utils.datasette.io/en/latest/python-api.html#optimizing-index-usage-with-analyze,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009273525,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009273525,IC_kwDOCGYnMM48KEq1,9599,simonw,2022-01-10T19:32:39Z,2022-01-10T19:32:39Z,OWNER,"I'm going to implement the Python library methods based on the prototype: ```diff commit 650f97a08f29a688c530e5f6c9eedc9269ed7bdc Author: Simon Willison Date: Sat Jan 8 13:34:01 2022 -0800 Initial prototype of .analyze(), refs #366 diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index dfc4723..1348b4a 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -923,6 +923,13 @@ class Database: ""Run a SQLite ``VACUUM`` against the database."" self.execute(""VACUUM;"") + def analyze(self, name=None): + ""Run ``ANALYZE`` against the entire database or a named table or index."" + sql = ""ANALYZE"" + if name is not None: + sql += "" [{}]"".format(name) + self.execute(sql) + class Queryable: def exists(self) -> bool: @@ -2902,6 +2909,10 @@ class Table(Queryable): ) return self + def analyze(self): + ""Run ANALYZE against this table"" + self.db.analyze(self.name) + def analyze_column( self, column: str, common_limit: int = 10, value_truncate=None, total_rows=None ) -> ""ColumnDetails"": ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008158616,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1008158616,IC_kwDOCGYnMM48F0eY,9599,simonw,2022-01-08T21:35:32Z,2022-01-08T21:35:32Z,OWNER,"Built a prototype in a branch, see #367.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1008157132,IC_kwDOCGYnMM48F0HM,9599,simonw,2022-01-08T21:23:08Z,2022-01-08T21:25:05Z,OWNER,"Running `ANALYZE` creates a new visible table called `sqlite_stat1`: https://www.sqlite.org/fileformat.html#the_sqlite_stat1_table This should be added to the default list of hidden tables in Datasette. It looks something like this: | tbl | idx | stat | |---------------------------------|------------------------------------|-----------| | _counts | sqlite_autoindex__counts_1 | 5 1 | | global-power-plants_fts_config | global-power-plants_fts_config | 1 1 | | global-power-plants_fts_docsize | | 33643 | | global-power-plants_fts_idx | global-power-plants_fts_idx | 199 40 1 | | global-power-plants_fts_data | | 136 | | global-power-plants | ""global-power-plants_owner"" | 33643 4 | | global-power-plants | ""global-power-plants_country_long"" | 33643 202 | > In each such row, the sqlite_stat.stat column will be a string consisting of a list of integers followed by zero or more arguments. The first integer in this list is the approximate number of rows in the index. (The number of rows in the index is the same as the number of rows in the table, except for partial indexes.) The second integer is the approximate number of rows in the index that have the same value in the first column of the index. The third integer is the number number of rows in the index that have the same value for the first two columns. The N-th integer (for N>1) is the estimated average number of rows in the index which have the same value for the first N-1 columns. For a K-column index, there will be K+1 integers in the stat column. If the index is unique, then the last integer will be 1. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007641634,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007641634,IC_kwDOCGYnMM48D2Qi,9599,simonw,2022-01-07T18:35:35Z,2022-01-07T18:35:35Z,OWNER,"Since the existing CLI feature is this: $ sqlite-utils analyze-tables github.db tags I can add `sqlite-utils analyze` to reflect the Python library method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007639860,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007639860,IC_kwDOCGYnMM48D100,9599,simonw,2022-01-07T18:32:59Z,2022-01-07T18:33:07Z,OWNER,"From the SQLite docs: > If no arguments are given, all attached databases are analyzed. If a schema name is given as the argument, then all tables and indices in that one database are analyzed. If the argument is a table name, then only that table and the indices associated with that table are analyzed. If the argument is an index name, then only that one index is analyzed. So I think this becomes two methods: - `db.analyze()` calls analyze on the whole database - `db.analyze(name_of_table_or_index)` for a specific named table or index - `table.analyze()` is a shortcut for `db.analyze(table.name)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007637963,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007637963,IC_kwDOCGYnMM48D1XL,9599,simonw,2022-01-07T18:30:13Z,2022-01-07T18:30:13Z,OWNER,"Annoyingly I use the word ""analyze"" to mean something else in the CLI - for these features: - #207 - #320 there's only one method with a similar name in the Python library though and that's this one: https://github.com/simonw/sqlite-utils/blob/6e46b9913411682f3a3ec66f4d58886c1db8654b/sqlite_utils/db.py#L2904-L2906","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009548580,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1009548580,IC_kwDOCGYnMM48LH0k,536941,fgregg,2022-01-11T02:43:34Z,2022-01-11T02:43:34Z,CONTRIBUTOR,thanks so much! always a pleasure to see how you work through these things,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009521921,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1009521921,IC_kwDOCGYnMM48LBUB,9599,simonw,2022-01-11T01:37:53Z,2022-01-11T01:37:53Z,OWNER,"I decided to go with making this opt-in, mainly for consistency with the other places where I added this feature - see: - #379 - #366 You can now run the following: sqlite-utils create-index mydb.db mytable mycolumn --analyze And ``ANALYZE`` will be run on the index once it has been created.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008275546,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008275546,IC_kwDOCGYnMM48GRBa,536941,fgregg,2022-01-09T11:01:15Z,2022-01-09T13:37:51Z,CONTRIBUTOR,"i don’t want to be such a partisan for analyze, but the query planner deciding *not* to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice. the original poster in that stack overflow doesn’t say there’s a performance regression ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008229839,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008229839,IC_kwDOCGYnMM48GF3P,9599,simonw,2022-01-09T04:51:44Z,2022-01-09T04:51:44Z,OWNER,"Found one report on Stack Overflow from 9 years ago of someone seeing broken performance after running `ANALYZE`, hard to say that's a trend and not a single weird edge-case though! https://stackoverflow.com/q/12947214/6083","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008166084,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008166084,IC_kwDOCGYnMM48F2TE,536941,fgregg,2022-01-08T22:32:47Z,2022-01-08T22:32:47Z,CONTRIBUTOR,or using “ pragma optimize”,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008164786,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008164786,IC_kwDOCGYnMM48F1-y,536941,fgregg,2022-01-08T22:24:19Z,2022-01-08T22:24:19Z,CONTRIBUTOR,the out-of-date scenario you describe could be addressed by automatically adding an analyze to the insert or convert commands if they implicate an index,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008164116,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008164116,IC_kwDOCGYnMM48F10U,536941,fgregg,2022-01-08T22:18:57Z,2022-01-08T22:18:57Z,CONTRIBUTOR,"the table with the query ran so bad was about 50k. i think the scenario should not be worse than no stats. i also did not know that sqlite was so different from postgres and needed an explicit analyze call.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163585,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008163585,IC_kwDOCGYnMM48F1sB,9599,simonw,2022-01-08T22:14:39Z,2022-01-09T03:03:07Z,OWNER,"The reason I'm hesitating on this is that I've not actually used ANALYZE at all in nearly five years of messing around with SQLite! So I'm nervous that there are surprise downsides I haven't thought of. My hunch is that ANALYZE is only worth worrying about on much larger databases, in which case I'm OK supporting it as a thoroughly documented power-user feature rather than a default.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163050,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008163050,IC_kwDOCGYnMM48F1jq,9599,simonw,2022-01-08T22:10:51Z,2022-01-08T22:10:51Z,OWNER,"Is there a downside to having a `sqlite_stat1` table if it has wildly incorrect statistics in it? Imagine the following sequence of events: - User imports a few records, creating the table, using `sqlite-utils insert` - User runs `sqlite-utils create-index ...` which also creates and populates the `sqlite_stat1` table - User runs `insert` again to populate several million new records The user now has a database file with several million records and a statistics table that is wildly out of date, having been populated when they only had a few. Will this result in surprisingly bad query performance compared to it that statistics table did not exist at all? If so, I lean much harder towards `ANALYZE` as a strictly opt-in optimization, maybe with the `--analyze` option added to `sqlite-utils insert` top to help users opt in to updating their statistics after running big inserts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008161965,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008161965,IC_kwDOCGYnMM48F1St,536941,fgregg,2022-01-08T22:02:56Z,2022-01-08T22:02:56Z,CONTRIBUTOR,"for options 2 and 3, i would worry about discoverablity. in other db’s it is not necessary to explicitly call analyze for most indices. ie for postgres > The system regularly collects statistics on all of a table's columns. Newly-created non-expression indexes can immediately use these statistics to determine an index's usefulness. i suppose i would propose raising a warning if the stats table is created that explains what is going on and informs users about a —no-analyze argument.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index,