{"html_url": "https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394619", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/96", "id": 606394619, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjM5NDYxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T04:38:17Z", "updated_at": "2020-03-31T04:40:23Z", "author_association": "OWNER", "body": "I wonder if there are any other Pandas conversions we should be doing? https://pandas.pydata.org/pandas-docs/stable/getting_started/basics.html#dtypes ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 589801352, "label": "Add type conversion for Panda's Timestamp"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394349", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/96", "id": 606394349, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjM5NDM0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T04:37:16Z", "updated_at": "2020-03-31T04:37:16Z", "author_association": "OWNER", "body": "Test failure was just a Black formatting issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 589801352, "label": "Add type conversion for Panda's Timestamp"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580584269", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/80", "id": 580584269, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDU4NDI2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T05:08:04Z", "updated_at": "2020-01-31T05:08:04Z", "author_association": "OWNER", "body": "Ditching this since it won't actually solve my problem.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557892819, "label": "on_create mechanism for after table creation"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580569059", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/80", "id": 580569059, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDU2OTA1OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T03:48:41Z", "updated_at": "2020-01-31T03:48:41Z", "author_association": "OWNER", "body": "This may not be the right feature after all, see https://github.com/simonw/geojson-to-sqlite/issues/6#issuecomment-580569002", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557892819, "label": "on_create mechanism for after table creation"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567886", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/80", "id": 580567886, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDU2Nzg4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T03:41:31Z", "updated_at": "2020-01-31T03:41:31Z", "author_association": "OWNER", "body": "I think it does make sense to be able to pass it to the `.table()` constructor.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557892819, "label": "on_create mechanism for after table creation"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567604", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/80", "id": 580567604, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDU2NzYwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T03:39:58Z", "updated_at": "2020-01-31T03:39:58Z", "author_association": "OWNER", "body": "Perhaps this should be called `after_create` instead of `on_create`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557892819, "label": "on_create mechanism for after table creation"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567505", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/80", "id": 580567505, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDU2NzUwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T03:39:19Z", "updated_at": "2020-01-31T03:39:19Z", "author_association": "OWNER", "body": "Still needs documentation and tests. Also I'm not certain that this should be an argument you can pass to the `.table()` constructor, need to think that over.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 557892819, "label": "on_create mechanism for after table creation"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/75#issuecomment-580523995", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/75", "id": 580523995, "node_id": "MDEyOklzc3VlQ29tbWVudDU4MDUyMzk5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-01-31T00:21:11Z", "updated_at": "2020-01-31T00:21:11Z", "author_association": "OWNER", "body": "This makes sense, thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 546078359, "label": "Explicitly include tests and docs in sdist"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/67#issuecomment-569844320", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/67", "id": 569844320, "node_id": "MDEyOklzc3VlQ29tbWVudDU2OTg0NDMyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-12-31T01:29:43Z", "updated_at": "2019-12-31T01:29:43Z", "author_association": "OWNER", "body": "I don't really care about 3.5 any more.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 529376481, "label": "Run tests against 3.5 too"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/67#issuecomment-559108591", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/67", "id": 559108591, "node_id": "MDEyOklzc3VlQ29tbWVudDU1OTEwODU5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-27T14:24:59Z", "updated_at": "2019-11-27T14:24:59Z", "author_association": "OWNER", "body": "Failed due to black testing dependency: https://travis-ci.com/simonw/sqlite-utils/jobs/260995814", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 529376481, "label": "Run tests against 3.5 too"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527266798", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/57", "id": 527266798, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzI2Njc5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-03T01:03:59Z", "updated_at": "2019-09-03T01:03:59Z", "author_association": "OWNER", "body": "Released in 1.11 - thanks again! https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-11", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487987958, "label": "Add triggers while enabling FTS"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527258212", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/57", "id": 527258212, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzI1ODIxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-02T23:40:56Z", "updated_at": "2019-09-02T23:40:56Z", "author_association": "OWNER", "body": "This is fantastic, thanks so much.\r\n\r\nI spotted a Datasette bug triggered by this: Datasette has its own version of the `detect_fts` function - at https://github.com/simonw/datasette/blob/d224ee2c98ac39c2c6e21a0ac0c62e5c3e1ccd11/datasette/utils/__init__.py#L466-L479 - which fails to pick up FTS tables created using the new escaping pattern.\r\n\r\nIt's a bug in Datasette, not sqlite-utils - so I'll open an issue there.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487987958, "label": "Add triggers while enabling FTS"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527091278", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/57", "id": 527091278, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzA5MTI3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-02T10:13:32Z", "updated_at": "2019-09-02T10:13:32Z", "author_association": "OWNER", "body": "This is brilliant!\r\n\r\nOne tiny suggestion: I like `--create-triggers` and `create_triggers=True` better for this as they are shorter but still capture what it dies - especially since the underlying SQL uses `CREATE TRIGGER` statements.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487987958, "label": "Add triggers while enabling FTS"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606315321", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606315321, "node_id": "IC_kwDOCGYnMM5fvm05", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T23:18:33Z", "updated_at": "2023-06-25T23:18:33Z", "author_association": "OWNER", "body": "Documentation preview: https://sqlite-utils--560.org.readthedocs.build/en/560/installation.html#alternatives-to-sqlite3", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606310630", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606310630, "node_id": "IC_kwDOCGYnMM5fvlrm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T23:06:07Z", "updated_at": "2023-06-25T23:06:07Z", "author_association": "OWNER", "body": "Filed an issue about the above with `pysqlite3` (which `sqlean.py` is based on) here:\r\n- https://github.com/coleifer/pysqlite3/issues/58", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606297356", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606297356, "node_id": "IC_kwDOCGYnMM5fvicM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T22:42:41Z", "updated_at": "2023-06-25T22:42:41Z", "author_association": "OWNER", "body": "Yes that does seem to do the trick:\r\n```pycon\r\n>>> import sqlean\r\n>>> db = sqlean.connect(\"/tmp/4.db\")\r\n>>> db.execute('PRAGMA journal_mode;').fetchall()\r\n[('delete',)]\r\n>>> db.isolation_level\r\n''\r\n>>> db.execute('PRAGMA journal_mode=wal;')\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\nsqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction\r\n>>> db.isolation_level = None\r\n>>> db.isolation_level\r\n>>> db.execute('PRAGMA journal_mode=wal;')\r\n\r\n```\r\nWeird how `isolation_level` of empty string causes the error, but setting that to `None` fixes the error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606294627", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606294627, "node_id": "IC_kwDOCGYnMM5fvhxj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T22:40:10Z", "updated_at": "2023-06-25T22:40:10Z", "author_association": "OWNER", "body": "I suspect this has something to do with `autocommit` mode in `sqlite3` - which I may be able to turn off by setting `con.isolation_level = None`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606293382", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606293382, "node_id": "IC_kwDOCGYnMM5fvheG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T22:34:47Z", "updated_at": "2023-06-25T22:34:47Z", "author_association": "OWNER", "body": "```pycon\r\n>>> import sqlite3\r\n>>> db = sqlite3.connect(\"/tmp/1.db\")\r\n>>> db.execute('PRAGMA journal_mode=wal;')\r\n\r\n>>> import sqlean\r\n>>> db2 = sqlean.connect(\"/tmp/2.db\")\r\n>>> db2.execute('PRAGMA journal_mode=wal;')\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\nsqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606290917", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606290917, "node_id": "IC_kwDOCGYnMM5fvg3l", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T22:32:28Z", "updated_at": "2023-06-25T22:32:28Z", "author_association": "OWNER", "body": "I've fixed most of the test failures, but I still need to fix this one:\r\n\r\n> cannot change into wal mode from within a transaction", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606273005", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606273005, "node_id": "IC_kwDOCGYnMM5fvcft", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T21:47:47Z", "updated_at": "2023-06-25T21:47:47Z", "author_association": "OWNER", "body": "I can use https://github.com/simonw/sqlite-dump as an optional dependency to handle the missing `.iterdump()` method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270887", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606270887, "node_id": "IC_kwDOCGYnMM5fvb-n", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T21:37:12Z", "updated_at": "2023-06-26T08:21:00Z", "author_association": "OWNER", "body": "On my own laptop I got a crash running the tests - details here:\r\n\r\n- https://github.com/nalgeon/sqlean.py/issues/3", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270055", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/560", "id": 1606270055, "node_id": "IC_kwDOCGYnMM5fvbxn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-25T21:31:56Z", "updated_at": "2023-06-25T21:31:56Z", "author_association": "OWNER", "body": "Lots of failures now that I'm trying to run the tests against `sqlean.py` on macOS and Python 3.10: https://github.com/simonw/sqlite-utils/actions/runs/5371800108/jobs/9744802953\r\n\r\nA bunch of these, because `pysqlite3` chooses not to implement `.iterdump()`:\r\n```\r\n @pytest.fixture\r\n def db_to_analyze_path(db_to_analyze, tmpdir):\r\n path = str(tmpdir / \"test.db\")\r\n db = sqlite3.connect(path)\r\n> db.executescript(\"\\n\".join(db_to_analyze.conn.iterdump()))\r\nE AttributeError: 'sqlean.dbapi2.Connection' object has no attribute 'iterdump'\r\n```\r\nAlso some of these:\r\n```\r\n def test_analyze_whole_database(db):\r\n assert set(db.table_names()) == {\"one_index\", \"two_indexes\"}\r\n db.analyze()\r\n> assert set(db.table_names()) == {\"one_index\", \"two_indexes\", \"sqlite_stat1\"}\r\nE AssertionError: assert {'one_index',...'two_indexes'} == {'one_index',...'two_indexes'}\r\nE Extra items in the left set:\r\nE 'sqlite_stat4'\r\nE Full diff:\r\nE - {'two_indexes', 'sqlite_stat1', 'one_index'}\r\nE + {'two_indexes', 'sqlite_stat1', 'sqlite_stat4', 'one_index'}\r\nE ? ++++++++++++++++\r\n```\r\nApparently `sqlean.py` adds a `sqlite_stat4` table that the tests are not expecting.\r\n\r\nPlus some errors that look like this:\r\n```\r\n def test_enable_wal():\r\n runner = CliRunner()\r\n dbs = [\"test.db\", \"test2.db\"]\r\n with runner.isolated_filesystem():\r\n for dbname in dbs:\r\n db = Database(dbname)\r\n db[\"t\"].create({\"pk\": int}, pk=\"pk\")\r\n assert db.journal_mode == \"delete\"\r\n result = runner.invoke(cli.cli, [\"enable-wal\"] + dbs)\r\n> assert 0 == result.exit_code\r\nE AssertionError: assert 0 == 1\r\nE + where 1 = .exit_code\r\n```\r\nTest summary:\r\n```\r\n============ 13 failed, 909 passed, 16 skipped, 2 errors in 19.29s =============\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1773458985, "label": "Use sqlean if available in environment"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/56#issuecomment-527090411", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/56", "id": 527090411, "node_id": "MDEyOklzc3VlQ29tbWVudDUyNzA5MDQxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-09-02T10:10:48Z", "updated_at": "2019-09-02T10:10:48Z", "author_association": "OWNER", "body": "Good spot, thanks!\r\n\r\nWould be useful to add a test as well, derived from your above example.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 487847945, "label": "Escape the table name in populate_fts and search."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556288300", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/553", "id": 1556288300, "node_id": "IC_kwDOCGYnMM5cwxMs", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T20:48:01Z", "updated_at": "2023-05-21T20:48:01Z", "author_association": "OWNER", "body": "If https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries looks good I can merge this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718635018, "label": "Reformatted CLI examples in docs"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/55#issuecomment-522238063", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/55", "id": 522238063, "node_id": "MDEyOklzc3VlQ29tbWVudDUyMjIzODA2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-08-17T13:42:40Z", "updated_at": "2019-08-17T13:42:40Z", "author_association": "OWNER", "body": "I still need to implement the part of this where certain actions against views (like updating/inserting) are not allowed.\r\n\r\nI think the better way to do that is to introduce a new class - `View` - rather than reusing the existing `Table` class - since there's a lot of stuff on Table (like primary key / foreign key introspection) which simply doesn't make sense for views.\r\n\r\nThis will get a bit more complicated - maybe I need both Table and View to extend a common subclass here?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 481887482, "label": "Ability to introspect and run queries against views"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556242262", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/549", "id": 1556242262, "node_id": "IC_kwDOCGYnMM5cwl9W", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T18:00:05Z", "updated_at": "2023-05-21T18:00:05Z", "author_association": "OWNER", "body": "Failing `mypy` test: https://github.com/simonw/sqlite-utils/actions/runs/5038983349/jobs/9036828465\r\n```\r\nsqlite_utils/cli.py:37: error: Skipping analyzing \"trogon\": module is installed, but missing library stubs or py.typed marker [import]\r\nsqlite_utils/cli.py:37: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports\r\nFound 1 error in 1 file (checked 52 source files)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718586377, "label": "TUI powered by Trogon"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556241812", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/549", "id": 1556241812, "node_id": "IC_kwDOCGYnMM5cwl2U", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T17:58:25Z", "updated_at": "2023-05-21T17:58:25Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718586377, "label": "TUI powered by Trogon"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213396", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/546", "id": 1556213396, "node_id": "IC_kwDOCGYnMM5cwe6U", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-21T15:58:12Z", "updated_at": "2023-05-21T16:18:46Z", "author_association": "OWNER", "body": "Documentation preview:\r\n- https://sqlite-utils--546.org.readthedocs.build/en/546/cli.html#cli-analyze-tables\r\n- https://sqlite-utils--546.org.readthedocs.build/en/546/cli-reference.html#analyze-tables\r\n- https://sqlite-utils--546.org.readthedocs.build/en/546/python-api.html#analyzing-a-column\r\n- https://sqlite-utils--546.org.readthedocs.build/en/546/reference.html#sqlite_utils.db.Table.analyze_column", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1718550688, "label": "Analyze tables options: --common-limit, --no-most, --no-least"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539157643", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/537", "id": 1539157643, "node_id": "IC_kwDOCGYnMM5bva6L", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T22:45:09Z", "updated_at": "2023-05-08T22:45:21Z", "author_association": "OWNER", "body": "Here's an example from the new tests:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/a75abeb61b91a28650d3b9933e7ec80ad0d92529/tests/test_create.py#L291-L307", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1665200812, "label": "Support self-referencing FKs in `Table.create`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539055393", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/537", "id": 1539055393, "node_id": "IC_kwDOCGYnMM5bvB8h", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T21:10:06Z", "updated_at": "2023-05-08T21:10:06Z", "author_association": "OWNER", "body": "Thanks!", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1665200812, "label": "Support self-referencing FKs in `Table.create`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465302343", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/531", "id": 1465302343, "node_id": "IC_kwDOCGYnMM5XVr1H", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-03-12T21:19:13Z", "updated_at": "2023-03-12T21:19:13Z", "author_association": "OWNER", "body": "Aah, I think I see why you wrote it like that.\r\n\r\nThe problem is that `init_spatialite()` does other stuff too:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L1161-L1171\r\n\r\nSo it needs to be able to load the SpatiaLite extension from the correct place, and THEN run `select InitSpatialMetadata()` to configure the database, if needed.\r\n\r\nSo the problem you're trying to solve here is to let people optionally pass in the path to SpatiaLite if it's not one of the ones that are searched by default.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1620164673, "label": "Add paths for homebrew on Apple silicon"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/528#issuecomment-1539053230", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/528", "id": 1539053230, "node_id": "IC_kwDOCGYnMM5bvBau", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T21:08:23Z", "updated_at": "2023-05-08T21:08:23Z", "author_association": "OWNER", "body": "I fixed this in:\r\n- #527\r\n\r\nWill fully remove this misfeature in:\r\n- #542", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1578793661, "label": "Enable `Table.convert()` on falsey values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/519#issuecomment-1539058795", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/519", "id": 1539058795, "node_id": "IC_kwDOCGYnMM5bvCxr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T21:12:52Z", "updated_at": "2023-05-08T21:12:52Z", "author_association": "OWNER", "body": "This is a really neat fix, thank you.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1505568103, "label": "Fixes breaking DEFAULT values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/515#issuecomment-1539077777", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/515", "id": 1539077777, "node_id": "IC_kwDOCGYnMM5bvHaR", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T21:27:10Z", "updated_at": "2023-05-08T21:27:10Z", "author_association": "OWNER", "body": "I should have spotted this PR before I shipped my own fix! https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1465194930, "label": "upsert new rows with constraints, fixes #514"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/51#issuecomment-515756563", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/51", "id": 515756563, "node_id": "MDEyOklzc3VlQ29tbWVudDUxNTc1NjU2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-28T11:56:10Z", "updated_at": "2019-07-28T11:56:10Z", "author_association": "OWNER", "body": "Interesting. The tests failed presumably because the version of SQLite I am running on Travis CI doesn't have that same 999 limit. I'm going to enforce the 999 limit within the library itself, to discourage people from writing code which will fail if it runs on a host that DOES have that limit.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 473733752, "label": "Fix for too many SQL variables, closes #50"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248593835", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248593835, "node_id": "IC_kwDOCGYnMM5KbAer", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:37:14Z", "updated_at": "2022-09-15T20:37:14Z", "author_association": "OWNER", "body": "I'm going to land this anyway. The lack of a streaming JSON parser is a separate issue, I don't think it should block landing this improvement.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248591268", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248591268, "node_id": "IC_kwDOCGYnMM5Ka_2k", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:36:02Z", "updated_at": "2022-09-15T20:40:03Z", "author_association": "OWNER", "body": "I had a big CSV file lying around, I converted it to other formats like this:\r\n\r\n sqlite-utils insert /tmp/t.db t /tmp/en.openfoodfacts.org.products.csv --csv\r\n sqlite-utils rows /tmp/t.db t --nl > /tmp/big.nl\r\n sqlite-utils rows /tmp/t.db t > /tmp/big.json\r\n\r\nThen tested the progress bar like this:\r\n\r\n sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl\r\n\r\nOutput:\r\n\r\n```\r\nsqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl\r\n [------------------------------------] 0%\r\n [#######-----------------------------] 20% 00:00:20\r\n```\r\nWith `--silent` it is silent.\r\n\r\nAnd for regular JSON:\r\n\r\n```\r\nsqlite-utils insert /tmp/t3.db t /tmp/big.json \r\n [####################################] 100%\r\n```\r\nThis is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248582147", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248582147, "node_id": "IC_kwDOCGYnMM5Ka9oD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:29:17Z", "updated_at": "2022-09-15T20:29:17Z", "author_association": "OWNER", "body": "This looks good to me. I need to run some manual tests before merging (it's a good sign that the automated tests pass though).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248568775", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248568775, "node_id": "IC_kwDOCGYnMM5Ka6XH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:16:14Z", "updated_at": "2022-09-15T20:16:14Z", "author_association": "OWNER", "body": "https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#using-the-python-version-input says can set the full version:\r\n\r\n```\r\n- uses: actions/setup-python@v4\r\n with:\r\n python-version: \"3.10.6\" \r\n```\r\nI'll try that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248567323", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248567323, "node_id": "IC_kwDOCGYnMM5Ka6Ab", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:14:45Z", "updated_at": "2022-09-15T20:14:45Z", "author_association": "OWNER", "body": "There's a fix for `mypy` that has landed but isn't out in a release yet:\r\n- https://github.com/python/mypy/issues/13385\r\n\r\nFor the moment looks like pinning to Python 3.10.6 could help. Need to figure out how to do that in GitHub Actions though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248565396", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248565396, "node_id": "IC_kwDOCGYnMM5Ka5iU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:12:50Z", "updated_at": "2022-09-15T20:12:50Z", "author_association": "OWNER", "body": "Annoying `mypy` test failure:\r\n\r\n```\r\n/Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/site-packages/numpy/__init__.pyi:636:\r\nerror: Positional-only parameters are only supported in Python 3.8 and greater\r\n```\r\nLooks like this:\r\n- https://github.com/python/mypy/issues/13627", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232419522", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/480", "id": 1232419522, "node_id": "IC_kwDOCGYnMM5JdTrC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-31T03:33:27Z", "updated_at": "2022-08-31T03:33:27Z", "author_association": "OWNER", "body": "Tests look great, thank you!", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1355433619, "label": "search_sql add include_rank option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232089808", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/480", "id": 1232089808, "node_id": "IC_kwDOCGYnMM5JcDLQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-30T19:43:02Z", "updated_at": "2022-08-30T19:43:02Z", "author_association": "OWNER", "body": "Yeah this seems like a reasonable addition to me. \r\n\r\nNeeds a test, which can go next to this one: https://github.com/simonw/sqlite-utils/blob/087753cd42c406f1e060c1822dcd9b5fda3d60f4/tests/test_fts.py#L561", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1355433619, "label": "search_sql add include_rank option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229303691", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468", "id": 1229303691, "node_id": "IC_kwDOCGYnMM5JRa-L", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T23:13:06Z", "updated_at": "2022-08-27T23:13:06Z", "author_association": "OWNER", "body": "Documentation:\r\n- https://sqlite-utils--468.org.readthedocs.build/en/468/python-api.html#explicitly-creating-a-table\r\n- https://sqlite-utils--468.org.readthedocs.build/en/468/cli.html#creating-tables", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1348294436, "label": "db[table].create(..., transform=True) and create-table --transform"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229284539", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468", "id": 1229284539, "node_id": "IC_kwDOCGYnMM5JRWS7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T22:54:07Z", "updated_at": "2022-08-27T22:55:01Z", "author_association": "OWNER", "body": "Just needs documentation now, which can go here: https://sqlite-utils--468.org.readthedocs.build/en/468/python-api.html#explicitly-creating-a-table", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1348294436, "label": "db[table].create(..., transform=True) and create-table --transform"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229279539", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468", "id": 1229279539, "node_id": "IC_kwDOCGYnMM5JRVEz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T22:24:33Z", "updated_at": "2022-08-27T22:24:33Z", "author_association": "OWNER", "body": "Interesting challenge with `default_value`: I need to be able to tell if the default values passed to `.create()` differ from those in the database already.\r\n\r\nIntrospecting that is a bit tricky:\r\n\r\n```pycon\r\n>>> import sqlite_utils\r\n>>> db = sqlite_utils.Database(memory=True)\r\n>>> db[\"blah\"].create({\"id\": int, \"name\": str}, not_null=(\"name\",), defaults={\"name\": \"bob\"})\r\n\r\n>>> db[\"blah\"].columns\r\n[Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value=\"'bob'\", is_pk=0)]\r\n```\r\nNote how a default value of the Python string `bob` is represented in the results of `PRAGMA table_info()` as `default_value=\"'bob'\"` - it's got single quotes added to it!\r\n\r\nSo comparing default values from introspecting the database needs me to first parse that syntax. This may require a new table introspection method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1348294436, "label": "db[table].create(..., transform=True) and create-table --transform"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229276554", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468", "id": 1229276554, "node_id": "IC_kwDOCGYnMM5JRUWK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T22:05:40Z", "updated_at": "2022-08-27T22:05:51Z", "author_association": "OWNER", "body": "For the moment I'm not going to pay attention to `foreign_keys` changes - I will note that these are not modified in the documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1348294436, "label": "db[table].create(..., transform=True) and create-table --transform"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229207725", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468", "id": 1229207725, "node_id": "IC_kwDOCGYnMM5JRDit", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T14:55:32Z", "updated_at": "2022-08-27T22:03:50Z", "author_association": "OWNER", "body": "The main challenge here is coming up with comprehensive tests. The cases I need to cover are from this block of code:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/c5f8a2eb1a81a18b52825cc649112f71fe419b12/sqlite_utils/db.py#L1468-L1480", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1348294436, "label": "db[table].create(..., transform=True) and create-table --transform"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/465#issuecomment-1218663096", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/465", "id": 1218663096, "node_id": "IC_kwDOCGYnMM5Io1K4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-17T23:37:51Z", "updated_at": "2022-08-17T23:37:51Z", "author_association": "OWNER", "body": "Source links work on the preview: https://sqlite-utils--465.org.readthedocs.build/en/465/reference.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1342357149, "label": "beanbag-docutils>=2.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/463#issuecomment-1218610320", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/463", "id": 1218610320, "node_id": "IC_kwDOCGYnMM5IooSQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-17T23:11:07Z", "updated_at": "2022-08-17T23:11:07Z", "author_association": "OWNER", "body": "Thanks!", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1334416486, "label": "Use Read the Docs action v1"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/462#issuecomment-1203207043", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/462", "id": 1203207043, "node_id": "IC_kwDOCGYnMM5Ht3uD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-02T20:57:30Z", "updated_at": "2022-08-02T20:57:30Z", "author_association": "OWNER", "body": "I created a new link for this - https://discord.gg/Ass7bCAMDw - which should drop people straight into the `#sqlite-utils` channel in the Datasette Discord.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1326391841, "label": "Discord badge"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1220010289", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/460", "id": 1220010289, "node_id": "IC_kwDOCGYnMM5It-Ex", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-18T21:58:00Z", "updated_at": "2022-08-18T21:58:00Z", "author_association": "OWNER", "body": "I'm going to merge this as-is and add more links separately.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1326087800, "label": "Cross-link CLI to Python docs"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1203190312", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/460", "id": 1203190312, "node_id": "IC_kwDOCGYnMM5Htzoo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-02T20:36:58Z", "updated_at": "2022-08-02T20:36:58Z", "author_association": "OWNER", "body": "> That preview link it added didn't work, maybe because I have a custom domain setup?\r\n\r\nEntirely my fault, fixed here :https://github.com/simonw/sqlite-utils/commit/98a28cbfe6cea67f6334b42b74f35b0ddd309561", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1326087800, "label": "Cross-link CLI to Python docs"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1202947222", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/460", "id": 1202947222, "node_id": "IC_kwDOCGYnMM5Hs4SW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-02T16:20:47Z", "updated_at": "2022-08-02T16:20:47Z", "author_association": "OWNER", "body": "That preview link it added didn't work, maybe because I have a custom domain setup?\r\n\r\nIt linked to: https://readthedocs-preview--460.org.readthedocs.build/en/460/ - which 404s\r\n\r\nHere's the correct link: https://sqlite-utils--460.org.readthedocs.build/en/460/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1326087800, "label": "Cross-link CLI to Python docs"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/457#issuecomment-1197108865", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/457", "id": 1197108865, "node_id": "IC_kwDOCGYnMM5HWm6B", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-27T17:58:10Z", "updated_at": "2022-07-27T17:58:10Z", "author_association": "OWNER", "body": "Here's the build from the PR: https://sqlite-utils--457.org.readthedocs.build/en/457/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1319881016, "label": "Link to installation instructions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/455#issuecomment-1229205990", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/455", "id": 1229205990, "node_id": "IC_kwDOCGYnMM5JRDHm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T14:44:45Z", "updated_at": "2022-08-27T14:44:45Z", "author_association": "OWNER", "body": "I did not know `is` worked against strings and integers and booleans!\r\n\r\nhttps://latest.datasette.io/_memory?sql=select+1+is+1%2C+2.3+is+2.3%2C+true+is+true%2C+true+is+false%2C+%27dog%27+is+%27dog%27%2C+%27dog%27+is+%27cat%27%2C+null+is+null%2C+null+is+not+null\r\n\r\n```sql\r\nselect 1 is 1, 2.3 is 2.3, true is true, true is false, 'dog' is 'dog', 'dog' is 'cat', null is null, null is not null\r\n```\r\n\r\n| 1 is 1 | 2.3 is 2.3 | true is true | true is false | 'dog' is 'dog' | 'dog' is 'cat' | null is null | null is not null |\r\n|----------|--------------|----------------|-----------------|------------------|------------------|----------------|--------------------|\r\n| 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1309542173, "label": "in extract code, check equality with IS instead of = for nulls"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/455#issuecomment-1190686273", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/455", "id": 1190686273, "node_id": "IC_kwDOCGYnMM5G-G5B", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-07-20T19:46:15Z", "updated_at": "2022-07-20T19:46:15Z", "author_association": "OWNER", "body": "Can you add a new test for this? Something derived from the example in #423 would work great.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1309542173, "label": "in extract code, check equality with IS instead of = for nulls"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/437#issuecomment-1141488533", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/437", "id": 1141488533, "node_id": "IC_kwDOCGYnMM5ECbuV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-30T21:32:36Z", "updated_at": "2022-05-30T21:32:36Z", "author_association": "OWNER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1244294227, "label": "docs to dogs"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115197644", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/429", "id": 1115197644, "node_id": "IC_kwDOCGYnMM5CeJDM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T18:04:28Z", "updated_at": "2022-05-02T18:04:28Z", "author_association": "OWNER", "body": "I'm going to ship this straight away as `3.26.1`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223177069, "label": "Depend on click-default-group-wheel"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115196863", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/429", "id": 1115196863, "node_id": "IC_kwDOCGYnMM5CeI2_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T18:03:47Z", "updated_at": "2022-05-02T18:52:42Z", "author_association": "OWNER", "body": "I made a build of this branch and tested it like this: https://pyodide.org/en/stable/console.html\r\n\r\n```pycon\r\n>>> import micropip\r\n>>> await micropip.install(\"https://s3.amazonaws.com/simonwillison-cors-allowed-public/sqlite_utils-3.26-py3-none-any.whl\")\r\n>>> import sqlite_utils\r\n>>> db = sqlite_utils.Database(memory=True)\r\n>>> list(db.query(\"select 32443 + 55\"))\r\n[{'32443 + 55': 32498}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223177069, "label": "Depend on click-default-group-wheel"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/410#issuecomment-1055856441", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/410", "id": 1055856441, "node_id": "IC_kwDOCGYnMM4-7xc5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-01T21:05:21Z", "updated_at": "2022-03-01T21:05:21Z", "author_association": "OWNER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1149729902, "label": "Correct spelling mistakes (found with codespell)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040959312", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/407", "id": 1040959312, "node_id": "IC_kwDOCGYnMM4-C8dQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-16T00:58:32Z", "updated_at": "2022-02-16T00:58:32Z", "author_association": "OWNER", "body": "This is honestly one of the most complete PRs I've ever seen for a feature of this size. Thanks so much for this!", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1138948786, "label": "Add SpatiaLite helpers to CLI"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040598665", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/407", "id": 1040598665, "node_id": "IC_kwDOCGYnMM4-BkaJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-15T17:58:11Z", "updated_at": "2022-02-15T17:58:11Z", "author_association": "OWNER", "body": "Wow, just found out I can edit files in this PR branch by hitting `.` on my keyboard while looking at the PR, then making changes in the VS Code for web on `github.dev`!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1138948786, "label": "Add SpatiaLite helpers to CLI"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040596969", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/407", "id": 1040596969, "node_id": "IC_kwDOCGYnMM4-Bj_p", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-15T17:56:22Z", "updated_at": "2022-02-15T17:56:35Z", "author_association": "OWNER", "body": "We should add SpatiaLite to the action that calculates code coverage - that way we can calculate coverage across the new GIS tests as well: https://github.com/simonw/sqlite-utils/blob/main/.github/workflows/test-coverage.yml\r\n\r\nShould just be a case of adding this to that workflow - we can do this in the same PR.\r\n\r\n```\r\n - name: Install SpatiaLite\r\n run: sudo apt-get install libsqlite3-mod-spatialite\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1138948786, "label": "Add SpatiaLite helpers to CLI"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040595572", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/407", "id": 1040595572, "node_id": "IC_kwDOCGYnMM4-Bjp0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-15T17:54:58Z", "updated_at": "2022-02-15T17:54:58Z", "author_association": "OWNER", "body": "This PR looks fantastic.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1138948786, "label": "Add SpatiaLite helpers to CLI"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/40#issuecomment-511266664", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/40", "id": 511266664, "node_id": "MDEyOklzc3VlQ29tbWVudDUxMTI2NjY2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-15T03:47:26Z", "updated_at": "2019-07-15T03:47:26Z", "author_association": "OWNER", "body": "This will close #36 and #39 once it is finished.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 467928674, "label": ".get() method plus support for compound primary keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029703216", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/385", "id": 1029703216, "node_id": "IC_kwDOCGYnMM49YAYw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-04T06:45:43Z", "updated_at": "2022-02-04T06:45:43Z", "author_association": "OWNER", "body": "Shipped this as `sqlite-utils` 3.23: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-23", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1102899312, "label": "Add new spatialite helper methods"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029682294", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/385", "id": 1029682294, "node_id": "IC_kwDOCGYnMM49X7R2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-04T05:53:26Z", "updated_at": "2022-02-04T05:53:26Z", "author_association": "OWNER", "body": "This looks fantastic, thanks for all of the work you put into this!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1102899312, "label": "Add new spatialite helper methods"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029335225", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/385", "id": 1029335225, "node_id": "IC_kwDOCGYnMM49Wmi5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-03T19:39:40Z", "updated_at": "2022-02-03T19:39:40Z", "author_association": "OWNER", "body": "> I thought about adding these as methods on `Database` and `Table`, and I'm back and forth on it for the same reasons you are. It's certainly cleaner, and it's clearer what you're operating on. I could go either way.\r\n> \r\n> I do sort of like having all the Spatialite stuff in its own module, just because it's built around an extension you might not have or want, but I don't know if that's a good reason to have a different API.\r\n> \r\n> You could have `init_spatialite` add methods to `Database` and `Table`, so they're only there if you have Spatialite set up. Is that too clever? It feels too clever.\r\n\r\nYeah that's too clever. You know what? I'm pretty confident we are both massively over-thinking this. We should put the methods on `Database` and `Table`! API simplicity and consistency matters more than vague concerns about purity.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1102899312, "label": "Add new spatialite helper methods"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029297971", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/385", "id": 1029297971, "node_id": "IC_kwDOCGYnMM49Wdcz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-03T18:52:50Z", "updated_at": "2022-02-03T18:52:50Z", "author_association": "OWNER", "body": "I'm not sure I like `name=\"geometry\"` as the default argument to `add_geometry_column` - mainly because of this example here:\r\n```python\r\nadd_geometry_column(db[\"locations\"], \"POINT\")\r\ncreate_spatial_index(db[\"locations\"], \"geometry\")\r\n```\r\nI had to go and look at the code to figure out if `\"POINT\"` was the name of the column - and I don't like how inconsistent it looks next to the following `create_spatial_index()` call where you DO need to pass the column name.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1102899312, "label": "Add new spatialite helper methods"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029296782", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/385", "id": 1029296782, "node_id": "IC_kwDOCGYnMM49WdKO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-03T18:51:21Z", "updated_at": "2022-02-03T18:51:21Z", "author_association": "OWNER", "body": "What do you think about adding these as methods on the `Database` class instead? Then you could do:\r\n\r\n```python\r\n# This is with an optional argument, which if omitted runs find_spatialite() for you:\r\ndb.init_spatialite()\r\n\r\n# Instead of:\r\ninit_spatialite(db, find_spatialite())\r\n```\r\nLikewise, the `add_geometry_column` and `create_spatial_index` methods could live on `Table`:\r\n```python\r\n# Instead of this:\r\nadd_geometry_column(db[\"locations\"], \"POINT\", \"geometry\")\r\ncreate_spatial_index(db[\"locations\"], \"geometry\")\r\n\r\n# Could have this:\r\ndb[\"locations\"].add_geometry_column(\"POINT\")\r\ndb[\"locations\"].create_spatial_index(\"geometry\")\r\n```\r\nOn the one hand, this is much more consistent with the existing `sqlite-utils` Python API.\r\n\r\nBut on the other hand... this is mixing SpatiaLite functionality directly into the core classes. Is that a good idea, seeing as SpatiaLite is both an optional extension (which can be tricky to install) AND something that has a very different release cadence and quality-of-documentation from SQLite itself?\r\n\r\nThere's a third option: the SpatiaLite could exist on subclasses of `Database` and `Table` - so the above examples would look something like this:\r\n\r\n```python\r\nfrom sqlite_utils.gis import SpatiaLiteDatabase\r\n\r\ndb = SpatiaLiteDatabase(\"geo.db\")\r\ndb.init_spatialite()\r\ndb[\"locations\"].add_geometry_column(\"POINT\")\r\ndb[\"locations\"].create_spatial_index(\"geometry\")\r\n```\r\n\r\nOn the one hand, this would keep the SpatiaLite-specific stuff out of the core Database/Table classes. But it feels a bit untidy to me, especially since it raises the spectre of someone who was already subclassing Database for some reason now needing to instead subclass `SpatiaLiteDatabase` (not too keen on that capitalization) - or even (horror) trying to dabble with multiple inheritance, which can only lead to pain.\r\n\r\nSo I don't have a strong opinion formed on this question yet!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1102899312, "label": "Add new spatialite helper methods"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029285985", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/385", "id": 1029285985, "node_id": "IC_kwDOCGYnMM49Wahh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-03T18:37:48Z", "updated_at": "2022-02-03T18:37:48Z", "author_association": "OWNER", "body": "`from sqlite_utils.utils import find_spatialite` is part of the documented API already:\r\n\r\nhttps://sqlite-utils.datasette.io/en/3.22.1/python-api.html#finding-spatialite\r\n\r\nTo avoid needing to bump the major version number to 4 to indicate a backwards incompatible change, we should keep a `from .gis import find_spatialite` line at the top of `utils.py` such that any existing code with that documented import continues to work.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1102899312, "label": "Add new spatialite helper methods"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029273853", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/385", "id": 1029273853, "node_id": "IC_kwDOCGYnMM49WXj9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-03T18:23:30Z", "updated_at": "2022-02-03T18:31:21Z", "author_association": "OWNER", "body": "OK, this change makes a bunch of sense to me - and also raises some interesting questions about future additions to `sqlite-utils` with regards to SpatiaLite. Would `sqlite-utils add-geometry-column ...` be a good CLI enhancement. for example?\r\n\r\nI see you've already talked about that in #79 - moving this conversation there!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1102899312, "label": "Add new spatialite helper methods"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029177015", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/385", "id": 1029177015, "node_id": "IC_kwDOCGYnMM49V_63", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-02-03T16:38:02Z", "updated_at": "2022-02-03T16:38:02Z", "author_association": "OWNER", "body": "Sorry had missed this - tests should run now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1102899312, "label": "Add new spatialite helper methods"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/38#issuecomment-514339130", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/38", "id": 514339130, "node_id": "MDEyOklzc3VlQ29tbWVudDUxNDMzOTEzMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-23T18:57:33Z", "updated_at": "2019-07-23T18:57:33Z", "author_association": "OWNER", "body": "This should switch to using `.get()`, introduced in https://github.com/simonw/sqlite-utils/commit/c65b67ca46f70e2da46a5b945f4ed358173262e9", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 467862459, "label": "table.update() method"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/38#issuecomment-511219187", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/38", "id": 511219187, "node_id": "MDEyOklzc3VlQ29tbWVudDUxMTIxOTE4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-14T17:05:26Z", "updated_at": "2019-07-14T17:05:26Z", "author_association": "OWNER", "body": "I think I need to solve #36 (compound primary key support) first.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 467862459, "label": "table.update() method"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009534817", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/377", "id": 1009534817, "node_id": "IC_kwDOCGYnMM48LEdh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T02:09:38Z", "updated_at": "2022-01-11T02:09:38Z", "author_association": "OWNER", "body": "I tested this like so:\r\n```\r\n% wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'\r\n% sqlite-utils create-database test.db\r\n% sqlite-utils create-table test.db power_plants url text owner text\r\n% sqlite-utils schema test.db \r\nCREATE TABLE [power_plants] (\r\n [url] TEXT,\r\n [owner] TEXT\r\n);\r\n% sqlite-utils bulk test.db 'insert into power_plants (url, owner) values (:url, :owner)' global_power_plant_database.csv --csv\r\n [------------------------------------] 0%\r\n [###################################-] 99%\r\n% sqlite-utils tables --counts test.db -t\r\ntable count\r\n------------ -------\r\npower_plants 33643\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097477582, "label": "`sqlite-utils bulk` command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009532125", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/377", "id": 1009532125, "node_id": "IC_kwDOCGYnMM48LDzd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T02:03:35Z", "updated_at": "2022-01-11T02:03:35Z", "author_association": "OWNER", "body": "Documentation: https://github.com/simonw/sqlite-utils/blob/f4ea0d32c0543373eefaa9b9f3911eb07549eecb/docs/cli.rst#executing-sql-in-bulk", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097477582, "label": "`sqlite-utils bulk` command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1009272446", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/367", "id": 1009272446, "node_id": "IC_kwDOCGYnMM48KEZ-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-10T19:31:08Z", "updated_at": "2022-01-10T19:31:08Z", "author_association": "OWNER", "body": "I'm going to implement this in a separate commit from this PR.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097041471, "label": "Initial prototype of .analyze() methods"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006315145", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006315145, "node_id": "IC_kwDOCGYnMM47-yaJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T06:20:51Z", "updated_at": "2022-01-06T06:20:51Z", "author_association": "OWNER", "body": "This is all documented. I'm going to rebase-merge it to keep the individual commits.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006311742", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006311742, "node_id": "IC_kwDOCGYnMM47-xk-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T06:12:19Z", "updated_at": "2022-01-06T06:12:19Z", "author_association": "OWNER", "body": "Got that working:\r\n```\r\n% echo 'This is cool' | sqlite-utils insert words.db words - --text --convert '({\"word\": w} for w in text.split())'\r\n% sqlite-utils dump words.db \r\nBEGIN TRANSACTION;\r\nCREATE TABLE [words] (\r\n [word] TEXT\r\n);\r\nINSERT INTO \"words\" VALUES('This');\r\nINSERT INTO \"words\" VALUES('is');\r\nINSERT INTO \"words\" VALUES('cool');\r\nCOMMIT;\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006309834", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006309834, "node_id": "IC_kwDOCGYnMM47-xHK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T06:08:01Z", "updated_at": "2022-01-06T06:08:01Z", "author_association": "OWNER", "body": "For `--text` the conversion function should be allowed to return an iterable instead of a dictionary, in which case it will be treated as the full list of records to be inserted.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006301546", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006301546, "node_id": "IC_kwDOCGYnMM47-vFq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T05:44:47Z", "updated_at": "2022-01-06T05:44:47Z", "author_association": "OWNER", "body": "Just need documentation for `--convert` now against the various different types of input.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006300280", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006300280, "node_id": "IC_kwDOCGYnMM47-ux4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T05:40:45Z", "updated_at": "2022-01-06T05:40:45Z", "author_association": "OWNER", "body": "I'm going to rename `--all` to `--text`:\r\n\r\n> - Use `--text` to write the entire input to a column called \"text\"\r\n\r\nTo avoid that clash with Python's `all()` function.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006299778", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006299778, "node_id": "IC_kwDOCGYnMM47-uqC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T05:39:10Z", "updated_at": "2022-01-06T05:39:10Z", "author_association": "OWNER", "body": "`all` is a bad variable name because it clashes with the Python `all()` built-in function.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006295276", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006295276, "node_id": "IC_kwDOCGYnMM47-tjs", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T05:26:11Z", "updated_at": "2022-01-06T05:26:11Z", "author_association": "OWNER", "body": "Here's the traceback if your `--convert` function doesn't return a dict right now:\r\n```\r\n% sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all \r\n\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils\", line 33, in \r\n sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 949, in insert\r\n insert_upsert_implementation(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 834, in insert_upsert_implementation\r\n db[table].insert_all(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 2602, in insert_all\r\n first_record = next(records)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 3044, in fix_square_braces\r\n for record in records:\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 831, in \r\n docs = (decode_base64_values(doc) for doc in docs)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 86, in decode_base64_values\r\n to_fix = [\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 89, in \r\n if isinstance(doc[k], dict)\r\nTypeError: string indices must be integers\r\n```\r\nI can live with that for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006294777", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006294777, "node_id": "IC_kwDOCGYnMM47-tb5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T05:24:54Z", "updated_at": "2022-01-06T05:24:54Z", "author_association": "OWNER", "body": "> I added a custom error message for if the user's `--convert` code doesn't return a dict.\r\n\r\nThat turned out to be a bad idea because it meant exhausting the iterator early for the check - before we got to the `.insert_all()` code that breaks the iterator up into chunks. I tried fixing that with `itertools.tee()` to run the generator twice but that's grossly memory-inefficient for large imports.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006288444", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006288444, "node_id": "IC_kwDOCGYnMM47-r48", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T05:07:10Z", "updated_at": "2022-01-06T05:07:10Z", "author_association": "OWNER", "body": "And here's a demo of `--convert` used with `--all` - I added a custom error message for if the user's `--convert` code doesn't return a dict.\r\n\r\n```\r\n% sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all \r\nError: Records returned by your --convert function must be dicts\r\n% sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '{\"all\": all.upper()}' --all\r\n% sqlite-utils dump /tmp/all.db \r\nBEGIN TRANSACTION;\r\nCREATE TABLE [blah] (\r\n [all] TEXT\r\n);\r\nINSERT INTO \"blah\" VALUES('INFO: 127.0.0.1:60581 - \"GET / HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:60581 - \"GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:60581 - \"GET /FAVICON.ICO HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:60581 - \"GET /FOO/TIDDLYWIKI HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:60581 - \"GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:60584 - \"GET /FOO/-/STATIC/SQL-FORMATTER-2.3.3.MIN.JS HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:60586 - \"GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.JS HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:60585 - \"GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.CSS HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:60588 - \"GET /FOO/-/STATIC/CODEMIRROR-5.57.0-SQL.MIN.JS HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:60587 - \"GET /FOO/-/STATIC/CM-RESIZE-1.0.1.MIN.JS HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:60586 - \"GET /FOO/TIDDLYWIKI/TIDDLERS HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:60586 - \"GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1\" 200 OK\r\nINFO: 127.0.0.1:60584 - \"GET /FOO/-/STATIC/TABLE.JS HTTP/1.1\" 200 OK\r\n');\r\nCOMMIT;\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006284673", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006284673, "node_id": "IC_kwDOCGYnMM47-q-B", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T04:55:52Z", "updated_at": "2022-01-06T04:55:52Z", "author_association": "OWNER", "body": "Test code that just worked for me:\r\n```\r\nsqlite-utils insert /tmp/blah.db blah /tmp/log.log --convert '\r\nbits = line.split()\r\nreturn dict([(\"b_{}\".format(i), bit) for i, bit in enumerate(bits)])' --lines\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006232013", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006232013, "node_id": "IC_kwDOCGYnMM47-eHN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T02:21:35Z", "updated_at": "2022-01-06T02:21:35Z", "author_association": "OWNER", "body": "I'm having second thoughts about this bit:\r\n\r\n> Your Python code will be passed a \"row\" variable representing the imported row, and can return a modified row.\r\n>\r\n> If you are using `--lines` your code will be passed a \"line\" variable, and for `--all` an \"all\" variable.\r\n\r\nThe code in question is this:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/500a35ad4d91c8a6232134ce9406efec11bedff8/sqlite_utils/utils.py#L296-L303\r\n\r\nDo I really want to add the complexity of supporting different variable names there? I think always using `value` might be better.\r\n\r\nExcept... `value` made sense for the existing `sqlite-utils convert` command where you are running a conversion function against the value for the column in the current row - is it confusing if applied to lines or documents or `all`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006230411", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006230411, "node_id": "IC_kwDOCGYnMM47-duL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T02:17:35Z", "updated_at": "2022-01-06T02:17:35Z", "author_association": "OWNER", "body": "Documentation: https://github.com/simonw/sqlite-utils/blob/33223856ff7fe746b7b77750fbe5b218531d0545/docs/cli.rst#inserting-unstructured-data-with---lines-and---all - I went with a single section titled \"Inserting unstructured data with --lines and --all\"", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006220129", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006220129, "node_id": "IC_kwDOCGYnMM47-bNh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T01:52:26Z", "updated_at": "2022-01-06T01:52:26Z", "author_association": "OWNER", "body": "I'm going to refactor all of the tests for `sqlite-utils insert` into a new `test_cli_insert.py` module.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006219848", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006219848, "node_id": "IC_kwDOCGYnMM47-bJI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T01:51:36Z", "updated_at": "2022-01-06T01:51:36Z", "author_association": "OWNER", "body": "So far I've just implemented the new help:\r\n```\r\n% sqlite-utils insert --help\r\nUsage: sqlite-utils insert [OPTIONS] PATH TABLE FILE\r\n\r\n Insert records from FILE into a table, creating the table if it does not\r\n already exist.\r\n\r\n By default the input is expected to be a JSON array of objects. Or:\r\n\r\n - Use --nl for newline-delimited JSON objects\r\n - Use --csv or --tsv for comma-separated or tab-separated input\r\n - Use --lines to write each incoming line to a column called \"line\"\r\n - Use --all to write the entire input to a column called \"all\"\r\n\r\n You can also use --convert to pass a fragment of Python code that will be\r\n used to convert each input.\r\n\r\n Your Python code will be passed a \"row\" variable representing the imported\r\n row, and can return a modified row.\r\n\r\n If you are using --lines your code will be passed a \"line\" variable, and for\r\n --all an \"all\" variable.\r\n\r\nOptions:\r\n --pk TEXT Columns to use as the primary key, e.g. id\r\n --flatten Flatten nested JSON objects, so {\"a\": {\"b\": 1}}\r\n becomes {\"a_b\": 1}\r\n --nl Expect newline-delimited JSON\r\n -c, --csv Expect CSV input\r\n --tsv Expect TSV input\r\n --lines Treat each line as a single value called 'line'\r\n --all Treat input as a single value called 'all'\r\n --convert TEXT Python code to convert each item\r\n --import TEXT Python modules to import\r\n --delimiter TEXT Delimiter to use for CSV files\r\n --quotechar TEXT Quote character to use for CSV/TSV\r\n --sniff Detect delimiter and quote character\r\n --no-headers CSV file has no header row\r\n --batch-size INTEGER Commit every X records\r\n --alter Alter existing table to add any missing columns\r\n --not-null TEXT Columns that should be created as NOT NULL\r\n --default ... Default value that should be set for a column\r\n --encoding TEXT Character encoding for input, defaults to utf-8\r\n -d, --detect-types Detect types for columns in CSV/TSV data\r\n --load-extension TEXT SQLite extensions to load\r\n --silent Do not show progress bar\r\n --ignore Ignore records if pk already exists\r\n --replace Replace records if pk already exists\r\n --truncate Truncate table before inserting records, if table\r\n already exists\r\n -h, --help Show this message and exit.\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-991397907", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 991397907, "node_id": "IC_kwDOCGYnMM47F4gT", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-11T01:01:40Z", "updated_at": "2021-12-11T01:01:40Z", "author_association": "OWNER", "body": "The change I made to that test in #354 might help with this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982137888", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982137888, "node_id": "IC_kwDOCGYnMM46ijwg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:50:54Z", "updated_at": "2021-11-29T23:50:54Z", "author_association": "OWNER", "body": "If I'm going to `skipIf()` those tests I need a way to check if `pysqlite3` is being used.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982137293", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982137293, "node_id": "IC_kwDOCGYnMM46ijnN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:49:29Z", "updated_at": "2021-11-29T23:49:29Z", "author_association": "OWNER", "body": "A short term fix would be to skip those tests against `pysqlite3` - but longer term it would be good to address the underlying issue, particularly for the WAL ones (the FTS ones aren't too worrying since if you deliberately try and break the FTS table it's not hugely problematic if you corrupt your database).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982136747", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982136747, "node_id": "IC_kwDOCGYnMM46ijer", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:48:05Z", "updated_at": "2021-11-29T23:48:05Z", "author_association": "OWNER", "body": "Some interesting test failures in the version that runs with `pysqlite3`:\r\n```\r\n=========================== short test summary info ============================\r\nFAILED tests/test_cli.py::test_enable_wal - assert 0 == 1\r\nFAILED tests/test_cli.py::test_disable_wal - pysqlite3.dbapi2.OperationalErro...\r\nFAILED tests/test_fts.py::test_rebuild_fts[searchable] - pysqlite3.dbapi2.Dat...\r\nFAILED tests/test_fts.py::test_rebuild_fts[searchable_fts] - pysqlite3.dbapi2...\r\nFAILED tests/test_wal.py::test_enable_disable_wal - pysqlite3.dbapi2.Operatio...\r\n================== 5 failed, 750 passed, 3 skipped in 15.20s ===================\r\n```\r\nhttps://github.com/simonw/sqlite-utils/runs/4360759085\r\n\r\nThe WAL errors look like this:\r\n```\r\nE pysqlite3.dbapi2.OperationalError: cannot change into wal mode from within a transaction\r\n```\r\nTriggered by a call to `db.enable_wal()`\r\n\r\nThe FTS errors are caused by tests that try to deliberately corrupt the FTS index by running `fresh_db[\"searchable_fts_data\"].delete_where()` - and then rebuilding it using `rebuild_fts()`:\r\n```\r\n @pytest.mark.parametrize(\"table_to_fix\", [\"searchable\", \"searchable_fts\"])\r\n def test_rebuild_fts(fresh_db, table_to_fix):\r\n table = fresh_db[\"searchable\"]\r\n table.insert(search_records[0])\r\n table.enable_fts([\"text\", \"country\"])\r\n # Run a search\r\n rows = list(table.search(\"tanuki\"))\r\n assert len(rows) == 1\r\n assert {\r\n \"rowid\": 1,\r\n \"text\": \"tanuki are running tricksters\",\r\n \"country\": \"Japan\",\r\n \"not_searchable\": \"foo\",\r\n }.items() <= rows[0].items()\r\n # Delete from searchable_fts_data\r\n fresh_db[\"searchable_fts_data\"].delete_where()\r\n # This should have broken the index\r\n with pytest.raises(sqlite3.DatabaseError):\r\n list(table.search(\"tanuki\"))\r\n # Running rebuild_fts() should fix it\r\n> fresh_db[table_to_fix].rebuild_fts()\r\n\r\ntests/test_fts.py:277: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\nsqlite_utils/db.py:1947: in rebuild_fts\r\n self.db.execute(\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n\r\nself = >\r\nsql = \"INSERT INTO [searchable_fts]([searchable_fts]) VALUES('rebuild');\"\r\nparameters = None\r\n\r\n def execute(\r\n self, sql: str, parameters: Optional[Union[Iterable, dict]] = None\r\n ) -> sqlite3.Cursor:\r\n \"Execute SQL query and return a ``sqlite3.Cursor``.\"\r\n if self._tracer:\r\n self._tracer(sql, parameters)\r\n if parameters is not None:\r\n return self.conn.execute(sql, parameters)\r\n else:\r\n> return self.conn.execute(sql)\r\nE pysqlite3.dbapi2.DatabaseError: database disk image is malformed\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982133970", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982133970, "node_id": "IC_kwDOCGYnMM46iizS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:41:17Z", "updated_at": "2021-11-29T23:41:17Z", "author_association": "OWNER", "body": "Took a bit of experimenting to get both `mypy` AND `flake8` to ignore the same line. The incantation that worked was this one:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/f990e134aa8219b687ff6c261330f36824b5df36/sqlite_utils/utils.py#L8\r\n\r\nOrder here matters - this did NOT work for both tools:\r\n\r\n```python\r\n from sqlite3.dump import _iterdump as iterdump # noqa: F401 # type: ignore\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982129727", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982129727, "node_id": "IC_kwDOCGYnMM46ihw_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:31:58Z", "updated_at": "2021-11-29T23:31:58Z", "author_association": "OWNER", "body": "It failed on other Python versions with `mypy`:\r\n```\r\nsqlite_utils/utils.py:8: error: Cannot find implementation or library stub for module named \"sqlite3.dump\"\r\nsqlite_utils/utils.py:8: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982129218", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/347", "id": 982129218, "node_id": "IC_kwDOCGYnMM46ihpC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-29T23:31:02Z", "updated_at": "2021-11-29T23:31:02Z", "author_association": "OWNER", "body": "Here's the test run that's installing `pysqlite3` and that version of SQLite: https://github.com/simonw/sqlite-utils/runs/4360663292", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1066603133, "label": "Test against pysqlite3 running SQLite 3.37"}, "performed_via_github_app": null}