{"html_url": "https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655786374", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114", "id": 655786374, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTc4NjM3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T22:16:54Z", "updated_at": "2020-07-08T22:16:54Z", "author_association": "OWNER", "body": "According to https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes the hardest bits to consider are how to deal with existing foreign key relationships, triggers and views.\r\n\r\nI'm OK leaving views as an exercise for the caller - many of these transformations may not need any view changes at all.\r\n\r\nForeign key relationships are important: it should handle these automatically as effectively as possible.\r\n\r\nLikewise trigger changes: need to think about what this means.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621989740, "label": "table.transform() method for advanced alter table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655785396", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114", "id": 655785396, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTc4NTM5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T22:14:10Z", "updated_at": "2020-07-08T22:14:10Z", "author_association": "OWNER", "body": "Work in progress: not quite right yet, I need smarter logic for how renamed columns are reflected in the generated `INSERT INTO ... SELECT ...` query:\r\n```python\r\n    def transform_table(\r\n        self,\r\n        columns=None,\r\n        rename=None,\r\n        change_type=None,\r\n        pk=None,\r\n        foreign_keys=None,\r\n        column_order=None,\r\n        not_null=None,\r\n        defaults=None,\r\n        hash_id=None,\r\n        extracts=None,\r\n    ):\r\n        assert self.exists(), \"Cannot transform a table that doesn't exist yet\"\r\n        columns = columns or self.columns_dict\r\n        if rename is not None or change_type is not None:\r\n            columns = {rename.get(key, key): change_type.get(key, value) for key, value in columns.items()}\r\n        new_table_name = \"{}_new_{}\".format(self.name, os.urandom(6).hex())\r\n        previous_columns = set(self.columns_dict.keys())\r\n        with self.db.conn:\r\n            columns = {name: value for (name, value) in columns.items()}\r\n            new_table = self.db.create_table(\r\n                new_table_name,\r\n                columns,\r\n                pk=pk,\r\n                foreign_keys=foreign_keys,\r\n                column_order=column_order,\r\n                not_null=not_null,\r\n                defaults=defaults,\r\n                hash_id=hash_id,\r\n                extracts=extracts,\r\n            )\r\n            # Copy across data - but only for columns that exist in both\r\n            new_columns = set(columns.keys())\r\n            columns_to_copy = new_columns.intersection(previous_columns)\r\n            copy_sql = \"INSERT INTO [{new_table}] ({new_cols}) SELECT {old_cols} FROM [{old_table}]\".format(\r\n                new_table=new_table_name,\r\n                old_table=self.name,\r\n                old_cols=\", \".join(\"[{}]\".format(col) for col in columns_to_copy),\r\n                new_cols=\", \".join(\"[{}]\".format(rename.get(col, col)) for col in columns_to_copy),\r\n            )\r\n            self.db.conn.execute(copy_sql)\r\n            # Drop the old table\r\n            self.db.conn.execute(\"DROP TABLE [{}]\".format(self.name))\r\n            # Rename the new one\r\n            self.db.conn.execute(\r\n                \"ALTER TABLE [{}] RENAME TO [{}]\".format(new_table_name, self.name)\r\n            )\r\n        return self\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621989740, "label": "table.transform() method for advanced alter table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655783875", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114", "id": 655783875, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTc4Mzg3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T22:09:51Z", "updated_at": "2020-07-08T22:10:16Z", "author_association": "OWNER", "body": "I can have a convenient `change_type={...}` parameter for changing column types too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621989740, "label": "table.transform() method for advanced alter table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655782477", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114", "id": 655782477, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTc4MjQ3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T22:06:23Z", "updated_at": "2020-07-08T22:06:23Z", "author_association": "OWNER", "body": "Thinking about the method signature:\r\n```python\r\n    def transform_table(\r\n        self,\r\n        columns,\r\n        pk=None,\r\n        foreign_keys=None,\r\n        column_order=None,\r\n        not_null=None,\r\n        defaults=None,\r\n        hash_id=None,\r\n        extracts=None,\r\n    ):\r\n```\r\nThis requires the caller to provide the exact set of columns for the new table.\r\n\r\nIt would be useful if this was optional - if you could omit the columns and have it automatically use the previous columns. This would let you change things like the primary key or the column order using the other arguments.\r\n\r\nEven better: allow column renaming using an optional `rename={...}` argument:\r\n\r\n```python\r\ndb[\"dogs\"].transform_table(rename={\"name\": \"dog_name\"})\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621989740, "label": "table.transform() method for advanced alter table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655778058", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114", "id": 655778058, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTc3ODA1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T21:54:30Z", "updated_at": "2020-07-08T21:54:30Z", "author_association": "OWNER", "body": "Don't forget this step:\r\n>  If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621989740, "label": "table.transform() method for advanced alter table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677909", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114", "id": 655677909, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTY3NzkwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T18:16:39Z", "updated_at": "2020-07-08T18:16:39Z", "author_association": "OWNER", "body": "Since neither the term \"transform\" or \"migrate\" are used in the codebase at the moment, I think I'll go with `.transform_table()` - that leaves the term \"migrate\" available for any future database migrations system (similar to Django's).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621989740, "label": "table.transform() method for advanced alter table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677396", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114", "id": 655677396, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTY3NzM5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T18:15:39Z", "updated_at": "2020-07-08T18:15:39Z", "author_association": "OWNER", "body": "Alternative possible names:\r\n- `.transform_table()`\r\n- `.migrate()`\r\n- `.transform()`\r\n\r\nI'm torn between `.migrate_table()` and `.transform_table()`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621989740, "label": "table.transform() method for advanced alter table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677099", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114", "id": 655677099, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTY3NzA5OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T18:15:02Z", "updated_at": "2020-07-08T18:15:02Z", "author_association": "OWNER", "body": "I'm not so keen on that chained API - it's pretty complicated.\r\n\r\nHere's an idea for a much simpler interface. Essentially it lets you say \"take table X and migrate its contents to a new table with this structure - then atomically rename the tables to switch them\":\r\n```python\r\ndb[\"mytable\"].migrate_table({\"id\": int, \"name\": str\"}, pk=\"id\")\r\n```\r\nThe `migrate_table()` method would take the same exact signature as the `table.create()` method: https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L615-L625", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621989740, "label": "table.transform() method for advanced alter table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/119#issuecomment-655674910", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/119", "id": 655674910, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTY3NDkxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T18:10:18Z", "updated_at": "2020-07-08T18:10:18Z", "author_association": "OWNER", "body": "This will work similar to how `.add_foreign_keys()` works: turn on `writable_schema` and rewrite the `sql` for that table in the `sqlite_master` table.\r\n\r\nHere's that code today - it could be adapted to include removal of foreign keys that we no longer want:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L391-L401", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 652700770, "label": "Ability to remove a foreign key"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655673896", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/121", "id": 655673896, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTY3Mzg5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T18:08:11Z", "updated_at": "2020-07-08T18:08:11Z", "author_association": "OWNER", "body": "I'm with you on most of this. Completely agreed that the CLI should do everything in a transaction.\r\n\r\nThe one thing I'm not keen on is forcing calling code to explicitly start a transaction, for a couple of reasons:\r\n\r\n1. It will break all of the existing code out there\r\n2. It doesn't match to how I most commonly use this library - as an interactive tool in a Jupyter notebook, where I'm generally working against a brand new scratch database and any errors don't actually matter\r\n\r\nSo... how about this: IF you wrap your code in a `with db:` block then the `.insert()` and suchlike methods expect you to manage transactions yourself. But if you don't use the context manager they behave like they do at the moment (or maybe a bit more sensibly).\r\n\r\nThat way existing code works as it does today, lazy people like me can call `.insert()` without thinking about transactions, but people writing actual production code (as opposed to Jupyter hacks) have a sensible way to take control of the transactions themselves.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 652961907, "label": "Improved (and better documented) support for transactions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655653292", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655653292, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTY1MzI5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T17:26:02Z", "updated_at": "2020-07-08T17:26:02Z", "author_association": "OWNER", "body": "Awesome, thank you very much.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655290625", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/114", "id": 655290625, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTI5MDYyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T05:15:45Z", "updated_at": "2020-07-08T05:15:45Z", "author_association": "OWNER", "body": "Ideally this would all happen in a single transaction, such that other processes talking to the database would not see any inconsistent state while the table copy was taking place. Need to confirm that this is possible. Also refs transactions thoughts in #121.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 621989740, "label": "table.transform() method for advanced alter table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/120#issuecomment-655289686", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/120", "id": 655289686, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTI4OTY4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T05:13:11Z", "updated_at": "2020-07-08T05:13:11Z", "author_association": "OWNER", "body": "This is an excellent fix, thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 652816158, "label": "Fix query command's support for DML"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655286864", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655286864, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTI4Njg2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T05:05:27Z", "updated_at": "2020-07-08T05:05:36Z", "author_association": "OWNER", "body": "The only thing missing from this PR is updates to the documentation. Those need to go in two places:\r\n\r\n- In the Python API docs. I suggest adding a note to this section about bulk inserts: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/python-api.rst#bulk-inserts\r\n- In the CLI docs, in this section: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/cli.rst#inserting-json-data\r\n\r\nHere's an example of a previous commit that includes updates to both CLI and API documentation: https://github.com/simonw/sqlite-utils/commit/f9473ace14878212c1fa968b7bd2f51e4f064dba#diff-e3e2a9bfd88566b05001b02a3f51d286", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284168", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655284168, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTI4NDE2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T04:58:00Z", "updated_at": "2020-07-08T04:58:00Z", "author_association": "OWNER", "body": "Oops didn't mean to click \"close\" there.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284054", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655284054, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTI4NDA1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T04:57:38Z", "updated_at": "2020-07-08T04:57:38Z", "author_association": "OWNER", "body": "Thoughts on transactions would be much appreciated in #121 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655283393, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTI4MzM5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T04:55:18Z", "updated_at": "2020-07-08T04:55:18Z", "author_association": "OWNER", "body": "This is a really good idea - and thank you for the detailed discussion in the pull request.\r\n\r\nI'm keen to discuss how transactions can work better. I tend to use this pattern in my own code:\r\n\r\n    with db.conn:\r\n        db[\"table\"].insert(...)\r\n\r\nBut it's not documented and I've not though very hard about it!\r\n\r\nI like having inserts that handle 10,000+ rows commit on every chunk so I can watch their progress from another process, but the library should absolutely support people who want to commit all of the rows in a single transaction - or combine changes with DML.\r\n\r\nLots to discuss here. I'll start a new issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}