{"html_url": "https://github.com/simonw/sqlite-utils/issues/119#issuecomment-683146200", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/119", "id": 683146200, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE0NjIwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T21:05:37Z", "updated_at": "2020-08-28T21:05:37Z", "author_association": "OWNER", "body": "Maybe use `transform_table()` in #114 for this? Would be less efficient as it would copy the whole table but it would reduce library complexity a bit.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 652700770, "label": "Ability to remove a foreign key"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682284908", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 682284908, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MjI4NDkwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T02:11:40Z", "updated_at": "2020-08-28T02:11:40Z", "author_association": "OWNER", "body": "This is deliberate behaviour, but I'm not at all attached to it - you're right in pointing out that it's actually pretty unexpected.\r\n\r\nI'd be happy to change this behaviour so if you pass `alter=True` and then use `.insert_all()` on more than 100 rows it works as you would expect, instead of silently ignoring new columns past the first 100 rows. I don't expect that anyone would be depending on the current behaviour (ignore new columns after the first 100) such that this should be considered a backwards incompatible change.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682285212", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 682285212, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MjI4NTIxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T02:12:51Z", "updated_at": "2020-08-28T02:12:51Z", "author_association": "OWNER", "body": "I'd be happy to accept a PR for this, provided it included updated unit tests that illustrate it working. I think this is a really good improvement.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682762911", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 682762911, "node_id": "MDEyOklzc3VlQ29tbWVudDY4Mjc2MjkxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T15:54:57Z", "updated_at": "2020-08-28T15:55:20Z", "author_association": "OWNER", "body": "Here's a suggested test update:\r\n```diff\r\ndiff --git a/sqlite_utils/db.py b/sqlite_utils/db.py\r\nindex a8791c3..12fa2f2 100644\r\n--- a/sqlite_utils/db.py\r\n+++ b/sqlite_utils/db.py\r\n@@ -1074,6 +1074,13 @@ class Table(Queryable):\r\n                 all_columns = list(sorted(all_columns))\r\n                 if hash_id:\r\n                     all_columns.insert(0, hash_id)\r\n+            else:\r\n+                all_columns += [\r\n+                    column\r\n+                    for record in chunk\r\n+                    for column in record\r\n+                    if column not in all_columns\r\n+                ]\r\n             validate_column_names(all_columns)\r\n             first = False\r\n             # values is the list of insert data that is passed to the\r\ndiff --git a/tests/test_create.py b/tests/test_create.py\r\nindex a84eb8d..3a7fafc 100644\r\n--- a/tests/test_create.py\r\n+++ b/tests/test_create.py\r\n@@ -707,13 +707,15 @@ def test_insert_thousands_using_generator(fresh_db):\r\n     assert 10000 == fresh_db[\"test\"].count\r\n \r\n \r\n-def test_insert_thousands_ignores_extra_columns_after_first_100(fresh_db):\r\n+def test_insert_thousands_adds_extra_columns_after_first_100(fresh_db):\r\n+    # https://github.com/simonw/sqlite-utils/issues/139\r\n     fresh_db[\"test\"].insert_all(\r\n         [{\"i\": i, \"word\": \"word_{}\".format(i)} for i in range(100)]\r\n-        + [{\"i\": 101, \"extra\": \"This extra column should cause an exception\"}]\r\n+        + [{\"i\": 101, \"extra\": \"Should trigger ALTER\"}],\r\n+        alter=True,\r\n     )\r\n     rows = fresh_db.execute_returning_dicts(\"select * from test where i = 101\")\r\n-    assert [{\"i\": 101, \"word\": None}] == rows\r\n+    assert [{\"i\": 101, \"word\": None, \"extra\": \"Should trigger ALTER\"}] == rows\r\n \r\n \r\n def test_insert_ignore(fresh_db):\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682771226", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 682771226, "node_id": "MDEyOklzc3VlQ29tbWVudDY4Mjc3MTIyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T15:57:42Z", "updated_at": "2020-08-28T15:57:42Z", "author_association": "OWNER", "body": "That pull request should update this section of the documentation too:\r\n\r\n> If you have more than one record to insert, the insert_all() method is a much more efficient way of inserting them. Just like insert() it will automatically detect the columns that should be created, but it will inspect the first batch of 100 items to help decide what those column types should be.\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/ea87c2b943fdd162c42a900ac0aea5ecc2f4b9d9/docs/python-api.rst#L393", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682815377", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 682815377, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MjgxNTM3Nw==", "user": {"value": 96218, "label": "simonwiles"}, "created_at": "2020-08-28T16:14:58Z", "updated_at": "2020-08-28T16:14:58Z", "author_association": "CONTRIBUTOR", "body": "Thanks!  And yeah, I had updating the docs on my list too :)  Will try to get to it this afternoon (budgeting time is fraught with uncertainty at the moment!).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-683178570", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 683178570, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE3ODU3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:48:51Z", "updated_at": "2020-08-28T22:48:51Z", "author_association": "OWNER", "body": "Thanks @simonwiles, this is now released in 2.16.1: https://sqlite-utils.readthedocs.io/en/stable/changelog.html", "reactions": "{\"total_count\": 2, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/953#issuecomment-682312494", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/953", "id": 682312494, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MjMxMjQ5NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T04:03:56Z", "updated_at": "2020-08-28T04:03:56Z", "author_association": "OWNER", "body": "Documentation says that the old dictionary mechanism will be deprecated by 1.0:\r\n\r\nhttps://github.com/simonw/datasette/blob/799ecae94824640bdff21f86997f69844048d5c3/docs/plugin_hooks.rst#L460", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 687681018, "label": "register_output_renderer render function should be able to return a Response"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/954#issuecomment-682312736", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/954", "id": 682312736, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MjMxMjczNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T04:05:01Z", "updated_at": "2020-08-28T04:05:10Z", "author_association": "OWNER", "body": ">  It can also return a dictionary with the following keys. This format is **deprecated** as-of Datasette 0.49 and will be removed by Datasette 1.0.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 687694947, "label": "Remove old register_output_renderer dict mechanism in Datasette 1.0"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/955#issuecomment-683185861", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/955", "id": 683185861, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE4NTg2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T23:17:09Z", "updated_at": "2020-08-28T23:17:09Z", "author_association": "OWNER", "body": "I released 0.49a0 which means I can update the main branches of those two plugins - I'll push a release of them once 0.49 is fully shipped.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 687711713, "label": "Release updated datasette-atom and datasette-ics"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/955#issuecomment-683189334", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/955", "id": 683189334, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE4OTMzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T23:30:48Z", "updated_at": "2020-08-28T23:30:48Z", "author_association": "OWNER", "body": "Also https://github.com/simonw/datasette-copyable", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 687711713, "label": "Release updated datasette-atom and datasette-ics"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172082", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/142", "id": 683172082, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE3MjA4Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:24:25Z", "updated_at": "2020-08-28T22:24:25Z", "author_association": "OWNER", "body": "Thanks very much!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688386219, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172829", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/142", "id": 683172829, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE3MjgyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:27:05Z", "updated_at": "2020-08-28T22:27:05Z", "author_association": "OWNER", "body": "Looks like it failed the \"black\" formatting test - possibly because there's a new release if black out. I'm going to merge despite that failure.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688386219, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683173375", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/142", "id": 683173375, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE3MzM3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:29:02Z", "updated_at": "2020-08-28T22:29:02Z", "author_association": "OWNER", "body": "Yeah I think that failure is actually because there's a brand new release of Black out and it subtly changes some of the formatting rules. I'll merge this and then run Black against the entire codebase.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688386219, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/143#issuecomment-683175491", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/143", "id": 683175491, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE3NTQ5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:37:15Z", "updated_at": "2020-08-28T22:37:15Z", "author_association": "OWNER", "body": "I'm going to start running black exclusively in the GitHub Actions workflow, rather than having it run by the unit tests themselves.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688389933, "label": "Move to GitHub Actions CI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/144#issuecomment-683179678", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/144", "id": 683179678, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE3OTY3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:53:17Z", "updated_at": "2020-08-28T22:53:17Z", "author_association": "OWNER", "body": "I'm going to try doing this as a GitHub Actions test matrix.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688395275, "label": "Run some tests against numpy"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/144#issuecomment-683180581", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/144", "id": 683180581, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE4MDU4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:57:04Z", "updated_at": "2020-08-28T22:57:04Z", "author_association": "OWNER", "body": "That worked! https://github.com/simonw/sqlite-utils/runs/1043640785?check_suite_focus=true\r\n\r\n<img width=\"1001\" alt=\"Add_numpy_to_the_matrix__refs__144_\u00b7_simonw_sqlite-utils_cbc22ef\" src=\"https://user-images.githubusercontent.com/9599/91621188-1b686e00-e947-11ea-906d-129ceadbb434.png\">\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688395275, "label": "Run some tests against numpy"}, "performed_via_github_app": null}