{"html_url": "https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515751719", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/50", "id": 515751719, "node_id": "MDEyOklzc3VlQ29tbWVudDUxNTc1MTcxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-28T10:40:11Z", "updated_at": "2019-07-28T10:40:11Z", "author_association": "OWNER", "body": "I think the fix here is for me to switch to using `executemany()` - example from the Python docs: https://docs.python.org/3/library/sqlite3.html\r\n```python\r\npurchases = [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),\r\n             ('2006-04-05', 'BUY', 'MSFT', 1000, 72.00),\r\n             ('2006-04-06', 'SELL', 'IBM', 500, 53.00),\r\n            ]\r\nc.executemany('INSERT INTO stocks VALUES (?,?,?,?,?)', purchases)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 473083260, "label": "\"Too many SQL variables\" on large inserts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515752129", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/50", "id": 515752129, "node_id": "MDEyOklzc3VlQ29tbWVudDUxNTc1MjEyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-28T10:46:49Z", "updated_at": "2019-07-28T10:46:49Z", "author_association": "OWNER", "body": "The problem with `.executemany()` is it breaks `lastrowid`:\r\n\r\n> This read-only attribute provides the rowid of the last modified row. It is only set if you issued an INSERT or a REPLACE statement using the execute() method. For operations other than INSERT or REPLACE or when executemany() is called, lastrowid is set to None.\r\n\r\nSo I think I need to continue to use my existing way of executing bulk inserts (with a giant repeated `INSERT INTO ... VALUES` block) but ensure that I calculate the chunk size such that I don't ever try to pass more than 999 values at once.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 473083260, "label": "\"Too many SQL variables\" on large inserts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515752204", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/50", "id": 515752204, "node_id": "MDEyOklzc3VlQ29tbWVudDUxNTc1MjIwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-28T10:48:14Z", "updated_at": "2019-07-28T10:48:14Z", "author_association": "OWNER", "body": "Here's the diff where I tried to use `.executemany()` and ran into the `lastrowid` problem:\r\n```diff\r\ndiff --git a/sqlite_utils/db.py b/sqlite_utils/db.py\r\nindex ef55976..7f85759 100644\r\n--- a/sqlite_utils/db.py\r\n+++ b/sqlite_utils/db.py\r\n@@ -881,13 +881,10 @@ class Table:\r\n                 or_what=or_what,\r\n                 table=self.name,\r\n                 columns=\", \".join(\"[{}]\".format(c) for c in all_columns),\r\n-                rows=\", \".join(\r\n-                    \"\"\"\r\n+                rows=\"\"\"\r\n                     ({placeholders})\r\n                 \"\"\".format(\r\n-                        placeholders=\", \".join([\"?\"] * len(all_columns))\r\n-                    )\r\n-                    for record in chunk\r\n+                    placeholders=\", \".join([\"?\"] * len(all_columns))\r\n                 ),\r\n             )\r\n             values = []\r\n@@ -902,15 +899,15 @@ class Table:\r\n                         extract_table = extracts[key]\r\n                         value = self.db[extract_table].lookup({\"value\": value})\r\n                     record_values.append(value)\r\n-                values.extend(record_values)\r\n+                values.append(record_values)\r\n             with self.db.conn:\r\n                 try:\r\n-                    result = self.db.conn.execute(sql, values)\r\n+                    result = self.db.conn.executemany(sql, values)\r\n                 except sqlite3.OperationalError as e:\r\n                     if alter and (\" has no column \" in e.args[0]):\r\n                         # Attempt to add any missing columns, then try again\r\n                         self.add_missing_columns(chunk)\r\n-                        result = self.db.conn.execute(sql, values)\r\n+                        result = self.db.conn.executemany(sql, values)\r\n                     else:\r\n                         raise\r\n                 self.last_rowid = result.lastrowid\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 473083260, "label": "\"Too many SQL variables\" on large inserts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/51#issuecomment-515756563", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/51", "id": 515756563, "node_id": "MDEyOklzc3VlQ29tbWVudDUxNTc1NjU2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-28T11:56:10Z", "updated_at": "2019-07-28T11:56:10Z", "author_association": "OWNER", "body": "Interesting. The tests failed presumably because the version of SQLite I am running on Travis CI doesn't have that same 999 limit. I'm going to enforce the 999 limit within the library itself, to discourage people from writing code which will fail if it runs on a host that DOES have that limit.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 473733752, "label": "Fix for too many SQL variables, closes #50"}, "performed_via_github_app": null}