issue_comments
7,983 rows sorted by issue_url descending
This data as json, CSV (advanced)
Suggested facets: reactions, created_at (date), updated_at (date)
| id | html_url | issue_url ▲ | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 612727400 | https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727400 | https://api.github.com/repos/simonw/sqlite-utils/issues/99 | MDEyOklzc3VlQ29tbWVudDYxMjcyNzQwMA== | simonw 9599 | 2020-04-13T03:03:09Z | 2020-04-13T03:03:09Z | OWNER | I think I'm going to leave this as intended behaviour. Or maybe passing multiple dictionaries to `.upsert_all()` with different numbers of keys should raise an error? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .upsert_all() should maybe error if dictionaries passed to it do not have the same keys 598640234 | |
| 612727814 | https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727814 | https://api.github.com/repos/simonw/sqlite-utils/issues/99 | MDEyOklzc3VlQ29tbWVudDYxMjcyNzgxNA== | simonw 9599 | 2020-04-13T03:05:04Z | 2020-04-13T03:05:04Z | OWNER | Bit trick from an implementation point of view this, since we want to be able to handle input that is a generator - so we can't scan through the input to validate that every dictionary has the same exact keys without consuming the entire iterator. The alternative would be to raise an error the first time we spot a dictionary with keys that differ... but that's weird because we commit changes in batches, so we may end up only applying half of the changes before exiting with the error. On that basis, I'm going to leave this as-is and mark this as wontfix. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .upsert_all() should maybe error if dictionaries passed to it do not have the same keys 598640234 | |
| 612173156 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612173156 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | MDEyOklzc3VlQ29tbWVudDYxMjE3MzE1Ng== | simonw 9599 | 2020-04-10T19:03:32Z | 2020-04-10T23:08:28Z | OWNER | Investigate this traceback: ``` Traceback (most recent call last): File "fetch_projects.py", line 60, in <module> fetch_projects(db, token) File "fetch_projects.py", line 41, in fetch_projects db["projects"].upsert(project, pk="id") File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py", line 1139, in upsert conversions=conversions, File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py", line 1168, in upsert_all upsert=True, File "/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py", line 1107, in insert_all row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0] IndexError: list index out of range ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
| 612258687 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612258687 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | MDEyOklzc3VlQ29tbWVudDYxMjI1ODY4Nw== | simonw 9599 | 2020-04-10T23:08:48Z | 2020-04-10T23:08:48Z | OWNER | I need a test that reproduces this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
| 612707293 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707293 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | MDEyOklzc3VlQ29tbWVudDYxMjcwNzI5Mw== | simonw 9599 | 2020-04-13T01:21:22Z | 2020-04-13T01:21:22Z | OWNER | I have a hunch that the root of the problem here is that accessing `result.lastrowid` during my version of an `.upsert()` doesn't actually make sense: https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1102-L1113 In the bug I'm seeing (which I still haven't reduced to a reproducible test) the debugger shows me this at that point: ``` (Pdb) query 'UPDATE [files] SET [createdAt] = ?, [ext] = ?, [updatedAt] = ?, [uri] = ?, [uriType] = ? WHERE [project] = ? AND [name] = ?' (Pdb) params ['2020-03-04T04:04:40.152000+00:00', 'csv', '2020-03-04T04:04:40.152000+00:00', 'https://storage.googleapis.com/bln_prod/...', 'download', 'UHJvamVjdDo4MTgyMjU2Ny01ZjI0LTQxM2ItYWZmNi05NTlmNGY3MjExMjI=', 'loans_to_documentation.csv'] (Pdb) result.lastrowid 100 ``` But here's the weird thing... there's no row in the table with a rowid of 100! ``` (Pdb) [r['rowid'] for r in self.db.execute_returning_dicts('select rowid, * from files')] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99] ``` So what the heck is going on? The last SQL statement I executed here was an `UPDATE`. The `lastrowid` docs say: https://kite.com/python/docs/sqlite3.Cursor.lastrowid > This read-only attribute provides the rowid of the last modified row. It is only set if you issued a INSERT statement using the execute() method. For operations other than INSERT or when executemany() is called, lastrowid is set to None. So where did that `100` come from? It should be `None`! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
| 612707828 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707828 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | MDEyOklzc3VlQ29tbWVudDYxMjcwNzgyOA== | simonw 9599 | 2020-04-13T01:24:05Z | 2020-04-13T01:24:16Z | OWNER | Why do I even care about `lastrowid` here? I'm trying to ensure that after you insert or upsert a row you can use `table.last_pk` to start doing things like building additional foreign key relationships. So maybe it doesn't make sense to make `.last_pk` available _at all_ for cases where you called `.upsert_all()` or `.insert_all()` - it should just be populated for `.upsert()` and `.insert()`. The documentation doesn't say it should work for `.upsert_all()` - it's only documented for the single actions. https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1113-L1124 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
| 612708274 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612708274 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | MDEyOklzc3VlQ29tbWVudDYxMjcwODI3NA== | simonw 9599 | 2020-04-13T01:25:59Z | 2020-04-13T01:26:11Z | OWNER | In mucking around with `sqlite3` it looks like `result.lastrowid` is indeed populated for `UPDATE` - in this case with the last inserted rowid in the table. This differs from the documented behaviour I linked to above. ``` In [1]: import sqlite3 In [2]: c = sqlite3.connect(":memory:") In [3]: c Out[3]: <sqlite3.Connection at 0x103c4d490> In [4]: c.execute('create table foo (bar integer);') Out[4]: <sqlite3.Cursor at 0x103f760a0> In [5]: c.execute('insert into foo (bar) values (1)') Out[5]: <sqlite3.Cursor at 0x103f76650> In [6]: c.execute('select * from foo').fetchall() Out[6]: [(1,)] In [7]: c.execute('insert into foo (bar) values (1)') Out[7]: <sqlite3.Cursor at 0x103f766c0> In [8]: c.execute('select * from foo').fetchall() Out[8]: [(1,), (1,)] In [9]: c.execute('insert into foo (bar) values (1)').lastrowid Out[9]: 3 In [10]: c.execute('select * from foo').fetchall() Out[10]: [(1,), (1,), (1,)] In [11]: c.execute('select rowid, bar from foo').fetchall() Out[11]: [(1, 1), (2, 1), (3, 1)] In [12]: c.execute('insert into foo (bar) values (1)').lastrowid Out[12]: 4 In [13]: c.execute('select rowid, bar from foo').fetchall() Out[13]: [(1, 1), (2, 1), (3, 1), (4, 1)] In [14]: r = c.execute('update foo set bar =2 where rowid = 1') In [15]: r.lastrowid Out[15]: 4 In [16]: c.execute('select rowid, bar from foo').fetchall() Out[16]: [(1, 2), (2, 1), (3, 1), (4, 1)] In [17]: r = c.execute('select rowid, bar from foo') In… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
| 612728047 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612728047 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | MDEyOklzc3VlQ29tbWVudDYxMjcyODA0Nw== | simonw 9599 | 2020-04-13T03:06:10Z | 2020-04-13T03:06:10Z | OWNER | Implementation plan: `.insert_all()` and `.upsert_all()` should only set `.last_rowid` and `last_pk` if they were called with a single item. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
| 928790381 | https://github.com/simonw/sqlite-utils/issues/98#issuecomment-928790381 | https://api.github.com/repos/simonw/sqlite-utils/issues/98 | IC_kwDOCGYnMM43XDdt | patricktrainer 36834097 | 2021-09-28T04:38:44Z | 2021-09-28T04:38:44Z | NONE | Hi @simonw - wondering if you might be able to shed some light here. I've seemed to reproduce this issue. Here's the stacktrace: ``` ... db["potholes"].insert(pothole, pk='id', alter=True, replace=True) ... Traceback (most recent call last): File "<stdin>", line 3, in <module> File "/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py", line 2481, in insert return self.insert_all( File "/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py", line 2596, in insert_all self.insert_chunk( File "/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py", line 2424, in insert_chunk row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0] IndexError: list index out of range ``` Interesting enough, I found that omitting the `pk` param does not throw the error. Let me know how I can help out! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records 597671518 | |
| 612732129 | https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732129 | https://api.github.com/repos/simonw/sqlite-utils/issues/97 | MDEyOklzc3VlQ29tbWVudDYxMjczMjEyOQ== | simonw 9599 | 2020-04-13T03:25:29Z | 2020-04-13T03:25:29Z | OWNER | Interesting thought. I've run into this myself a lot - many of my scripts intend to create the database from scratch, so I end up running `!rm /tmp/blah.db` in Jupyter and occasionally getting errors if the file doesn't exist. I think adding `recreate=True` could make sense. It could throw an error if you attempt to use it after passing in something other than a path to a file on disk. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Adding a "recreate" flag to the `Database` constructor 593751293 | |
| 612732453 | https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732453 | https://api.github.com/repos/simonw/sqlite-utils/issues/97 | MDEyOklzc3VlQ29tbWVudDYxMjczMjQ1Mw== | simonw 9599 | 2020-04-13T03:26:46Z | 2020-04-13T03:26:46Z | OWNER | I wonder if it should delete an recreate the file or if it would be safer to drop every table instead? Dropping tables gets messy: then you need to drop triggers and views, and you need to run `vacuum` to clean up the space. My worry with deleting and recreating the file is that it could trigger errors in other processes that are currently attached to that database file. But... if you know that's going to be likely, maybe you shouldn't use the `recreate=True` feature? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Adding a "recreate" flag to the `Database` constructor 593751293 | |
| 612738311 | https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612738311 | https://api.github.com/repos/simonw/sqlite-utils/issues/97 | MDEyOklzc3VlQ29tbWVudDYxMjczODMxMQ== | simonw 9599 | 2020-04-13T03:55:11Z | 2020-04-13T03:55:11Z | OWNER | Shipped in 2.5 - documentation is here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#connecting-to-or-creating-a-database | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Adding a "recreate" flag to the `Database` constructor 593751293 | |
| 614073859 | https://github.com/simonw/sqlite-utils/issues/97#issuecomment-614073859 | https://api.github.com/repos/simonw/sqlite-utils/issues/97 | MDEyOklzc3VlQ29tbWVudDYxNDA3Mzg1OQ== | betatim 1448859 | 2020-04-15T14:29:30Z | 2020-04-15T14:29:30Z | NONE | Woah! Thanks a lot. Next time I will add a more obvious/explicit "if you like this idea let me know I'd love to work on it to get my feet wet here" :D | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Adding a "recreate" flag to the `Database` constructor 593751293 | |
| 606394349 | https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394349 | https://api.github.com/repos/simonw/sqlite-utils/issues/96 | MDEyOklzc3VlQ29tbWVudDYwNjM5NDM0OQ== | simonw 9599 | 2020-03-31T04:37:16Z | 2020-03-31T04:37:16Z | OWNER | Test failure was just a Black formatting issue. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add type conversion for Panda's Timestamp 589801352 | |
| 606394619 | https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394619 | https://api.github.com/repos/simonw/sqlite-utils/issues/96 | MDEyOklzc3VlQ29tbWVudDYwNjM5NDYxOQ== | simonw 9599 | 2020-03-31T04:38:17Z | 2020-03-31T04:40:23Z | OWNER | I wonder if there are any other Pandas conversions we should be doing? https://pandas.pydata.org/pandas-docs/stable/getting_started/basics.html#dtypes | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add type conversion for Panda's Timestamp 589801352 | |
| 599125455 | https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599125455 | https://api.github.com/repos/simonw/sqlite-utils/issues/92 | MDEyOklzc3VlQ29tbWVudDU5OTEyNTQ1NQ== | simonw 9599 | 2020-03-14T19:34:35Z | 2020-03-14T19:34:35Z | OWNER | From https://www.sqlite.org/datatype3.html it looks like `FLOAT` is a supported keyword for creating tables but `REAL` is the correct keyword. So actually `sqlite-utils` gets this wrong, because when we create a table we turn Python `float` values into a `FLOAT` column. Looks like the correct behaviour would be to turn them into a `REAL` column. https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L28-L48 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .columns_dict doesn't work for all possible column types 581339961 | |
| 599125557 | https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599125557 | https://api.github.com/repos/simonw/sqlite-utils/issues/92 | MDEyOklzc3VlQ29tbWVudDU5OTEyNTU1Nw== | simonw 9599 | 2020-03-14T19:35:29Z | 2020-03-14T19:35:29Z | OWNER | Fixing that would technically constitute a breaking change for library consumers, so it should be a major version release. I'm not inclined to release `3.0` just for this one issue, so I'm going to hold back on fixing that and address the smaller issue in this bug as a dot release instead for the moment. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .columns_dict doesn't work for all possible column types 581339961 | |
| 599126831 | https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599126831 | https://api.github.com/repos/simonw/sqlite-utils/issues/92 | MDEyOklzc3VlQ29tbWVudDU5OTEyNjgzMQ== | simonw 9599 | 2020-03-14T19:45:28Z | 2020-03-14T19:45:28Z | OWNER | Turns out there are a TON of valid column definitions that aren't being considered yet - https://www.sqlite.org/datatype3.html#affinity_name_examples - stuff like `VARYING CHARACTER(255)` and `DECIMAL(10,5)`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .columns_dict doesn't work for all possible column types 581339961 | |
| 599127197 | https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599127197 | https://api.github.com/repos/simonw/sqlite-utils/issues/92 | MDEyOklzc3VlQ29tbWVudDU5OTEyNzE5Nw== | simonw 9599 | 2020-03-14T19:48:06Z | 2020-03-14T19:48:06Z | OWNER | Actually it looks like I should implement the exact rules described in https://www.sqlite.org/datatype3.html#determination_of_column_affinity > The affinity of a column is determined by the declared type of the column, according to the following rules in the order shown: > > 1. If the declared type contains the string "INT" then it is assigned INTEGER affinity. > 2. If the declared type of the column contains any of the strings "CHAR", "CLOB", or "TEXT" then that column has TEXT affinity. Notice that the type VARCHAR contains the string "CHAR" and is thus assigned TEXT affinity. > 3. If the declared type for a column contains the string "BLOB" or if no type is specified then the column has affinity BLOB. > 4. If the declared type for a column contains any of the strings "REAL", "FLOA", or "DOUB" then the column has REAL affinity. > 5. Otherwise, the affinity is NUMERIC. > > Note that the order of the rules for determining column affinity is important. A column whose declared type is "CHARINT" will match both rules 1 and 2 but the first rule takes precedence and so the column affinity will be INTEGER. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .columns_dict doesn't work for all possible column types 581339961 | |
| 599127453 | https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599127453 | https://api.github.com/repos/simonw/sqlite-utils/issues/92 | MDEyOklzc3VlQ29tbWVudDU5OTEyNzQ1Mw== | simonw 9599 | 2020-03-14T19:50:08Z | 2020-03-14T19:50:08Z | OWNER | > If the declared type for a column contains the string "BLOB" or if no type is specified then the column has affinity BLOB I currently treat those as `str` - it sounds like I should treat them as `bytes`: https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L68-L69 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .columns_dict doesn't work for all possible column types 581339961 | |
| 599128891 | https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599128891 | https://api.github.com/repos/simonw/sqlite-utils/issues/92 | MDEyOklzc3VlQ29tbWVudDU5OTEyODg5MQ== | simonw 9599 | 2020-03-14T20:03:45Z | 2020-03-14T20:03:45Z | OWNER | I'm going to keep treating them as `str`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .columns_dict doesn't work for all possible column types 581339961 | |
| 599247833 | https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599247833 | https://api.github.com/repos/simonw/sqlite-utils/issues/92 | MDEyOklzc3VlQ29tbWVudDU5OTI0NzgzMw== | simonw 9599 | 2020-03-15T18:37:28Z | 2020-03-15T18:37:43Z | OWNER | Released in 2.4.2. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .columns_dict doesn't work for all possible column types 581339961 | |
| 723350956 | https://github.com/simonw/sqlite-utils/issues/91#issuecomment-723350956 | https://api.github.com/repos/simonw/sqlite-utils/issues/91 | MDEyOklzc3VlQ29tbWVudDcyMzM1MDk1Ng== | simonw 9599 | 2020-11-06T23:53:25Z | 2020-11-06T23:53:25Z | OWNER | This is now possible, for both FTS4 and FTS5 - see #197. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Enable ordering FTS results by rank 577302229 | |
| 593122605 | https://github.com/simonw/sqlite-utils/issues/89#issuecomment-593122605 | https://api.github.com/repos/simonw/sqlite-utils/issues/89 | MDEyOklzc3VlQ29tbWVudDU5MzEyMjYwNQ== | chrishas35 35075 | 2020-03-01T17:33:11Z | 2020-03-01T17:33:11Z | NONE | If you're happy with the proposed implementation, I have code & tests written that I'll get ready for a PR. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to customize columns used by extracts= feature 573578548 | |
| 615515867 | https://github.com/simonw/sqlite-utils/issues/89#issuecomment-615515867 | https://api.github.com/repos/simonw/sqlite-utils/issues/89 | MDEyOklzc3VlQ29tbWVudDYxNTUxNTg2Nw== | simonw 9599 | 2020-04-18T00:00:41Z | 2020-04-18T00:00:41Z | OWNER | Yes pleas, I'd love to see that pull request! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to customize columns used by extracts= feature 573578548 | |
| 710460242 | https://github.com/simonw/sqlite-utils/issues/89#issuecomment-710460242 | https://api.github.com/repos/simonw/sqlite-utils/issues/89 | MDEyOklzc3VlQ29tbWVudDcxMDQ2MDI0Mg== | simonw 9599 | 2020-10-16T19:17:27Z | 2020-10-16T19:17:50Z | OWNER | I came up with potential syntax for that here: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550 - based on how `table.extract(...)` works: ```python fresh_db.table("tree", extracts=[Extract( columns=("CommonName", "LatinName"), table="Species", fk_column="species_id", rename={"CommonName": "name", "LatinName": "latin"} )]) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to customize columns used by extracts= feature 573578548 | |
| 591768604 | https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591768604 | https://api.github.com/repos/simonw/sqlite-utils/issues/88 | MDEyOklzc3VlQ29tbWVudDU5MTc2ODYwNA== | simonw 9599 | 2020-02-27T04:03:03Z | 2020-02-27T04:03:03Z | OWNER | `drop table resources_fts` drops the FTS table and the other ones that it created (resources_fts_data, resources_fts_idx, resources_fts_docsize, resources_fts_config) - but keeps the triggers. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.disable_fts() method and "sqlite-utils disable-fts ..." command 571805300 | |
| 591769046 | https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769046 | https://api.github.com/repos/simonw/sqlite-utils/issues/88 | MDEyOklzc3VlQ29tbWVudDU5MTc2OTA0Ng== | simonw 9599 | 2020-02-27T04:05:15Z | 2020-02-27T04:05:15Z | OWNER | I can reliably get the list of triggers to delete from `select name from sqlite_master where type = 'trigger' and tbl_name = 'resources';` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.disable_fts() method and "sqlite-utils disable-fts ..." command 571805300 | |
| 591769171 | https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769171 | https://api.github.com/repos/simonw/sqlite-utils/issues/88 | MDEyOklzc3VlQ29tbWVudDU5MTc2OTE3MQ== | simonw 9599 | 2020-02-27T04:05:58Z | 2020-02-27T04:26:31Z | OWNER | Strange - https://www.sqlite.org/lang_droptrigger.html says "Note that triggers are automatically dropped when the associated table is dropped" but that doesn't seem to be true in my experimenting. UPDATE: no that makes sense - the triggers are on `resources` which still exists, it was `resources_fts` that was dropped. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.disable_fts() method and "sqlite-utils disable-fts ..." command 571805300 | |
| 591769373 | https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769373 | https://api.github.com/repos/simonw/sqlite-utils/issues/88 | MDEyOklzc3VlQ29tbWVudDU5MTc2OTM3Mw== | simonw 9599 | 2020-02-27T04:06:47Z | 2020-02-27T04:06:47Z | OWNER | Looks like safest option is to loop through those trigger names and run `DROP TRIGGER IF EXISTS foo` on each one. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.disable_fts() method and "sqlite-utils disable-fts ..." command 571805300 | |
| 591769759 | https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769759 | https://api.github.com/repos/simonw/sqlite-utils/issues/88 | MDEyOklzc3VlQ29tbWVudDU5MTc2OTc1OQ== | simonw 9599 | 2020-02-27T04:08:29Z | 2020-02-27T04:08:29Z | OWNER | I think the method should be called `table.disable_fts()` - the opposite of `table.enable_fts(...)`. There should be a `sqlite-utils disable-fts database.db tablename` command to match it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | table.disable_fts() method and "sqlite-utils disable-fts ..." command 571805300 | |
| 586661250 | https://github.com/simonw/sqlite-utils/issues/87#issuecomment-586661250 | https://api.github.com/repos/simonw/sqlite-utils/issues/87 | MDEyOklzc3VlQ29tbWVudDU4NjY2MTI1MA== | simonw 9599 | 2020-02-16T02:19:33Z | 2020-02-16T02:19:33Z | OWNER | Here's the code: https://github.com/simonw/sqlite-utils/blob/5e0000609f9be6efafea1b96f610988eb18d6d89/sqlite_utils/utils.py#L18-L24 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Should detect collections.OrderedDict as a regular dictionary 565837965 | |
| 586661276 | https://github.com/simonw/sqlite-utils/issues/87#issuecomment-586661276 | https://api.github.com/repos/simonw/sqlite-utils/issues/87 | MDEyOklzc3VlQ29tbWVudDU4NjY2MTI3Ng== | simonw 9599 | 2020-02-16T02:20:14Z | 2020-02-16T02:20:14Z | OWNER | `OrderedDict` is actually a subclass of `dict` - so a smart fix would be for this logic to check and see if the type `t` is a subclass of one of `list`, `tuple` or `dict`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Should detect collections.OrderedDict as a regular dictionary 565837965 | |
| 586661934 | https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586661934 | https://api.github.com/repos/simonw/sqlite-utils/issues/86 | MDEyOklzc3VlQ29tbWVudDU4NjY2MTkzNA== | simonw 9599 | 2020-02-16T02:33:07Z | 2020-02-16T02:33:07Z | OWNER | Thanks for the example file - looks like it can be trimmed down to just these two lines to replicate the bug: ```csv "MTU (CET)","Day-ahead Price [EUR/MWh]" "01.01.2016 00:00 - 01.01.2016 01:00","23.86" ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Problem with square bracket in CSV column name 564579430 | |
| 586662404 | https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586662404 | https://api.github.com/repos/simonw/sqlite-utils/issues/86 | MDEyOklzc3VlQ29tbWVudDU4NjY2MjQwNA== | simonw 9599 | 2020-02-16T02:43:12Z | 2020-02-16T02:43:12Z | OWNER | https://stackoverflow.com/a/22694438 looks like the answer: > When using square brackets, it is not possible to have these characters in the identifier. > > When using double quotes, you can escape them in the name by doubling them: > > `CREATE TABLE "hello ""world"""(key INTEGER PRIMARY KEY);` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Problem with square bracket in CSV column name 564579430 | |
| 586676640 | https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676640 | https://api.github.com/repos/simonw/sqlite-utils/issues/86 | MDEyOklzc3VlQ29tbWVudDU4NjY3NjY0MA== | simonw 9599 | 2020-02-16T07:16:31Z | 2020-02-16T07:16:31Z | OWNER | There's something weird about this. I created a test database file like so: ``` sqlite3 /tmp/demo.db <<EOF BEGIN TRANSACTION; CREATE TABLE "data" ( "MTU (CET)" TEXT, "Day-ahead Price [EUR/MWh]" TEXT ); INSERT INTO "data" VALUES('01.01.2016 00:00 - 01.01.2016 01:00','23.86'); COMMIT; EOF ``` Then confirmed that it works as expected in SQLite: ``` $ sqlite3 /tmp/demo.db SQLite version 3.24.0 2018-06-04 14:10:15 Enter ".help" for usage hints. sqlite> .schema CREATE TABLE IF NOT EXISTS "data" ( "MTU (CET)" TEXT, "Day-ahead Price [EUR/MWh]" TEXT ); sqlite> .headers on sqlite> select * from data; MTU (CET)|Day-ahead Price [EUR/MWh] 01.01.2016 00:00 - 01.01.2016 01:00|23.86 sqlite> ``` BUT... if I open the same database in Python, something weird happens: ``` In [1]: import sqlite3 In [2]: conn = sqlite3.connect("/tmp/demo.db") In [3]: cursor = conn.cursor() In [4]: cursor.execute("select * from data") Out[4]: <sqlite3.Cursor at 0x10c70a0a0> In [5]: cursor.fetchall() Out[5]: [('01.01.2016 00:00 - 01.01.2016 01:00', '23.86')] In [6]: cursor.description Out[6]: (('MTU (CET)', None, None, None, None, None, None), ('Day-ahead Price', None, None, None, None, None, None)) In [7]: conn.row_factory = sqlite3.Row In [8]: cursor = conn.cursor() … | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Problem with square bracket in CSV column name 564579430 | |
| 586676856 | https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856 | https://api.github.com/repos/simonw/sqlite-utils/issues/86 | MDEyOklzc3VlQ29tbWVudDU4NjY3Njg1Ng== | simonw 9599 | 2020-02-16T07:20:34Z | 2020-02-16T07:20:34Z | OWNER | I'm not sure what to do about this one. I can't fix it: this bug in Python's `sqlite3` module means that even if I write a database out with column names that include `[]` I won't be able to read them back again. So... I could do one of the following: - Throw an error if a column name includes those characters. That's my preferred option I think. - Automatically replace `[` in column names with `(` and `]` with `)` - Do the automatic replacement but show a user-visible warning when I do it - Throw an error, but give the user an option to run with e.g. `--fix-column-names` which applies that automatic fix. Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Problem with square bracket in CSV column name 564579430 | |
| 586683572 | https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586683572 | https://api.github.com/repos/simonw/sqlite-utils/issues/86 | MDEyOklzc3VlQ29tbWVudDU4NjY4MzU3Mg== | foscoj 8149512 | 2020-02-16T09:03:54Z | 2020-02-16T09:03:54Z | NONE | Probably the best option to just throw the error. Is there any active dev chan where we could post the issue to python sqlite3? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Problem with square bracket in CSV column name 564579430 | |
| 586729798 | https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586729798 | https://api.github.com/repos/simonw/sqlite-utils/issues/86 | MDEyOklzc3VlQ29tbWVudDU4NjcyOTc5OA== | simonw 9599 | 2020-02-16T17:11:02Z | 2020-02-16T17:11:02Z | OWNER | I filed a bug in the Python issue tracker here: https://bugs.python.org/issue39652 | {"total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Problem with square bracket in CSV column name 564579430 | |
| 591770623 | https://github.com/simonw/sqlite-utils/issues/86#issuecomment-591770623 | https://api.github.com/repos/simonw/sqlite-utils/issues/86 | MDEyOklzc3VlQ29tbWVudDU5MTc3MDYyMw== | simonw 9599 | 2020-02-27T04:12:39Z | 2020-02-27T04:12:39Z | OWNER | I pushed a branch with my experiment in it, but I'm going to fix this by throwing an error on `[` or `]` in a column name instead - I won't implement the changes from that branch. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Problem with square bracket in CSV column name 564579430 | |
| 584426938 | https://github.com/simonw/sqlite-utils/issues/85#issuecomment-584426938 | https://api.github.com/repos/simonw/sqlite-utils/issues/85 | MDEyOklzc3VlQ29tbWVudDU4NDQyNjkzOA== | simonw 9599 | 2020-02-11T00:35:09Z | 2020-02-11T00:35:09Z | OWNER | Here's why: https://github.com/simonw/sqlite-utils/blob/0c2451e0690c5f4e6463a2f339b0a280e30ed806/sqlite_utils/db.py#L627-L636 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Create index doesn't work for columns containing spaces 562911863 | |
| 583789015 | https://github.com/simonw/sqlite-utils/issues/83#issuecomment-583789015 | https://api.github.com/repos/simonw/sqlite-utils/issues/83 | MDEyOklzc3VlQ29tbWVudDU4Mzc4OTAxNQ== | simonw 9599 | 2020-02-08T23:58:35Z | 2020-02-08T23:58:35Z | OWNER | Shipped as 2.3 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Make db["table"].exists a documented API 559374410 | |
| 581651409 | https://github.com/simonw/sqlite-utils/issues/82#issuecomment-581651409 | https://api.github.com/repos/simonw/sqlite-utils/issues/82 | MDEyOklzc3VlQ29tbWVudDU4MTY1MTQwOQ== | simonw 9599 | 2020-02-03T22:32:41Z | 2020-02-03T22:32:41Z | OWNER | This should work - the data should be chunked automatically. It looks like this is a bug. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tutorial command no longer works 559197745 | |
| 581652388 | https://github.com/simonw/sqlite-utils/issues/82#issuecomment-581652388 | https://api.github.com/repos/simonw/sqlite-utils/issues/82 | MDEyOklzc3VlQ29tbWVudDU4MTY1MjM4OA== | simonw 9599 | 2020-02-03T22:35:44Z | 2020-02-03T22:35:44Z | OWNER | I can't replicate this problem: ``` /tmp $ sqlite-utils --version sqlite-utils, version 2.2 /tmp $ curl "https://data.nasa.gov/resource/y77d-th95.json" | sqlite-utils insert meteorites.db meteorites - --pk=id % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 240k 0 240k 0 0 185k 0 --:--:-- 0:00:01 --:--:-- 185k ``` Could you run `sqlite-utils --version` and tell me what you get? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tutorial command no longer works 559197745 | |
| 591771532 | https://github.com/simonw/sqlite-utils/issues/82#issuecomment-591771532 | https://api.github.com/repos/simonw/sqlite-utils/issues/82 | MDEyOklzc3VlQ29tbWVudDU5MTc3MTUzMg== | simonw 9599 | 2020-02-27T04:16:30Z | 2020-02-27T04:16:30Z | OWNER | Closing as can't reproduce. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tutorial command no longer works 559197745 | |
| 581071010 | https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071010 | https://api.github.com/repos/simonw/sqlite-utils/issues/81 | MDEyOklzc3VlQ29tbWVudDU4MTA3MTAxMA== | simonw 9599 | 2020-02-01T21:27:00Z | 2020-02-01T21:27:00Z | OWNER | Here's the current method: https://github.com/simonw/sqlite-utils/blob/f7289174e66ae4d91d57de94bbd9d09fabf7aff4/sqlite_utils/db.py#L823-L845 If I make it a utility function instead of a class method I could ensure it is directly importable like so: ```python from sqlite_utils import detect_column_types ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove .detect_column_types() from table, make it a documented API 558600274 | |
| 581071116 | https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071116 | https://api.github.com/repos/simonw/sqlite-utils/issues/81 | MDEyOklzc3VlQ29tbWVudDU4MTA3MTExNg== | simonw 9599 | 2020-02-01T21:28:35Z | 2020-02-01T21:28:53Z | OWNER | Should I keep `table.detect_column_types()` working so as not to break existing code? If it was part of the documented API then I wouldn't break that without bumping to 3.x. Since it's undocumented I'm going to make it as a breaking change instead (and bump the `geojson-to-sqlite` dependency version). | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove .detect_column_types() from table, make it a documented API 558600274 | |
| 581071235 | https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071235 | https://api.github.com/repos/simonw/sqlite-utils/issues/81 | MDEyOklzc3VlQ29tbWVudDU4MTA3MTIzNQ== | simonw 9599 | 2020-02-01T21:30:09Z | 2020-02-01T21:30:09Z | OWNER | Actually I'll put it in the `utils.py` module. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove .detect_column_types() from table, make it a documented API 558600274 | |
| 581071434 | https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071434 | https://api.github.com/repos/simonw/sqlite-utils/issues/81 | MDEyOklzc3VlQ29tbWVudDU4MTA3MTQzNA== | simonw 9599 | 2020-02-01T21:32:34Z | 2020-02-01T21:32:34Z | OWNER | While I'm at it I think I'll rename it to `suggest_column_types` - it's not really detecting them since the input is just a list of dictionaries. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Remove .detect_column_types() from table, make it a documented API 558600274 | |
| 580567505 | https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567505 | https://api.github.com/repos/simonw/sqlite-utils/issues/80 | MDEyOklzc3VlQ29tbWVudDU4MDU2NzUwNQ== | simonw 9599 | 2020-01-31T03:39:19Z | 2020-01-31T03:39:19Z | OWNER | Still needs documentation and tests. Also I'm not certain that this should be an argument you can pass to the `.table()` constructor, need to think that over. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | on_create mechanism for after table creation 557892819 | |
| 580567604 | https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567604 | https://api.github.com/repos/simonw/sqlite-utils/issues/80 | MDEyOklzc3VlQ29tbWVudDU4MDU2NzYwNA== | simonw 9599 | 2020-01-31T03:39:58Z | 2020-01-31T03:39:58Z | OWNER | Perhaps this should be called `after_create` instead of `on_create`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | on_create mechanism for after table creation 557892819 | |
| 580567886 | https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567886 | https://api.github.com/repos/simonw/sqlite-utils/issues/80 | MDEyOklzc3VlQ29tbWVudDU4MDU2Nzg4Ng== | simonw 9599 | 2020-01-31T03:41:31Z | 2020-01-31T03:41:31Z | OWNER | I think it does make sense to be able to pass it to the `.table()` constructor. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | on_create mechanism for after table creation 557892819 | |
| 580569059 | https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580569059 | https://api.github.com/repos/simonw/sqlite-utils/issues/80 | MDEyOklzc3VlQ29tbWVudDU4MDU2OTA1OQ== | simonw 9599 | 2020-01-31T03:48:41Z | 2020-01-31T03:48:41Z | OWNER | This may not be the right feature after all, see https://github.com/simonw/geojson-to-sqlite/issues/6#issuecomment-580569002 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | on_create mechanism for after table creation 557892819 | |
| 580584269 | https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580584269 | https://api.github.com/repos/simonw/sqlite-utils/issues/80 | MDEyOklzc3VlQ29tbWVudDU4MDU4NDI2OQ== | simonw 9599 | 2020-01-31T05:08:04Z | 2020-01-31T05:08:04Z | OWNER | Ditching this since it won't actually solve my problem. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | on_create mechanism for after table creation 557892819 | |
| 464341721 | https://github.com/simonw/sqlite-utils/issues/8#issuecomment-464341721 | https://api.github.com/repos/simonw/sqlite-utils/issues/8 | MDEyOklzc3VlQ29tbWVudDQ2NDM0MTcyMQ== | psychemedia 82988 | 2019-02-16T12:08:41Z | 2019-02-16T12:08:41Z | NONE | We also get an error if a column name contains a `.` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Problems handling column names containing spaces or - 403922644 | |
| 466695500 | https://github.com/simonw/sqlite-utils/issues/8#issuecomment-466695500 | https://api.github.com/repos/simonw/sqlite-utils/issues/8 | MDEyOklzc3VlQ29tbWVudDQ2NjY5NTUwMA== | simonw 9599 | 2019-02-23T21:09:03Z | 2019-02-23T21:09:03Z | OWNER | Fixed in https://github.com/simonw/sqlite-utils/commit/228d595f7d10994f34e948888093c2cd290267c4 <img width="745" alt="screen shot 2019-02-23 at 1 07 14 pm" src="https://user-images.githubusercontent.com/9599/53291690-2d72ba80-376c-11e9-99b8-074045d6290d.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Problems handling column names containing spaces or - 403922644 | |
| 482994231 | https://github.com/simonw/sqlite-utils/issues/8#issuecomment-482994231 | https://api.github.com/repos/simonw/sqlite-utils/issues/8 | MDEyOklzc3VlQ29tbWVudDQ4Mjk5NDIzMQ== | psychemedia 82988 | 2019-04-14T15:04:07Z | 2019-04-14T15:29:33Z | NONE | PLEASE IGNORE THE BELOW... I did a package update and rebuilt the kernel I was working in... may just have been an old version of sqlite_utils, seems to be working now. (Too many containers / too many environments!) Has an issue been reintroduced here with FTS? eg I'm getting an error thrown by spaces in column names here: ``` /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) def enable_fts(self, columns, fts_version="FTS5"): --> 329 "Enables FTS on the specified columns" 330 sql = """ 331 CREATE VIRTUAL TABLE "{table}_fts" USING {fts_version} ( ``` when trying an `insert_all`. Also, if a col has a `.` in it, I seem to get: ``` /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) 327 jsonify_if_needed(record.get(key, None)) for key in all_columns 328 ) --> 329 result = self.db.conn.execute(sql, values) 330 self.db.conn.commit() 331 self.last_id = result.lastrowid OperationalError: near ".": syntax error ``` (Can't post a worked minimal example right now; racing trying to build something against a live timing screen that will stop until next weekend in an hour or two...) PS Hmmm I did a test and they seem to work; I must be messing up s/where else... ``` import sqlite3 from sqlite_utils import Database dbname='testingDB_sqlite_utils.db' #!rm $dbname conn = sqlite3.connect(dbname, timeout=10) #Setup database tables c = conn.cursor() setup=''' CREATE TABLE IF NOT EXISTS "test1" ( "NO" INTEGER, "NAME" TEXT ); CREATE TABLE IF NOT EXISTS "test2" ( "NO" INTEGER, `TIME OF DAY` TEXT ); CREATE TABLE IF NOT EXISTS "test3" ( "NO" INTEGER, `AVG. SPEED (MPH)` FLOAT ); ''' c.executescript(setup) DB = Database(conn) … | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Problems handling column names containing spaces or - 403922644 | |
| 1012158895 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012158895 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM48VFGv | eyeseast 25778 | 2022-01-13T13:55:59Z | 2022-01-13T13:55:59Z | CONTRIBUTOR | Came here to add this. I might pick it up. Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Helper methods for working with SpatiaLite 557842245 | |
| 1012230212 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012230212 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM48VWhE | eyeseast 25778 | 2022-01-13T15:15:13Z | 2022-01-13T15:15:13Z | CONTRIBUTOR | Some proposals I'd add to sqlite-utils: Some version of this, from [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite/blob/main/geojson_to_sqlite/utils.py#L124-L130): ```python def init_spatialite(db, lib): db.conn.enable_load_extension(True) db.conn.load_extension(lib) # Initialize SpatiaLite if not yet initialized if "spatial_ref_sys" in db.table_names(): return db.conn.execute("select InitSpatialMetadata(1)") ``` Also a function for creating a spatial index: ```python db.conn.execute("select CreateSpatialIndex(?, ?)", [table, "geometry"]) ``` I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like: ```sh sqlite-utils spatial-index spatial.db table-name column-name ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Helper methods for working with SpatiaLite 557842245 | |
| 1012253198 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012253198 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM48VcIO | eyeseast 25778 | 2022-01-13T15:39:14Z | 2022-01-13T15:39:14Z | CONTRIBUTOR | Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (`gis.py` or something) just so it's easier to find later. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Helper methods for working with SpatiaLite 557842245 | |
| 1012413729 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012413729 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM48WDUh | eyeseast 25778 | 2022-01-13T18:50:00Z | 2022-01-13T18:50:00Z | CONTRIBUTOR | One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Helper methods for working with SpatiaLite 557842245 | |
| 1013698557 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1013698557 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM48a8_9 | eyeseast 25778 | 2022-01-15T15:15:22Z | 2022-01-15T15:15:22Z | CONTRIBUTOR | @simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Helper methods for working with SpatiaLite 557842245 | |
| 1029317527 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029317527 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM49WiOX | eyeseast 25778 | 2022-02-03T19:18:02Z | 2022-02-03T19:18:02Z | CONTRIBUTOR | Taking part of the conversation from #385 here. > Would sqlite-utils add-geometry-column ... be a good CLI enhancement. for example? Yes. And also `sqlite-utils create-spatial-index` would be great to have. My plan would be to add those once the Python API is settled. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Helper methods for working with SpatiaLite 557842245 | |
| 1029683977 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029683977 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM49X7sJ | simonw 9599 | 2022-02-04T05:58:15Z | 2022-02-04T05:58:15Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#spatialite-helpers | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Helper methods for working with SpatiaLite 557842245 | |
| 1029703503 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029703503 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM49YAdP | simonw 9599 | 2022-02-04T06:46:32Z | 2022-02-04T06:46:32Z | OWNER | Shipped in 3.23: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-23 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Helper methods for working with SpatiaLite 557842245 | |
| 580515506 | https://github.com/simonw/sqlite-utils/issues/77#issuecomment-580515506 | https://api.github.com/repos/simonw/sqlite-utils/issues/77 | MDEyOklzc3VlQ29tbWVudDU4MDUxNTUwNg== | simonw 9599 | 2020-01-30T23:48:41Z | 2020-01-30T23:48:41Z | OWNER | Potential design: a `conversions={}` option. Used like this: ```python db[table].insert(record, conversions={"geom": "GeomFromText(?, 4326)"}) ``` The `conversions=` key would be supported on `.insert()`, `.insert_all()`, `.upsert()` etc. It could also be passed to the `db.table()` constructor function: ```python table = db.table( "features", pk="id", conversions={ "geom": "GeomFromText(?, 4326)" } ) # Then used like this: table.insert(record) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to insert data that is transformed by a SQL function 557825032 | |
| 580527238 | https://github.com/simonw/sqlite-utils/issues/77#issuecomment-580527238 | https://api.github.com/repos/simonw/sqlite-utils/issues/77 | MDEyOklzc3VlQ29tbWVudDU4MDUyNzIzOA== | simonw 9599 | 2020-01-31T00:34:02Z | 2020-01-31T00:34:02Z | OWNER | Documentation: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#python-api-conversions | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Ability to insert data that is transformed by a SQL function 557825032 | |
| 614354219 | https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614354219 | https://api.github.com/repos/simonw/sqlite-utils/issues/76 | MDEyOklzc3VlQ29tbWVudDYxNDM1NDIxOQ== | simonw 9599 | 2020-04-16T01:01:34Z | 2020-04-16T01:01:34Z | OWNER | I think a neat way to do this would be with an optional argument for `.rows_where()`: ```python rows = db["table"].rows_where("age > 10", order_by="age desc") ``` If you want everything you can use this: ```python rows = db["table"].rows_where(order_by="age desc") ``` It's a tiny bit weird calling `.rows_where()` without a where clause, but I think it makes sense here - especially since `.rows` is a property that can't take any arguments - though under the hood it actually does this: https://github.com/simonw/sqlite-utils/blob/ad6ac19470a67867b96cb4c086450b8e4e46bf02/sqlite_utils/db.py#L436-L443 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | order_by mechanism 549287310 | |
| 614400454 | https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614400454 | https://api.github.com/repos/simonw/sqlite-utils/issues/76 | MDEyOklzc3VlQ29tbWVudDYxNDQwMDQ1NA== | simonw 9599 | 2020-04-16T03:51:01Z | 2020-04-16T03:51:01Z | OWNER | Released in 2.6 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | order_by mechanism 549287310 | |
| 614400533 | https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614400533 | https://api.github.com/repos/simonw/sqlite-utils/issues/76 | MDEyOklzc3VlQ29tbWVudDYxNDQwMDUzMw== | simonw 9599 | 2020-04-16T03:51:26Z | 2020-04-16T03:51:26Z | OWNER | Documentation here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#listing-rows | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | order_by mechanism 549287310 | |
| 614440032 | https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614440032 | https://api.github.com/repos/simonw/sqlite-utils/issues/76 | MDEyOklzc3VlQ29tbWVudDYxNDQ0MDAzMg== | metab0t 10501166 | 2020-04-16T06:23:29Z | 2020-04-16T06:23:29Z | NONE | Thanks for your hard work! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | order_by mechanism 549287310 | |
| 580523995 | https://github.com/simonw/sqlite-utils/pull/75#issuecomment-580523995 | https://api.github.com/repos/simonw/sqlite-utils/issues/75 | MDEyOklzc3VlQ29tbWVudDU4MDUyMzk5NQ== | simonw 9599 | 2020-01-31T00:21:11Z | 2020-01-31T00:21:11Z | OWNER | This makes sense, thanks! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Explicitly include tests and docs in sdist 546078359 | |
| 572871797 | https://github.com/simonw/sqlite-utils/issues/74#issuecomment-572871797 | https://api.github.com/repos/simonw/sqlite-utils/issues/74 | MDEyOklzc3VlQ29tbWVudDU3Mjg3MTc5Nw== | simonw 9599 | 2020-01-10T04:47:55Z | 2020-01-10T04:47:55Z | OWNER | This is odd. I'd love to see more about that result object. Could you try running `pytest --pdb` and then `result.exit_code, result.exception` in the PDB prompt, something like this? ``` $ pytest --pdb ========================================================= test session starts ========================================================= platform darwin -- Python 3.7.4, pytest-5.2.2, py-1.8.0, pluggy-0.13.0 rootdir: /Users/simonw/Dropbox/Development/sqlite-utils plugins: cov-2.8.1 collected 216 items tests/test_black.py s [ 0%] tests/test_cli.py F >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> db_path = '/private/var/folders/bl/5x847xbj2yb7xmp7f2tz7l280000gn/T/pytest-of-simonw/pytest-3/test_tables0/test.db' def test_tables(db_path): result = CliRunner().invoke(cli.cli, ["tables1", db_path]) > assert '[{"table": "Gosh"},\n {"table": "Gosh2"}]' == result.output.strip() E assert '[{"table": "...e": "Gosh2"}]' == '' E - [{"table": "Gosh"}, E - {"table": "Gosh2"}] tests/test_cli.py:28: AssertionError >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > /Users/simonw/Dropbox/Development/sqlite-utils/tests/test_cli.py(28)test_tables() -> assert '[{"table": "Gosh"},\n {"table": "Gosh2"}]' == result.output.strip() (Pdb) result.exit_code, result.exception (1, OperationalError('near "/": syntax error')) ``` That should show the exception that caused the script to fail to run. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column 546073980 | |
| 573388052 | https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573388052 | https://api.github.com/repos/simonw/sqlite-utils/issues/74 | MDEyOklzc3VlQ29tbWVudDU3MzM4ODA1Mg== | jayvdb 15092 | 2020-01-12T06:51:30Z | 2020-01-12T06:51:30Z | CONTRIBUTOR | Thanks. That showed me that there was a click cli runner error, and setting `export LANG=en_US.UTF-8` fixed it. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column 546073980 | |
| 573389669 | https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573389669 | https://api.github.com/repos/simonw/sqlite-utils/issues/74 | MDEyOklzc3VlQ29tbWVudDU3MzM4OTY2OQ== | jayvdb 15092 | 2020-01-12T07:21:17Z | 2020-01-12T07:21:17Z | CONTRIBUTOR | I guess there is some extra flag for ` CliRunner.invoke` to check exitcode and raise the exception, or that should be an extra assert added. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column 546073980 | |
| 570930239 | https://github.com/simonw/sqlite-utils/issues/73#issuecomment-570930239 | https://api.github.com/repos/simonw/sqlite-utils/issues/73 | MDEyOklzc3VlQ29tbWVudDU3MDkzMDIzOQ== | simonw 9599 | 2020-01-05T17:15:18Z | 2020-01-05T17:15:18Z | OWNER | I think this is because you forgot to include a `pk=` argument. I'll change the code to throw a more useful error in this case. | {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | upsert_all() throws issue when upserting to empty table 545407916 | |
| 570931650 | https://github.com/simonw/sqlite-utils/issues/73#issuecomment-570931650 | https://api.github.com/repos/simonw/sqlite-utils/issues/73 | MDEyOklzc3VlQ29tbWVudDU3MDkzMTY1MA== | simonw 9599 | 2020-01-05T17:34:33Z | 2020-01-05T17:34:33Z | OWNER | Released as 2.0.1 https://github.com/simonw/sqlite-utils/releases/tag/2.0.1 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | upsert_all() throws issue when upserting to empty table 545407916 | |
| 571138093 | https://github.com/simonw/sqlite-utils/issues/73#issuecomment-571138093 | https://api.github.com/repos/simonw/sqlite-utils/issues/73 | MDEyOklzc3VlQ29tbWVudDU3MTEzODA5Mw== | psychemedia 82988 | 2020-01-06T13:28:31Z | 2020-01-06T13:28:31Z | NONE | I think I actually had several issues in play... The missing key was one, but I think there is also an issue as per below. For example, in the following: ```python def init_testdb(dbname='test.db'): if os.path.exists(dbname): os.remove(dbname) conn = sqlite3.connect(dbname) db = Database(conn) return conn, db conn, db = init_testdb() c = conn.cursor() c.executescript('CREATE TABLE "test1" ("Col1" TEXT, "Col2" TEXT, PRIMARY KEY ("Col1"));') c.executescript('CREATE TABLE "test2" ("Col1" TEXT, "Col2" TEXT, PRIMARY KEY ("Col1"));') print('Test 1...') for i in range(3): db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) print('Test 2...') for i in range(3): db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}, {'Col1':'c','Col2':'x'}], pk=('Col1')) print('Done...') --------------------------------------------------------------------------- Test 1... Test 2... IndexError: list index out of range --------------------------------------------------------------------------- IndexError Traceback (most recent call last) <ipython-input-763-444132ca189f> in <module> 22 print('Test 2...') 23 for i in range(3): ---> 24 db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) 25 db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}, 26 {'Col1':'c','Col2':'x'}], pk=('Col1')) /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts) 1157 alter=alter, 1158 extracts=extracts, -> 1… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | upsert_all() throws issue when upserting to empty table 545407916 | |
| 572870032 | https://github.com/simonw/sqlite-utils/issues/73#issuecomment-572870032 | https://api.github.com/repos/simonw/sqlite-utils/issues/73 | MDEyOklzc3VlQ29tbWVudDU3Mjg3MDAzMg== | simonw 9599 | 2020-01-10T04:38:41Z | 2020-01-10T04:38:41Z | OWNER | Odd.. I'm not able to replicate that error. Here's what I got: <img width="869" alt="Tony_Bug" src="https://user-images.githubusercontent.com/9599/72126164-025f6e00-3320-11ea-9737-c0679950acbd.png"> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | upsert_all() throws issue when upserting to empty table 545407916 | |
| 573047321 | https://github.com/simonw/sqlite-utils/issues/73#issuecomment-573047321 | https://api.github.com/repos/simonw/sqlite-utils/issues/73 | MDEyOklzc3VlQ29tbWVudDU3MzA0NzMyMQ== | psychemedia 82988 | 2020-01-10T14:02:56Z | 2020-01-10T14:09:23Z | NONE | Hmmm... just tried with installs from pip and the repo (v2.0.0 and v2.0.1) and I get the error each time (start of second run through the second loop). Could it be sqlite3? I'm on 3.30.1. UPDATE: just tried it on jupyter.org/try and I get the error there, too. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | upsert_all() throws issue when upserting to empty table 545407916 | |
| 580745213 | https://github.com/simonw/sqlite-utils/issues/73#issuecomment-580745213 | https://api.github.com/repos/simonw/sqlite-utils/issues/73 | MDEyOklzc3VlQ29tbWVudDU4MDc0NTIxMw== | psychemedia 82988 | 2020-01-31T14:02:38Z | 2020-01-31T14:21:09Z | NONE | So the conundrum continues.. The simple test case above now runs, but if I upsert a large number of new records (successfully) and then try to upsert a fewer number of new records to a different table, I get the same error. If I run the same upserts again (which in the first case means there are no new records to add, because they were already added), the second upsert works correctly. It feels as if the number of items added via an upsert >> the number of items I try to add in an upsert immediately after, I get the error. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | upsert_all() throws issue when upserting to empty table 545407916 | |
| 569233996 | https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569233996 | https://api.github.com/repos/simonw/sqlite-utils/issues/71 | MDEyOklzc3VlQ29tbWVudDU2OTIzMzk5Ng== | simonw 9599 | 2019-12-27T09:45:17Z | 2019-12-27T09:45:17Z | OWNER | It looks like those backports no longer include sqlite3 - Google Searches still find it but when you click through to launchpad you get 404s: https://launchpad.net/~jonathonf/+archive/ubuntu/codelite/+build/10511920 Maybe Travis have a newer Ubuntu I can use that ships with FTS5 in its SQLite? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tests are failing due to missing FTS5 542814756 | |
| 569234096 | https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569234096 | https://api.github.com/repos/simonw/sqlite-utils/issues/71 | MDEyOklzc3VlQ29tbWVudDU2OTIzNDA5Ng== | simonw 9599 | 2019-12-27T09:45:52Z | 2019-12-27T09:45:52Z | OWNER | I'll try `bionic`: https://docs.travis-ci.com/user/reference/bionic/ | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tests are failing due to missing FTS5 542814756 | |
| 569234571 | https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569234571 | https://api.github.com/repos/simonw/sqlite-utils/issues/71 | MDEyOklzc3VlQ29tbWVudDU2OTIzNDU3MQ== | simonw 9599 | 2019-12-27T09:48:48Z | 2019-12-27T09:48:48Z | OWNER | That fixed it: https://travis-ci.com/simonw/sqlite-utils/builds/142443259 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Tests are failing due to missing FTS5 542814756 | |
| 569130037 | https://github.com/simonw/sqlite-utils/issues/70#issuecomment-569130037 | https://api.github.com/repos/simonw/sqlite-utils/issues/70 | MDEyOklzc3VlQ29tbWVudDU2OTEzMDAzNw== | simonw 9599 | 2019-12-26T20:39:04Z | 2019-12-26T20:39:04Z | OWNER | I hadn't thought about those at all. Are you suggesting a utility mechanism in the library for setting it up so that, for a specific foreign key, rows are deleted from other tables if the row they are pointing at is deleted? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Implement ON DELETE and ON UPDATE actions for foreign keys 539204432 | |
| 575799104 | https://github.com/simonw/sqlite-utils/issues/70#issuecomment-575799104 | https://api.github.com/repos/simonw/sqlite-utils/issues/70 | MDEyOklzc3VlQ29tbWVudDU3NTc5OTEwNA== | LucasElArruda 26292069 | 2020-01-17T21:20:17Z | 2020-01-17T21:20:17Z | NONE | Omg sorry I took so long to reply! On SQL we can say how the foreign key behaves when it is deleted or updated on the parent table (see https://www.sqlitetutorial.net/sqlite-foreign-key/ for more details). I did not see clearly how to create tables with this feature on sqlite-utils library. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Implement ON DELETE and ON UPDATE actions for foreign keys 539204432 | |
| 457980966 | https://github.com/simonw/sqlite-utils/issues/7#issuecomment-457980966 | https://api.github.com/repos/simonw/sqlite-utils/issues/7 | MDEyOklzc3VlQ29tbWVudDQ1Nzk4MDk2Ng== | simonw 9599 | 2019-01-28T02:29:32Z | 2019-01-28T02:29:32Z | OWNER | Remember to remove this TODO (and turn the `[]` into `()` on this line) as part of this task: https://github.com/simonw/sqlite-utils/blob/5309c5c7755818323a0f5353bad0de98ecc866be/sqlite_utils/cli.py#L78-L80 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .insert_all() should accept a generator and process it efficiently 403625674 | |
| 458011885 | https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011885 | https://api.github.com/repos/simonw/sqlite-utils/issues/7 | MDEyOklzc3VlQ29tbWVudDQ1ODAxMTg4NQ== | simonw 9599 | 2019-01-28T06:25:48Z | 2019-01-28T06:25:48Z | OWNER | Re-opening for the second bit involving the cli tool. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .insert_all() should accept a generator and process it efficiently 403625674 | |
| 458011906 | https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011906 | https://api.github.com/repos/simonw/sqlite-utils/issues/7 | MDEyOklzc3VlQ29tbWVudDQ1ODAxMTkwNg== | simonw 9599 | 2019-01-28T06:25:55Z | 2019-01-28T06:25:55Z | OWNER | I tested this with a script called `churn_em_out.py` ``` i = 0 while True: i += 1 print( '{"id": I, "another": "row", "number": J}'.replace("I", str(i)).replace( "J", str(i + 1) ) ) ``` Then I ran this: ``` python churn_em_out.py | \ sqlite-utils insert /tmp/getbig.db stats - \ --nl --batch-size=10000 ``` And used `watch 'ls -lah /tmp/getbig.db'` to watch the file growing as it had 10,000 lines of junk committed in batches. The memory used by the process never grew about around 50MB. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | .insert_all() should accept a generator and process it efficiently 403625674 | |
| 569130196 | https://github.com/simonw/sqlite-utils/issues/69#issuecomment-569130196 | https://api.github.com/repos/simonw/sqlite-utils/issues/69 | MDEyOklzc3VlQ29tbWVudDU2OTEzMDE5Ng== | simonw 9599 | 2019-12-26T20:40:21Z | 2019-12-26T20:40:21Z | OWNER | This is a good idea. Datasette has this in the form of the `--load-extension` CLI argument, e.g. for SpatiaLite here: https://datasette.readthedocs.io/en/stable/spatialite.html#installation Having that available for `sqlite-utils` definitely makes sense. | {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1} | Feature request: enable extensions loading 534507142 | |
| 710405658 | https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710405658 | https://api.github.com/repos/simonw/sqlite-utils/issues/69 | MDEyOklzc3VlQ29tbWVudDcxMDQwNTY1OA== | simonw 9599 | 2020-10-16T18:42:48Z | 2020-10-16T18:42:48Z | OWNER | Did some work on this for #134 in 7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d I still need to add `--load-extension` to other CLI methods, see #137. Closing this issue in favour of that one. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature request: enable extensions loading 534507142 | |
| 710768396 | https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710768396 | https://api.github.com/repos/simonw/sqlite-utils/issues/69 | MDEyOklzc3VlQ29tbWVudDcxMDc2ODM5Ng== | aborruso 30607 | 2020-10-17T07:46:59Z | 2020-10-17T07:46:59Z | NONE | Great @simonw thank you very much | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Feature request: enable extensions loading 534507142 | |
| 695695776 | https://github.com/simonw/sqlite-utils/issues/68#issuecomment-695695776 | https://api.github.com/repos/simonw/sqlite-utils/issues/68 | MDEyOklzc3VlQ29tbWVudDY5NTY5NTc3Ng== | simonw 9599 | 2020-09-20T04:25:47Z | 2020-09-20T04:25:47Z | OWNER | This is a dupe of #130 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Add support for porter stemming in FTS 531583658 | |
| 559108591 | https://github.com/simonw/sqlite-utils/pull/67#issuecomment-559108591 | https://api.github.com/repos/simonw/sqlite-utils/issues/67 | MDEyOklzc3VlQ29tbWVudDU1OTEwODU5MQ== | simonw 9599 | 2019-11-27T14:24:59Z | 2019-11-27T14:24:59Z | OWNER | Failed due to black testing dependency: https://travis-ci.com/simonw/sqlite-utils/jobs/260995814 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Run tests against 3.5 too 529376481 | |
| 569844320 | https://github.com/simonw/sqlite-utils/pull/67#issuecomment-569844320 | https://api.github.com/repos/simonw/sqlite-utils/issues/67 | MDEyOklzc3VlQ29tbWVudDU2OTg0NDMyMA== | simonw 9599 | 2019-12-31T01:29:43Z | 2019-12-31T01:29:43Z | OWNER | I don't really care about 3.5 any more. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Run tests against 3.5 too 529376481 | |
| 553170650 | https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553170650 | https://api.github.com/repos/simonw/sqlite-utils/issues/66 | MDEyOklzc3VlQ29tbWVudDU1MzE3MDY1MA== | simonw 9599 | 2019-11-12T23:49:29Z | 2019-11-12T23:49:29Z | OWNER | This relates to this bug: https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549233778 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | The ".upsert()" method is misnamed 521868864 | |
| 553171011 | https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553171011 | https://api.github.com/repos/simonw/sqlite-utils/issues/66 | MDEyOklzc3VlQ29tbWVudDU1MzE3MTAxMQ== | simonw 9599 | 2019-11-12T23:50:52Z | 2019-11-12T23:50:52Z | OWNER | Fixing this is going to be a real pain. There's lots of code out there that uses `sqlite-utils` with the expectation that `upsert()` behaves as it currently does. Maybe I need to introduce new terms for both of these different patterns and deprecate the existing `.upsert()` and `.upsert_all()` since their behaviour can't be changed? Or maybe I fix this and ship `sqlite-utils 2.0` with a breaking change? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | The ".upsert()" method is misnamed 521868864 | |
| 553171414 | https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553171414 | https://api.github.com/repos/simonw/sqlite-utils/issues/66 | MDEyOklzc3VlQ29tbWVudDU1MzE3MTQxNA== | simonw 9599 | 2019-11-12T23:52:35Z | 2019-11-12T23:52:35Z | OWNER | If I do implement the correct definition of `.upsert()` I think I'll use this pattern, since it works in versions of SQLite prior to 3.24: ```sql INSERT OR IGNORE INTO book(id) VALUES(1001); UPDATE book SET name = 'Programming' WHERE id = 1001; ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | The ".upsert()" method is misnamed 521868864 | |
| 553526685 | https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553526685 | https://api.github.com/repos/simonw/sqlite-utils/issues/66 | MDEyOklzc3VlQ29tbWVudDU1MzUyNjY4NQ== | simonw 9599 | 2019-11-13T17:58:59Z | 2019-11-13T17:58:59Z | OWNER | This warrants making a backwards compatible change, which means I'll need to bump the major version number and release 2.0. I'm going to rename the existing `upsert()` and `upsert_all()` methods to `replace()` and `replace_all()` - then write new `upsert()` and `upsert_all()` methods that implement the correct behavior. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | The ".upsert()" method is misnamed 521868864 | |
| 553527384 | https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553527384 | https://api.github.com/repos/simonw/sqlite-utils/issues/66 | MDEyOklzc3VlQ29tbWVudDU1MzUyNzM4NA== | simonw 9599 | 2019-11-13T18:00:41Z | 2019-11-13T18:00:41Z | OWNER | Is `replace()` a good name here? It doesn't really convey the idea that a brand new record will be created if there isn't an existing one to replace. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | The ".upsert()" method is misnamed 521868864 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] (
[html_url] TEXT,
[issue_url] TEXT,
[id] INTEGER PRIMARY KEY,
[node_id] TEXT,
[user] INTEGER REFERENCES [users]([id]),
[created_at] TEXT,
[updated_at] TEXT,
[author_association] TEXT,
[body] TEXT,
[reactions] TEXT,
[issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
ON [issue_comments] ([user]);
author_association 4 ✖