html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098288158,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098288158,IC_kwDOCGYnMM5Bdowe,9599,simonw,2022-04-13T17:07:53Z,2022-04-13T17:07:53Z,OWNER,"I can't replicate the bug I'm afraid:
```
% wget ""https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true""
...
2022-04-13 10:06:29 (8.97 MB/s) - ‘global_power_plant_database.csv?raw=true’ saved [8856038/8856038]
% sqlite-utils insert global.db power_plants \
'global_power_plant_database.csv?raw=true' --csv
[------------------------------------] 0%
[###################################-] 99% 00:00:00%
% sqlite-utils indexes global.db --table
table index_name seqno cid name desc coll key
------- ------------ ------- ----- ------ ------ ------ -----
% sqlite-utils extract global.db power_plants country country_long \
--table countries \
--fk-column country_id \
--rename country_long name
% sqlite-utils indexes global.db --table
table index_name seqno cid name desc coll key
--------- -------------------------- ------- ----- ------- ------ ------ -----
countries idx_countries_country_name 0 1 country 0 BINARY 1
countries idx_countries_country_name 1 2 name 0 BINARY 1
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098295517,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098295517,IC_kwDOCGYnMM5Bdqjd,9599,simonw,2022-04-13T17:16:20Z,2022-04-13T17:16:20Z,OWNER,"Aha! I was able to replicate the bug using your `Dockerfile` - thanks very much for providing that.
```
(app-root) sqlite-utils indexes global.db --table
Error: near ""("": syntax error
```
(That wa sbefore I even ran the `extract` command.)
To build your `Dockerfile` I copied it into an empty folder and ran the following:
```
wget https://www.sqlite.org/2021/sqlite-autoconf-3360000.tar.gz
docker build . -t centos-sqlite-utils
docker run -it centos-sqlite-utils /bin/bash
```
This gave me a shell in which I could replicate the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098531354,IC_kwDOCGYnMM5BekIa,9599,simonw,2022-04-13T22:08:20Z,2022-04-13T22:08:20Z,OWNER,"OK I figured out what's going on here. First I added an extra `print(sql)` statement to the `indexes` command to see what SQL it was running:
```
(app-root) sqlite-utils indexes global.db --table
select
sqlite_master.name as ""table"",
indexes.name as index_name,
xinfo.*
from sqlite_master
join pragma_index_list(sqlite_master.name) indexes
join pragma_index_xinfo(index_name) xinfo
where
sqlite_master.type = 'table'
and xinfo.key = 1
Error: near ""("": syntax error
```
This made me suspicious that the SQLite version being used here didn't support joining against the `pragma_index_list(...)` table-valued functions in that way. So I checked the version:
```
(app-root) sqlite3
SQLite version 3.36.0 2021-06-18 18:36:39
```
That version should be fine - it's the one you compiled in the Dockerfile.
Then I checked the version that `sqlite-utils` itself was using:
```
(app-root) sqlite-utils memory 'select sqlite_version()'
[{""sqlite_version()"": ""3.7.17""}]
```
It's running SQLite 3.7.17!
So the problem here is that the Python in that Docker image is running a very old version of SQLite.
I tried using the trick in https://til.simonwillison.net/sqlite/ld-preload as a workaround, and it almost worked:
```
(app-root) python3 -c 'import sqlite3; print(sqlite3.connect("":memory"").execute(""select sqlite_version()"").fetchone())'
('3.7.17',)
(app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so python3 -c 'import sqlite3; print(sqlite3.connect("":memory"").execute(""select sqlite_version()"").fetchone())'
('3.36.0',)
```
But when I try to run `sqlite-utils` like that I get an error:
```
(app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db
...
File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py"", line 1624, in query
db.register_fts4_bm25()
File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 412, in register_fts4_bm25
self.register_function(rank_bm25, deterministic=True)
File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 408, in register_function
register(fn)
File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 401, in register
self.conn.create_function(name, arity, fn, **kwargs)
sqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098532220,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098532220,IC_kwDOCGYnMM5BekV8,9599,simonw,2022-04-13T22:09:52Z,2022-04-13T22:09:52Z,OWNER,That error is weird - it's not supposed to happen according to this code here: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L400,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098535531,IC_kwDOCGYnMM5BelJr,9599,simonw,2022-04-13T22:15:48Z,2022-04-13T22:15:48Z,OWNER,"Trying this alternative implementation of the `register()` method:
```python
def register(fn):
name = fn.__name__
arity = len(inspect.signature(fn).parameters)
if not replace and (name, arity) in self._registered_functions:
return fn
kwargs = {}
done = False
if deterministic:
# Try this, but fall back if sqlite3.NotSupportedError
try:
self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True))
done = True
except sqlite3.NotSupportedError:
pass
if not done:
self.conn.create_function(name, arity, fn, **kwargs)
self._registered_functions.add((name, arity))
return fn
```
With that fix, the following worked!
```
LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db --table
table index_name seqno cid name desc coll key
--------- -------------------------- ------- ----- ------- ------ ------ -----
countries idx_countries_country_name 0 1 country 0 BINARY 1
countries idx_countries_country_name 1 2 name 0 BINARY 1
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098548931,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098548931,IC_kwDOCGYnMM5BeobD,9599,simonw,2022-04-13T22:41:59Z,2022-04-13T22:41:59Z,OWNER,"I'm going to close this ticket since it looks like this is a bug in the way the Dockerfile builds Python, but I'm going to ship a fix for that issue I found so the `LD_PRELOAD` workaround above should work OK with the next release of `sqlite-utils`. Thanks for the detailed bug report!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/424#issuecomment-1098548090,https://api.github.com/repos/simonw/sqlite-utils/issues/424,1098548090,IC_kwDOCGYnMM5BeoN6,9599,simonw,2022-04-13T22:40:15Z,2022-04-13T22:40:15Z,OWNER,"New error:
```pycon
>>> from sqlite_utils import Database
>>> db = Database(memory=True)
>>> db[""foo""].create({})
Traceback (most recent call last):
File """", line 1, in
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1465, in create
self.db.create_table(
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 885, in create_table
sql = self.create_table_sql(
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 771, in create_table_sql
assert columns, ""Tables must have at least one column""
AssertionError: Tables must have at least one column
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200866134,Better error message if you try to create a table with no columns,
https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1098537000,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1098537000,IC_kwDOCGYnMM5Belgo,9599,simonw,2022-04-13T22:18:22Z,2022-04-13T22:18:22Z,OWNER,"I figured out a workaround in https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531
The current `register(fn)` method looks like this: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L403
This alternative implementation worked in the environment where that failed:
```python
def register(fn):
name = fn.__name__
arity = len(inspect.signature(fn).parameters)
if not replace and (name, arity) in self._registered_functions:
return fn
kwargs = {}
done = False
if deterministic:
# Try this, but fall back if sqlite3.NotSupportedError
try:
self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True))
done = True
except sqlite3.NotSupportedError:
pass
if not done:
self.conn.create_function(name, arity, fn, **kwargs)
self._registered_functions.add((name, arity))
return fn
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher,
https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1098545390,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1098545390,IC_kwDOCGYnMM5Benju,9599,simonw,2022-04-13T22:34:52Z,2022-04-13T22:34:52Z,OWNER,"That broke Python 3.7 because it doesn't support `deterministic=True` even being passed:
> function takes at most 3 arguments (4 given)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher,