html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1172697090,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1172697090,IC_kwDOCGYnMM5F5fAC,474467,2022-07-01T20:37:40Z,2022-07-01T20:37:55Z,NONE,"I just ran into what appears to be the same issue on a MacBook Pro, M1 Pro. 

Environment:
```
markd@Marks-MacBook-Pro metabase % python --version                                                             
Python 3.8.9
markd@Marks-MacBook-Pro metabase % sqlite3 --version                                                            
3.37.0 2021-12-09 01:34:53 9ff244ce0739f8ee52a3e9671adb4ee54c83c640b02e3f9d185fd2f9a179aapl
markd@Marks-MacBook-Pro metabase % sqlite-utils --version                                                        
sqlite-utils, version 3.27
markd@Marks-MacBook-Pro metabase % sqlite3 gh.db 
SQLite version 3.37.0 2021-12-09 01:34:53
Enter "".help"" for usage hints.
sqlite> .dbconfig
          defensive off
            dqs_ddl on
            dqs_dml on
        enable_fkey off
        enable_qpsg off
     enable_trigger on
        enable_view on
     fts3_tokenizer off
 legacy_alter_table on
 legacy_file_format off
     load_extension off
   no_ckpt_on_close off
     reset_database off
        trigger_eqp off
     trusted_schema on
    writable_schema off
``` 

Error
```
markd@Marks-MacBook-Pro metabase % github-to-sqlite repos gh.db a8cteam51                                        
Traceback (most recent call last):
  File ""/Users/markd/Library/Python/3.8/bin/github-to-sqlite"", line 8, in <module>
    sys.exit(cli())
  File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py"", line 1130, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py"", line 1055, in main
    rv = self.invoke(ctx)
  File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py"", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py"", line 760, in invoke
    return __callback(*args, **kwargs)
  File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/github_to_sqlite/cli.py"", line 268, in repos
    utils.ensure_db_shape(db)
  File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/github_to_sqlite/utils.py"", line 688, in ensure_db_shape
    ensure_foreign_keys(db)
  File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/github_to_sqlite/utils.py"", line 682, in ensure_foreign_keys
    db[table].add_foreign_key(column, table2, column2)
  File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/sqlite_utils/db.py"", line 2004, in add_foreign_key
    self.db.add_foreign_keys([(self.name, column, other_table, other_column)])
  File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/sqlite_utils/db.py"", line 1019, in add_foreign_keys
    cursor.execute(
sqlite3.OperationalError: table sqlite_master may not be modified
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1172766270,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1172766270,IC_kwDOCGYnMM5F5v4-,474467,2022-07-01T22:40:26Z,2022-07-01T22:40:26Z,NONE,"Note, I do not get this issue using my Intel MacBook Pro =/

Environment
```
markd@Marks-MBP metabase % python3 --version
Python 3.9.13
markd@Marks-MBP metabase % sqlite3 --version
3.37.0 2021-12-09 01:34:53 9ff244ce0739f8ee52a3e9671adb4ee54c83c640b02e3f9d185fd2f9a179aapl
markd@Marks-MBP metabase % sqlite-utils --version
sqlite-utils, version 3.27
markd@Marks-MBP metabase % sqlite3 github.db 
SQLite version 3.37.0 2021-12-09 01:34:53
Enter "".help"" for usage hints.
sqlite> .dbconfig
          defensive off
            dqs_ddl on
            dqs_dml on
        enable_fkey off
        enable_qpsg off
     enable_trigger on
        enable_view on
     fts3_tokenizer off
 legacy_alter_table on
 legacy_file_format off
     load_extension off
   no_ckpt_on_close off
     reset_database off
        trigger_eqp off
     trusted_schema on
    writable_schema off
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1198414383,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1198414383,IC_kwDOCGYnMM5Hblov,474467,2022-07-28T17:10:06Z,2022-07-28T17:10:06Z,NONE,"I was able to fight through this by capturing the SQL commands from the `add_foreign_keys()` function in sqlite-utils and then executing them manually via the sqlite3 client, first setting PRAGMA writable_schema on and then updating the sqlite_master table. Still no clue why they were failing when run in context...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1206241356,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1206241356,IC_kwDOCGYnMM5H5chM,503614,2022-08-05T09:26:15Z,2022-08-05T09:29:42Z,NONE,"I am getting the same error when using github-to-sqlite (which uses sqlite-utils internally). I am also using an M1 MacBook Pro, with macOS Monterey 12.5, with Python 3.10.6 for arm64 installed using pyenv. I have sqlite-utils 3.28 installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1223378004,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1223378004,IC_kwDOCGYnMM5I60RU,1558033,2022-08-23T00:44:11Z,2022-08-23T00:44:11Z,NONE,"This bug affects me as well. Env:

```
Python 3.8.12
sqlite-utils, version 3.28
sqlite3 3.32.3
MacOS Big Sur 11.6.7
Intel
```

Similar to @mdrovdahl, I was able to work around this bug by piping the SQL string constructed in `add_foreign_keys` to the `sqlite3` command itself. Specifically, if you're trying to patch this yourself, replace [lines 1026-1039 of db.py in your site packages](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L1026-L1039) with something similar to the following:

```
print(""PRAGMA writable_schema = 1;"")
for table_name, new_sql in table_sql.items():
    print(""UPDATE sqlite_master SET sql = '{}' WHERE name = '{}';"".format(
         new_sql, table_name)
    )
print(""PRAGMA writable_schema = 0;"")
print(""VACUUM;"")
```
 
Then from your terminal:
`db-to-sqlite ""<connection string>""  your.db --all > output.sql && sqlite3 your.db < output.sql`

If you want to run this with `-p`, you'll have to actually open a file in code to write to instead of redirecting the output.","{""total_count"": 3, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 3, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1304539296,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1304539296,IC_kwDOCGYnMM5NwbCg,559711,2022-11-05T12:40:12Z,2022-11-05T12:40:12Z,NONE,"I had the problem this morning when running:

`Python==3.9.6
 sqlite3.sqlite_version==3.37.0
  sqlite-utils==3.30
`

I upgraded to:
`Python ==3.10.8 
 sqlite3.sqlite_version==3.37.2
 sqlite-utils==3.30
`

and the error did not appear anymore.

Hope this helps
Ryan

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354160286,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1354160286,IC_kwDOCGYnMM5Qttie,9020979,2022-12-16T03:51:19Z,2022-12-16T03:52:13Z,NONE,"Hi @ryascott, thanks for sharing this! How did you upgrade your sqlite3 version? I'm running into this issue (also on an m1) with 

`Python ==3.10.7 sqlite3.sqlite_version==3.37.0 sqlite-utils==3.30`

Unfortunately, `3.10.8` isn't listed in `pyenv` so I'm unable to install it.

For me, the trigger is trying to use the `add-foreign-key` command on its own:

```bash
sqlite-utils add-foreign-key library.db book_creators creator_id creators id
```

Some stackoverflow searching suggests that brew installing sqlite may fix it ( https://stackoverflow.com/questions/26345972/how-do-i-upgrade-the-sqlite-version-used-by-pythons-sqlite3-module-on-mac ), but I don't want to risk breaking the version of sqlite used by some other system, I'd only like to upgrade sqlite3 inside my current virtual environment.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354192168,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1354192168,IC_kwDOCGYnMM5Qt1Uo,9020979,2022-12-16T04:35:30Z,2022-12-16T04:35:38Z,NONE,"A related historical problem:

https://github.com/tekartik/sqflite/issues/525#issuecomment-714500720

I wonder if the version of Sqlite or Python for Intel chip have defensive mode disabled by default, whereas M1 chips versions have it enabled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1492777509,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1492777509,IC_kwDOCGYnMM5Y-fol,9020979,2023-04-01T01:31:48Z,2023-04-01T01:31:48Z,NONE,"My current workaround is to use this library from a python script instead of as a CLI tool.

This lets me set the foreign key constraint at table creation time, instead of trying to modify an existing table. [docs](https://sqlite-utils.datasette.io/en/stable/python-api.html#specifying-foreign-keys)

I found this [stackoverflow helpful](https://stackoverflow.com/a/1884841/5129731), as it explained that Sqlite doesn't support modifying existing tables directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1495780111,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1495780111,IC_kwDOCGYnMM5ZJ8sP,19786848,2023-04-04T11:09:56Z,2023-04-04T11:13:40Z,NONE,"@wpears' workaround also worked for me, but also required me to manually set `PRAGMA writable_schema`.

```sql
PRAGMA writable_schema = 1;
UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (...)'
PRAGMA writable_schema = 0;
```

```shell
$ python --version
Python 3.11.2

$ sqlite3 --version
3.41.2 2023-03-22 11:56:21 0d1fc92f94cb6b76bffe3ec34d69cffde2924203304e8ffc4155597af0c191da

$ sqlite-utils --version
sqlite-utils, version 3.30
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502556111,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1502556111,IC_kwDOCGYnMM5Zjy_P,9599,2023-04-11T01:28:41Z,2023-04-11T01:28:41Z,OWNER,"Investigating this one now.

The `sqlite-utils` test suite passes without errors on my Python 3.11.2 installation... but it fails with this error on a Python 3.9.6 installation.

In the broken version's virtual environment directory I ran this:
```
cat pyvenv.cfg 
home = /Applications/Xcode.app/Contents/Developer/usr/bin
implementation = CPython
version_info = 3.9.6.final.0
virtualenv = 20.17.1
include-system-site-packages = false
base-prefix = /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9
base-exec-prefix = /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9
base-executable = /Applications/Xcode.app/Contents/Developer/usr/bin/python3
```
So it looks like the Xcode `python3` has ""defensive"" mode turned on for SQLite.

As far as I can tell there's no way to turn it OFF again in Python.

My virtual environment that DOES work has this:

```
home = /opt/homebrew/opt/python@3.11/bin
implementation = CPython
version_info = 3.11.2.final.0
virtualenv = 20.17.1
include-system-site-packages = false
base-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11
base-exec-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11
base-executable = /opt/homebrew/opt/python@3.11/bin/python3.11
```
So the Python 3.11 I installed through Homebrew doesn't exhibit this bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502557629,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1502557629,IC_kwDOCGYnMM5ZjzW9,9599,2023-04-11T01:30:12Z,2023-04-11T01:30:12Z,OWNER,"I'll ask on the SQLite forum if it's possible to toggle that mode on and off using regular SQL. My hunch is that it isn't.

In which case `sqlite-utils` should at least know how to catch this error and display a much more readable error message, maybe with a link to further documentation.

A utility function that can detect this mode would be really useful too. I'd probably have to do a test that tries to modify `sqlite_master` on a new in-memory database to catch if it's possible or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502559442,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1502559442,IC_kwDOCGYnMM5ZjzzS,9599,2023-04-11T01:32:30Z,2023-04-11T01:33:27Z,OWNER,"This seems to work:
```python
import sqlite3
db = sqlite3.connect("":memory:"")
db.executescript(""""""
PRAGMA writable_schema = 1;
UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)';
PRAGMA writable_schema = 0;
"""""")
```
It succeeds on my Python 3.11 and raises the following exception on my broken Python 3.9:
```
sqlite3.OperationalError: table sqlite_master may not be modified
```
Removing the `PRAGMA writable_schema = 1;` causes the same exception to be raised on both Pythons.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1504245029,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1504245029,IC_kwDOCGYnMM5ZqPUl,9599,2023-04-11T23:13:41Z,2023-04-11T23:14:39Z,OWNER,"I also tested this against the current `ubuntu:latest` Docker image (on an M2 Mac), in Python 3.10 and 3.11:
```
docker run -it ubuntu:latest /bin/bash
```
Then in the container:
```
apt-get update
apt-get install python3
python3
# pasted in the above recipe
apt install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt install python3.11
python3.11 
# pasted it in again
```
In both cases the Python code did not raise an exception, which suggests that on Ubuntu those two Python versions do not have the defensive mode set.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1504288134,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1504288134,IC_kwDOCGYnMM5ZqZ2G,9599,2023-04-11T23:55:06Z,2023-04-12T03:34:32Z,OWNER,"Also checked the official Datasette Docker image - I had to run that in Codespaces because it doesn't currently work on my M2 Mac:

```
codespace@codespaces-112c61:/workspaces/sqlite-utils$ docker pull datasetteproject/datasette
Using default tag: latest
...
codespace@codespaces-112c61:/workspaces/sqlite-utils$ docker run -it datasetteproject/datasette /
bin/bash
root@75ba34f501ec:/# python
Python 3.11.0 (main, Dec  6 2022, 13:31:55) [GCC 10.2.1 20210110] on linux
Type ""help"", ""copyright"", ""credits"" or ""license"" for more information.
>>> import sqlite3
.executescript(""""""
PRAGMA writable_schema = 1;
UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)';
PRAGMA writable_schema = 0;
"""""")>>> db = sqlite3.connect("":memory:"")
>>> db.executescript(""""""
... PRAGMA writable_schema = 1;
... UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)';
... PRAGMA writable_schema = 0;
... """""")
<sqlite3.Cursor object at 0x7fd9b0561140>
>>>
```
So that confirms that the official image also has a Python with a SQLite that's not in defensive mode.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1604379952,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1604379952,IC_kwDOCGYnMM5foOUw,9599,2023-06-23T14:39:55Z,2023-06-23T15:39:32Z,OWNER,"Ideally a workaround for this right now would be to install `pysqlite3` in the same virtual environment:

    sqlite-utils install pysqlite3-binary

But `pysqlite3-binary` doesn't yet ship a wheel for macOS so this probably won't work for most people.

The ""easiest"" fix at the moment is to use Python from Homebrew - so `brew install sqlite-utils` for example won't suffer from this problem. Not a great solution for people who aren't using Homebrew though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606411508,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1606411508,IC_kwDOCGYnMM5fv-T0,9599,2023-06-26T01:42:10Z,2023-06-26T01:42:22Z,OWNER,https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-33 - upgrading to `sqlite-utils>=3.33` and then installing both `sqlean.py` and `sqlite-dump` in the same virtual environment as `sqlite-utils` should fix this issue.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606415188,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1606415188,IC_kwDOCGYnMM5fv_NU,9599,2023-06-26T01:46:47Z,2023-06-26T01:47:01Z,OWNER,"I just tested this in a brand new virtual environment using the macOS Python 3:
```bash
pipenv shell --python /Applications/Xcode.app/Contents/Developer/usr/bin/python3
```
Then in that virtual environment I ran:
```bash
pip install sqlite-utils
# Confirm the right one is on the path:
which sqlite-utils

curl ""https://data.nasa.gov/resource/y77d-th95.json"" | \
    sqlite-utils insert meteorites.db meteorites - --pk=id
sqlite-utils extract meteorites.db meteorites  recclass
```
This threw the same error reported above. Then I did this:
```bash
rm meteorites.db
pip install sqlean.py

curl ""https://data.nasa.gov/resource/y77d-th95.json"" | \
    sqlite-utils insert meteorites.db meteorites - --pk=id
sqlite-utils extract meteorites.db meteorites  recclass
```
And that second time it worked correctly.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",810618495,