html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/1851#issuecomment-1294281451,https://api.github.com/repos/simonw/datasette/issues/1851,1294281451,IC_kwDOBm6k_c5NJSrr,9599,simonw,2022-10-28T00:59:25Z,2022-10-28T00:59:25Z,OWNER,"I'm going to use this endpoint for bulk inserts too, so I'm closing this issue and continuing the work here:
- #1866","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table,
https://github.com/simonw/datasette/issues/1866#issuecomment-1294282263,https://api.github.com/repos/simonw/datasette/issues/1866,1294282263,IC_kwDOBm6k_c5NJS4X,9599,simonw,2022-10-28T01:00:42Z,2022-10-28T01:00:42Z,OWNER,"I'm going to set the limit at 1,000 rows inserted at a time. I'll make this configurable using a new `max_insert_rows` setting (for consistency with `max_returned_rows`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table,
https://github.com/simonw/datasette/issues/1866#issuecomment-1294296767,https://api.github.com/repos/simonw/datasette/issues/1866,1294296767,IC_kwDOBm6k_c5NJWa_,9599,simonw,2022-10-28T01:22:25Z,2022-10-28T01:23:09Z,OWNER,"Nasty catch on this one: I wanted to return the IDs of the freshly inserted rows. But... the `insert_all()` method I was planning to use from `sqlite-utils` doesn't appear to have a way of doing that:
https://github.com/simonw/sqlite-utils/blob/529110e7d8c4a6b1bbf5fb61f2e29d72aa95a611/sqlite_utils/db.py#L2813-L2835
SQLite itself added a `RETURNING` statement which might help, but that is only available from version 3.35 released in March 2021: https://www.sqlite.org/lang_returning.html - which isn't commonly available yet. https://latest.datasette.io/-/versions right now shows 3.34, and https://lite.datasette.io/#/-/versions shows 3.27.2 (from Feb 2019).
Two options then:
1. Even for bulk inserts do one insert at a time so I can use `cursor.lastrowid` to get the ID of the inserted record. This isn't terrible since SQLite is very fast, but it may still be a big performance hit for large inserts.
2. Don't return the list of inserted rows for bulk inserts
3. Default to not returning the list of inserted rows for bulk inserts, but allow the user to request that - in which case we use the slower path
That third option might be the way to go here.
I should benchmark first to figure out how much of a difference this actually makes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table,
https://github.com/simonw/datasette/issues/1866#issuecomment-1294306071,https://api.github.com/repos/simonw/datasette/issues/1866,1294306071,IC_kwDOBm6k_c5NJYsX,9599,simonw,2022-10-28T01:37:14Z,2022-10-28T01:37:59Z,OWNER,"Quick crude benchmark:
```python
import sqlite3
db = sqlite3.connect("":memory:"")
def create_table(db, name):
db.execute(f""create table {name} (id integer primary key, title text)"")
create_table(db, ""single"")
create_table(db, ""multi"")
create_table(db, ""bulk"")
def insert_singles(titles):
inserted = []
for title in titles:
cursor = db.execute(f""insert into single (title) values (?)"", [title])
inserted.append((cursor.lastrowid, title))
return inserted
def insert_many(titles):
db.executemany(f""insert into multi (title) values (?)"", ((t,) for t in titles))
def insert_bulk(titles):
db.execute(""insert into bulk (title) values {}"".format(
"", "".join(""(?)"" for _ in titles)
), titles)
titles = [""title {}"".format(i) for i in range(1, 10001)]
```
Then in iPython I ran these:
```
In [14]: %timeit insert_singles(titles)
23.8 ms ± 535 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [13]: %timeit insert_many(titles)
12 ms ± 520 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [12]: %timeit insert_bulk(titles)
2.59 ms ± 25 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
So the bulk insert really is a lot faster - 3ms compared to 24ms for single inserts, so ~8x faster.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table,
https://github.com/simonw/datasette/issues/1866#issuecomment-1294316640,https://api.github.com/repos/simonw/datasette/issues/1866,1294316640,IC_kwDOBm6k_c5NJbRg,9599,simonw,2022-10-28T01:51:40Z,2022-10-28T01:51:40Z,OWNER,"This needs to support the following:
- Rows do not include a primary key - one is assigned by the database
- Rows provide their own primary key, any clashes are errors
- Rows provide their own primary key, clashes are silently ignored
- Rows provide their own primary key, replacing any existing records","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table,
https://github.com/simonw/datasette/issues/1866#issuecomment-1295200988,https://api.github.com/repos/simonw/datasette/issues/1866,1295200988,IC_kwDOBm6k_c5NMzLc,9599,simonw,2022-10-28T16:29:55Z,2022-10-28T16:29:55Z,OWNER,"I wonder if there's something clever I could do here within a transaction?
Start a transaction. Write out a temporary in-memory table with all of the existing primary keys in the table. Run the bulk insert. Then run `select pk from table where pk not in (select pk from old_pks)` to see what has changed.
I don't think that's going to work well for large tables.
I'm going to go with not returning inserted rows by default, unless you pass a special option requesting that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table,
https://github.com/simonw/datasette/pull/1870#issuecomment-1294285471,https://api.github.com/repos/simonw/datasette/issues/1870,1294285471,IC_kwDOBm6k_c5NJTqf,536941,fgregg,2022-10-28T01:06:03Z,2022-10-28T01:06:03Z,CONTRIBUTOR,"as far as i can tell, [this is where the ""immutable"" argument is used](https://github.com/sqlite/sqlite/blob/c97bb14fab566f6fa8d967c8fd1e90f3702d5b73/src/pager.c#L4926-L4931) in sqlite:
```c
pPager->noLock = sqlite3_uri_boolean(pPager->zFilename, ""nolock"", 0);
if( (iDc & SQLITE_IOCAP_IMMUTABLE)!=0
|| sqlite3_uri_boolean(pPager->zFilename, ""immutable"", 0) ){
vfsFlags |= SQLITE_OPEN_READONLY;
goto act_like_temp_file;
}
```
so it does set the read only flag, but then has a goto.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro",
https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1294408928,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1294408928,IC_kwDOCGYnMM5NJxzg,39538958,justmars,2022-10-28T03:36:56Z,2022-10-28T03:37:50Z,NONE,"With respect to the typing of Table class itself, my interim solution:
```python
from sqlite_utils.db import Table
def tbl(self, table_name: str) -> Table:
tbl = self.db[table_name]
if isinstance(tbl, Table):
return tbl
raise Exception(f""Missing {table_name=}"")
```
With respect to @chapmanjacobd concern on the `DEFAULT` being an empty class, have also been using `# type: ignore`, e.g.
```python
@classmethod
def insert_list(cls, areas: list[str]):
return meta.tbl(meta.Areas).insert_all(
({""area"": a} for a in areas), ignore=True # type: ignore
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting,