html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/861#issuecomment-647889674,https://api.github.com/repos/simonw/datasette/issues/861,647889674,MDEyOklzc3VlQ29tbWVudDY0Nzg4OTY3NA==,9599,simonw,2020-06-23T03:44:17Z,2020-06-23T03:44:17Z,OWNER,https://github.com/simonw/sqlite-generate is now ready to be used - see also https://pypi.org/project/sqlite-generate/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642652808,Script to generate larger SQLite test files,
https://github.com/simonw/datasette/issues/859#issuecomment-647890378,https://api.github.com/repos/simonw/datasette/issues/859,647890378,MDEyOklzc3VlQ29tbWVudDY0Nzg5MDM3OA==,9599,simonw,2020-06-23T03:47:19Z,2020-06-23T03:47:19Z,OWNER,"I generated a 600MB database using [sqlite-generate](https://github.com/simonw/sqlite-generate) just now - with 100 tables at 100,00 rows and 3 tables at 1,000,000 rows - and performance of the database page was fine, 250ms.
Those tables only had 4 columns each though.
You said ""200k+, 50+ rows in a couple of tables"" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647890619,https://api.github.com/repos/simonw/datasette/issues/859,647890619,MDEyOklzc3VlQ29tbWVudDY0Nzg5MDYxOQ==,9599,simonw,2020-06-23T03:48:21Z,2020-06-23T03:48:21Z,OWNER," sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50
Looks like that will take 35 minutes to run (it's not a particularly fast tool).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/862#issuecomment-647892930,https://api.github.com/repos/simonw/datasette/issues/862,647892930,MDEyOklzc3VlQ29tbWVudDY0Nzg5MjkzMA==,9599,simonw,2020-06-23T03:58:48Z,2020-06-23T03:58:48Z,OWNER,Should this be controlled be a separate configuration setting? I'm inclined to say no - I think instead I'll set the limit to be 10 * whatever `facet_suggest_time_limit_ms` is.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",643510821,Set an upper limit on total facet suggestion time for a page,
https://github.com/simonw/datasette/issues/596#issuecomment-647893140,https://api.github.com/repos/simonw/datasette/issues/596,647893140,MDEyOklzc3VlQ29tbWVudDY0Nzg5MzE0MA==,9599,simonw,2020-06-23T03:59:51Z,2020-06-23T03:59:51Z,OWNER,Related: #862 - a time limit on the total time spent considering suggested facets for a table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",507454958,Handle really wide tables better,
https://github.com/simonw/datasette/issues/859#issuecomment-647894903,https://api.github.com/repos/simonw/datasette/issues/859,647894903,MDEyOklzc3VlQ29tbWVudDY0Nzg5NDkwMw==,9599,simonw,2020-06-23T04:07:59Z,2020-06-23T04:07:59Z,OWNER,"Just to check: are you seeing the problem on this page: https://latest.datasette.io/fixtures (the database page) - or this page (the table page): https://latest.datasette.io/fixtures/compound_three_primary_keys
If it's the table page then the problem may well be #862.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647923666,https://api.github.com/repos/simonw/datasette/issues/859,647923666,MDEyOklzc3VlQ29tbWVudDY0NzkyMzY2Ng==,3243482,abdusco,2020-06-23T05:49:31Z,2020-06-23T05:49:31Z,CONTRIBUTOR,"I think I should mention that having FTS on all tables mean I have 5 visible, 25 hidden (FTS) tables displayed on database page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647925594,https://api.github.com/repos/simonw/datasette/issues/859,647925594,MDEyOklzc3VlQ29tbWVudDY0NzkyNTU5NA==,3243482,abdusco,2020-06-23T05:55:21Z,2020-06-23T06:28:29Z,CONTRIBUTOR,"Hmm, not seeing the problem now.
I've removed the commented out sections in `database.py` and restarted the process. Database page now loads in <250ms.
I have couple of workers that check some pages regularly and scrape new content and save to the DB. Could it be that datasette tries to recount tables every time database size changes? Normally it keeps a count cache, but as DB gets updated so often (new content every 5 min or so) it's practically recounting every time I go to the database page?
EDIT:
It turns out it doesn't hold cache with mutable databases.
I'll update the issue with more findings and a better way to reproduce the problem if I encounter it again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647935300,https://api.github.com/repos/simonw/datasette/issues/859,647935300,MDEyOklzc3VlQ29tbWVudDY0NzkzNTMwMA==,3243482,abdusco,2020-06-23T06:23:01Z,2020-06-23T06:23:01Z,CONTRIBUTOR,"> You said ""200k+, 50+ rows in a couple of tables"" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.
Ah that was a typo, I meant 50k.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-647936117,https://api.github.com/repos/simonw/datasette/issues/859,647936117,MDEyOklzc3VlQ29tbWVudDY0NzkzNjExNw==,3243482,abdusco,2020-06-23T06:25:17Z,2020-06-23T06:25:17Z,CONTRIBUTOR,">
>
> ```
> sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50
> ```
>
> Looks like that will take 35 minutes to run (it's not a particularly fast tool).
Try chunking write operations into batches every 1000 records or so.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-648163272,https://api.github.com/repos/simonw/datasette/issues/859,648163272,MDEyOklzc3VlQ29tbWVudDY0ODE2MzI3Mg==,9599,simonw,2020-06-23T13:52:23Z,2020-06-23T13:52:23Z,OWNER,"I'm chunking inserts at 100 at a time right now: https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L1030
I think the performance is more down to using Faker to create the test data - generating millions of entirely fake, randomized records takes a fair bit of time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-648232645,https://api.github.com/repos/simonw/datasette/issues/859,648232645,MDEyOklzc3VlQ29tbWVudDY0ODIzMjY0NQ==,3243482,abdusco,2020-06-23T15:19:53Z,2020-06-23T15:19:53Z,CONTRIBUTOR,"The issue seems to appear sporadically, like when I return to database page after a while, during which some records have been added to the database.
I've just visited database, page first visit took ~10s, consecutive visits took 0.3s.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-648234787,https://api.github.com/repos/simonw/datasette/issues/859,648234787,MDEyOklzc3VlQ29tbWVudDY0ODIzNDc4Nw==,9599,simonw,2020-06-23T15:22:51Z,2020-06-23T15:22:51Z,OWNER,"I wonder if this is a SQLite caching issue then?
Datasette has a configuration option for this but I haven't spent much time experimenting with it so I don't know how much of an impact it can have: https://datasette.readthedocs.io/en/stable/config.html#cache-size-kb","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/694#issuecomment-648296323,https://api.github.com/repos/simonw/datasette/issues/694,648296323,MDEyOklzc3VlQ29tbWVudDY0ODI5NjMyMw==,3903726,kwladyka,2020-06-23T17:10:51Z,2020-06-23T17:10:51Z,NONE,"@simonw
Did you find the reason? I had similar situation and I check this on millions ways. I am sure app doesn't consume such memory.
I was trying the app with:
`docker run --rm -it -p 80:80 -m 128M foo`
I was watching app with `docker stats`. Even limited memory by `CMD [""java"", ""-Xms60M"", ""-Xmx60M"", ""-jar"", ""api.jar""]`.
Checked memory usage by app in code and print bash commands. The app definitely doesn't use this memory. Also doesn't write files.
Only one solution is to change memory to 512M.
It is definitely something wrong with `cloud run`.
I even did special app for testing this. It looks like when I cross very small amount of code / memory / app size in random when, then memory needs grow +hundreds. Nothing make sense here. Especially it works everywhere expect cloud run.
Please let me know if you discovered something more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",576582604,datasette publish cloudrun --memory option,
https://github.com/simonw/sqlite-utils/issues/116#issuecomment-648403834,https://api.github.com/repos/simonw/sqlite-utils/issues/116,648403834,MDEyOklzc3VlQ29tbWVudDY0ODQwMzgzNA==,9599,simonw,2020-06-23T20:36:29Z,2020-06-23T20:36:29Z,OWNER,Should go in this section https://sqlite-utils.readthedocs.io/en/stable/python-api.html#introspection - under `.columns_dict` and before `.foreign_keys`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644122661,Documentation for table.pks introspection property,
https://github.com/simonw/sqlite-utils/issues/116#issuecomment-648434885,https://api.github.com/repos/simonw/sqlite-utils/issues/116,648434885,MDEyOklzc3VlQ29tbWVudDY0ODQzNDg4NQ==,9599,simonw,2020-06-23T21:21:33Z,2020-06-23T21:21:33Z,OWNER,New docs: https://github.com/simonw/sqlite-utils/blob/2.10.1/docs/python-api.rst#introspection,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644122661,Documentation for table.pks introspection property,
https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648440525,https://api.github.com/repos/simonw/sqlite-utils/issues/117,648440525,MDEyOklzc3VlQ29tbWVudDY0ODQ0MDUyNQ==,9599,simonw,2020-06-23T21:35:01Z,2020-06-23T21:35:01Z,OWNER,"Here's what's missing:
```
In [11]: db.conn.execute('PRAGMA foreign_key_list(song)').fetchall()
Out[11]:
[(0,
0,
'album',
'songartist',
'albumartist',
'NO ACTION',
'NO ACTION',
'NONE'),
(0, 1, 'album', 'songalbum', 'albumname', 'NO ACTION', 'NO ACTION', 'NONE')]
```
Compare with this code here:
https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/sqlite_utils/db.py#L563-L579
The first two columns returned by `PRAGMA foreign_key_list(table)` are `id` and `seq` - these show when two foreign key records are part of the same compound foreign key. `sqlite-utils` entirely ignores those at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644161221,Support for compound (composite) foreign keys,
https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648440634,https://api.github.com/repos/simonw/sqlite-utils/issues/117,648440634,MDEyOklzc3VlQ29tbWVudDY0ODQ0MDYzNA==,9599,simonw,2020-06-23T21:35:16Z,2020-06-23T21:35:16Z,OWNER,Relevant discussion: https://github.com/simonw/sqlite-generate/issues/8#issuecomment-648438056,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644161221,Support for compound (composite) foreign keys,
https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648442511,https://api.github.com/repos/simonw/sqlite-utils/issues/117,648442511,MDEyOklzc3VlQ29tbWVudDY0ODQ0MjUxMQ==,9599,simonw,2020-06-23T21:39:41Z,2020-06-23T21:39:41Z,OWNER,"So there are two sides to supporting this:
- Being able to sensibly introspect composite foreign keys
- Being able to define composite foreign keys when creating a table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644161221,Support for compound (composite) foreign keys,