issue_comments: 1248591268
This data as json
| html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
|---|---|---|---|---|---|---|---|---|---|---|---|
| https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248591268 | https://api.github.com/repos/simonw/sqlite-utils/issues/486 | 1248591268 | IC_kwDOCGYnMM5Ka_2k | 9599 | 2022-09-15T20:36:02Z | 2022-09-15T20:40:03Z | OWNER | I had a big CSV file lying around, I converted it to other formats like this: sqlite-utils insert /tmp/t.db t /tmp/en.openfoodfacts.org.products.csv --csv sqlite-utils rows /tmp/t.db t --nl > /tmp/big.nl sqlite-utils rows /tmp/t.db t > /tmp/big.json Then tested the progress bar like this: sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl Output: ``` sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl [------------------------------------] 0% [#######-----------------------------] 20% 00:00:20 ``` With `--silent` it is silent. And for regular JSON: ``` sqlite-utils insert /tmp/t3.db t /tmp/big.json [####################################] 100% ``` This is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | 1366512990 |