issue_comments
42 rows where "updated_at" is on date 2022-01-09
This data as json, CSV (advanced)
Suggested facets: issue_url, issue, created_at (date)
| id ▼ | html_url | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1008158799 | https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1008158799 | https://api.github.com/repos/simonw/sqlite-utils/issues/367 | IC_kwDOCGYnMM48F0hP | codecov[bot] 22429695 | 2022-01-08T21:36:55Z | 2022-01-09T02:34:44Z | NONE | # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#367](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9848eaa) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a8f9cc6f64f299830834428509940d448b82b4ed?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a8f9cc6) will **decrease** coverage by `0.20%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #367 +/- ## ========================================== - Coverage 96.44% 96.24% -0.21% ========================================== Files 5 6 +1 Lines 2307 2317 +10 ========================================== + Hits 2225 2230 +5 - Misses 82 87 +5 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.15% <28.57%> (-0.42%)` … | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Initial prototype of .analyze() methods 1097041471 | |
| 1008163585 | https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163585 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | IC_kwDOCGYnMM48F1sB | simonw 9599 | 2022-01-08T22:14:39Z | 2022-01-09T03:03:07Z | OWNER | The reason I'm hesitating on this is that I've not actually used ANALYZE at all in nearly five years of messing around with SQLite! So I'm nervous that there are surprise downsides I haven't thought of. My hunch is that ANALYZE is only worth worrying about on much larger databases, in which case I'm OK supporting it as a thoroughly documented power-user feature rather than a default. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | create-index should run analyze after creating index 1096558279 | |
| 1008214406 | https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214406 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | IC_kwDOCGYnMM48GCGG | simonw 9599 | 2022-01-09T02:18:21Z | 2022-01-09T02:18:21Z | OWNER | I'm having trouble figuring out the best way to write a unit test for this. Filed a relevant feature request for Click here: - https://github.com/pallets/click/issues/2171 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `--batch-size 1` doesn't seem to commit for every item 1095570074 | |
| 1008214998 | https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | IC_kwDOCGYnMM48GCPW | simonw 9599 | 2022-01-09T02:23:20Z | 2022-01-09T02:23:20Z | OWNER | Possible way of running the test: add this to `sqlite_utils/cli.py`: ```python if __name__ == "__main__": cli() ``` Now the tool can be run using `python -m sqlite_utils.cli --help` Then in the test use `subprocess` to call `sys.executable` (the path to the current Python interpreter) and pass it `-m sqlite_utils.cli` to run the script! | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `--batch-size 1` doesn't seem to commit for every item 1095570074 | |
| 1008215912 | https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008215912 | https://api.github.com/repos/simonw/sqlite-utils/issues/368 | IC_kwDOCGYnMM48GCdo | simonw 9599 | 2022-01-09T02:30:59Z | 2022-01-09T02:30:59Z | OWNER | Even better, inspired by `rich`, support `python -m sqlite_utils`. https://github.com/Textualize/rich/blob/master/rich/__main__.py | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Offer `python -m sqlite_utils` as an alternative to `sqlite-utils` 1097087280 | |
| 1008216201 | https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008216201 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | IC_kwDOCGYnMM48GCiJ | simonw 9599 | 2022-01-09T02:34:12Z | 2022-01-09T02:34:12Z | OWNER | I can now write tests that look like this: https://github.com/simonw/sqlite-utils/blob/539f5ccd90371fa87f946018f8b77d55929e06db/tests/test_cli.py#L2024-L2030 Which means I can write a test that exercises this bug. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `--batch-size 1` doesn't seem to commit for every item 1095570074 | |
| 1008216271 | https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216271 | https://api.github.com/repos/simonw/sqlite-utils/issues/368 | IC_kwDOCGYnMM48GCjP | simonw 9599 | 2022-01-09T02:35:09Z | 2022-01-09T02:35:09Z | OWNER | Test failure on Python 3.6: > `E TypeError: __init__() got an unexpected keyword argument 'capture_output'` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Offer `python -m sqlite_utils` as an alternative to `sqlite-utils` 1097087280 | |
| 1008216371 | https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216371 | https://api.github.com/repos/simonw/sqlite-utils/issues/368 | IC_kwDOCGYnMM48GCkz | simonw 9599 | 2022-01-09T02:36:22Z | 2022-01-09T02:36:22Z | OWNER | In Python 3.6: https://docs.python.org/3.6/library/subprocess.html > This does not capture stdout or stderr by default. To do so, pass [`PIPE`](https://docs.python.org/3.6/library/subprocess.html#subprocess.PIPE "subprocess.PIPE") for the *stdout* and/or *stderr* arguments. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Offer `python -m sqlite_utils` as an alternative to `sqlite-utils` 1097087280 | |
| 1008219191 | https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219191 | https://api.github.com/repos/simonw/sqlite-utils/issues/369 | IC_kwDOCGYnMM48GDQ3 | simonw 9599 | 2022-01-09T03:03:53Z | 2022-01-09T03:03:53Z | OWNER | Refs: - #366 - #365 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research how much of a difference analyze / sqlite_stat1 makes 1097091527 | |
| 1008219484 | https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219484 | https://api.github.com/repos/simonw/sqlite-utils/issues/369 | IC_kwDOCGYnMM48GDVc | simonw 9599 | 2022-01-09T03:05:44Z | 2022-01-09T03:05:44Z | OWNER | I'll start by running some experiments against the 11MB database file from https://global-power-plants.datasettes.com/global-power-plants.db | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research how much of a difference analyze / sqlite_stat1 makes 1097091527 | |
| 1008219588 | https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219588 | https://api.github.com/repos/simonw/sqlite-utils/issues/369 | IC_kwDOCGYnMM48GDXE | simonw 9599 | 2022-01-09T03:06:42Z | 2022-01-09T03:06:42Z | OWNER | ``` analyze % sqlite-utils indexes global-power-plants.db -t table index_name seqno cid name desc coll key ------------------------------ ------------------------------------------------- ------- ----- ------------ ------ ------ ----- global-power-plants "global-power-plants_owner" 0 12 owner 0 BINARY 1 global-power-plants "global-power-plants_country_long" 0 1 country_long 0 BINARY 1 global-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 0 0 segid 0 BINARY 1 global-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 1 1 term 0 BINARY 1 global-power-plants_fts_config sqlite_autoindex_global-power-plants_fts_config_1 0 0 k 0 BINARY 1 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research how much of a difference analyze / sqlite_stat1 makes 1097091527 | |
| 1008219844 | https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219844 | https://api.github.com/repos/simonw/sqlite-utils/issues/369 | IC_kwDOCGYnMM48GDbE | simonw 9599 | 2022-01-09T03:08:09Z | 2022-01-09T03:08:09Z | OWNER | ``` analyze % sqlite-utils global-power-plants-analyzed.db 'analyze' [{"rows_affected": -1}] analyze % sqlite-utils tables global-power-plants-analyzed.db [{"table": "global-power-plants"}, {"table": "global-power-plants_fts"}, {"table": "global-power-plants_fts_data"}, {"table": "global-power-plants_fts_idx"}, {"table": "global-power-plants_fts_docsize"}, {"table": "global-power-plants_fts_config"}, {"table": "sqlite_stat1"}] analyze % sqlite-utils rows global-power-plants-analyzed.db sqlite_stat1 -t tbl idx stat ------------------------------- ---------------------------------- --------- global-power-plants_fts_config global-power-plants_fts_config 1 1 global-power-plants_fts_docsize 33643 global-power-plants_fts_idx global-power-plants_fts_idx 199 40 1 global-power-plants_fts_data 136 global-power-plants "global-power-plants_owner" 33643 4 global-power-plants "global-power-plants_country_long" 33643 202 ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research how much of a difference analyze / sqlite_stat1 makes 1097091527 | |
| 1008220270 | https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008220270 | https://api.github.com/repos/simonw/sqlite-utils/issues/369 | IC_kwDOCGYnMM48GDhu | simonw 9599 | 2022-01-09T03:12:38Z | 2022-01-09T03:13:15Z | OWNER | Basically no difference using this very basic benchmark: ``` analyze % python3 -m timeit '__import__("sqlite3").connect("global-power-plants.db").execute("select country_long, count(*) from [global-power-plants] group by country_long").fetchall()' 100 loops, best of 5: 2.39 msec per loop analyze % python3 -m timeit '__import__("sqlite3").connect("global-power-plants-analyzed.db").execute("select country_long, count(*) from [global-power-plants] group by country_long").fetchall()' 100 loops, best of 5: 2.38 msec per loop ``` I should try this against a much larger database. https://covid-19.datasettes.com/covid.db is 879MB. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research how much of a difference analyze / sqlite_stat1 makes 1097091527 | |
| 1008226487 | https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226487 | https://api.github.com/repos/simonw/sqlite-utils/issues/369 | IC_kwDOCGYnMM48GFC3 | simonw 9599 | 2022-01-09T04:14:05Z | 2022-01-09T04:14:05Z | OWNER | Didn't manage to spot a meaningful difference with that database either: ``` analyze % python3 -m timeit '__import__("sqlite3").connect("covid.db").execute("select fips, count(*) from [ny_times_us_counties] group by fips").fetchall()' 2 loops, best of 5: 101 msec per loop analyze % python3 -m timeit '__import__("sqlite3").connect("covid-analyzed.db").execute("select fips, count(*) from [ny_times_us_counties] group by fips").fetchall()' 2 loops, best of 5: 103 msec per loop ``` Maybe `select fips, count(*) from [ny_times_us_counties] group by fips` isn't a good query for testing this? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research how much of a difference analyze / sqlite_stat1 makes 1097091527 | |
| 1008226862 | https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226862 | https://api.github.com/repos/simonw/sqlite-utils/issues/369 | IC_kwDOCGYnMM48GFIu | simonw 9599 | 2022-01-09T04:17:55Z | 2022-01-09T04:17:55Z | OWNER | There are some clues as to what effect ANALYZE has in https://www.sqlite.org/optoverview.html Some quotes: > SQLite might use a skip-scan on an index if it knows that the first one or more columns contain many duplication values. If there are too few duplicates in the left-most columns of the index, then it would be faster to simply step ahead to the next value, and thus do a full table scan, than to do a binary search on an index to locate the next left-column value. > > The only way that SQLite can know that there are many duplicates in the left-most columns of an index is if the ANALYZE command has been run on the database. Without the results of ANALYZE, SQLite has to guess at the "shape" of the data in the table, and the default guess is that there are an average of 10 duplicates for every value in the left-most column of the index. Skip-scan only becomes profitable (it only gets to be faster than a full table scan) when the number of duplicates is about 18 or more. Hence, a skip-scan is never used on a database that has not been analyzed. And > Join reordering is automatic and usually works well enough that programmers do not have to think about it, especially if ANALYZE has been used to gather statistics about the available indexes, though occasionally some hints from the programmer are needed. And > The various sqlite_statN tables contain information on how selective the various indexes are. For example, the sqlite_stat1 table might indicate that an equality constraint on column x reduces the search space to 10 rows on average, whereas an equality constraint on column y reduces the search space to 3 rows on average. In that case, SQLite would prefer to use index ex2i2 since that index is more selective. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research how much of a difference analyze / sqlite_stat1 makes 1097091527 | |
| 1008227436 | https://github.com/simonw/datasette/issues/1588#issuecomment-1008227436 | https://api.github.com/repos/simonw/datasette/issues/1588 | IC_kwDOBm6k_c48GFRs | simonw 9599 | 2022-01-09T04:23:37Z | 2022-01-09T04:25:04Z | OWNER | Relevant code: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L163-L170 https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L195-L204 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `explain query plan select` is too strict about whitespace 1097101917 | |
| 1008227491 | https://github.com/simonw/datasette/issues/1588#issuecomment-1008227491 | https://api.github.com/repos/simonw/datasette/issues/1588 | IC_kwDOBm6k_c48GFSj | simonw 9599 | 2022-01-09T04:24:09Z | 2022-01-09T04:24:09Z | OWNER | I think this is the fix: ```python re.compile(r"^explain\s+query\s+plan\s+select\b"), ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `explain query plan select` is too strict about whitespace 1097101917 | |
| 1008227625 | https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008227625 | https://api.github.com/repos/simonw/sqlite-utils/issues/369 | IC_kwDOCGYnMM48GFUp | simonw 9599 | 2022-01-09T04:25:38Z | 2022-01-09T04:25:38Z | OWNER | ```sql EXPLAIN QUERY PLAN select country_long, count(*) from [global-power-plants] group by country_long ``` https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+country_long%2C+count%28*%29+from+%5Bglobal-power-plants%5D+group+by+country_long > SCAN TABLE global-power-plants USING COVERING INDEX "global-power-plants_country_long" | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research how much of a difference analyze / sqlite_stat1 makes 1097091527 | |
| 1008229341 | https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008229341 | https://api.github.com/repos/simonw/sqlite-utils/issues/369 | IC_kwDOCGYnMM48GFvd | simonw 9599 | 2022-01-09T04:45:38Z | 2022-01-09T04:47:11Z | OWNER | This is probably too fancy. I think maybe the way to do this is with `select * from [global-power-plants] where "country_long" = 'United Kingdom'` - then mess around with stats to see if I can get it to use the index or not based on them. Here's the explain for that: https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+*+from+[global-power-plants]+where+%22country_long%22+%3D+%27United+Kingdom%27 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research how much of a difference analyze / sqlite_stat1 makes 1097091527 | |
| 1008229839 | https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008229839 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | IC_kwDOCGYnMM48GF3P | simonw 9599 | 2022-01-09T04:51:44Z | 2022-01-09T04:51:44Z | OWNER | Found one report on Stack Overflow from 9 years ago of someone seeing broken performance after running `ANALYZE`, hard to say that's a trend and not a single weird edge-case though! https://stackoverflow.com/q/12947214/6083 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | create-index should run analyze after creating index 1096558279 | |
| 1008232075 | https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008232075 | https://api.github.com/repos/simonw/sqlite-utils/issues/369 | IC_kwDOCGYnMM48GGaL | simonw 9599 | 2022-01-09T05:13:15Z | 2022-01-09T05:13:56Z | OWNER | I think the query that will help solve this is: `explain query plan select * from ny_times_us_counties where state = 1 and county = 2` In this case, the query planner needs to decide if it should use the index for the `state` column or the index for the `county` column. That's where the statistics come into play. In particular: | tbl | idx | stat | |----------------------|---------------------------------|---------------| | ny_times_us_counties | idx_ny_times_us_counties_date | 2092871 2915 | | ny_times_us_counties | idx_ny_times_us_counties_fips | 2092871 651 | | ny_times_us_counties | idx_ny_times_us_counties_county | 2092871 1085 | | ny_times_us_counties | idx_ny_times_us_counties_state | 2092871 37373 | Those numbers are explained by this comment in the SQLite C code: https://github.com/sqlite/sqlite/blob/5622c7f97106314719740098cf0854e7eaa81802/src/analyze.c#L41-L55 ``` ** There is normally one row per index, with the index identified by the ** name in the idx column. The tbl column is the name of the table to ** which the index belongs. In each such row, the stat column will be ** a string consisting of a list of integers. The first integer in this ** list is the number of rows in the index. (This is the same as the ** number of rows in the table, except for partial indices.) The second ** integer is the average number of rows in the index that have the same ** value in the first column of the index. ``` So that table is telling us that using a value in the `county` column will filter down to an average of 1,085 rows, whereas filtering on the `state` column will filter down to an average of 37,373 - so clearly the `county` index is the better index to use here! Just one catch: against both my` covid.db` and my `covid-analyzed.db` databases the `county` index is picked for both of them - so SQLite is somehow guessing that `county` is a better index even though it doesn't have statistics for that. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Research how much of a difference analyze / sqlite_stat1 makes 1097091527 | |
| 1008233910 | https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008233910 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | IC_kwDOCGYnMM48GG22 | simonw 9599 | 2022-01-09T05:32:53Z | 2022-01-09T05:35:45Z | OWNER | This is strange. The following: ```pycon >>> import subprocess >>> p = subprocess.Popen(["sqlite-utils", "insert", "/tmp/stream.db", "stream", "-", "--nl"], stdin=subprocess.PIPE) >>> p.stdin.write(b'\n'.join(b'{"id": %s}' % str(i).encode("utf-8") for i in range(1000))) 11889 >>> # At this point /tmp/stream.db is still 0 bytes - but if I then run this: >>> p.stdin.close() >>> # /tmp/stream.db is now 20K and contains the written data ``` No wait, mystery solved - I can add `p.stdin.flush()` instead of `p.stdin.close()` and the file suddenly jumps up in size. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `--batch-size 1` doesn't seem to commit for every item 1095570074 | |
| 1008234293 | https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008234293 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | IC_kwDOCGYnMM48GG81 | simonw 9599 | 2022-01-09T05:37:02Z | 2022-01-09T05:37:02Z | OWNER | Calling `p.stdin.close()` and then `p.wait()` terminates the subprocess. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `--batch-size 1` doesn't seem to commit for every item 1095570074 | |
| 1008246239 | https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246239 | https://api.github.com/repos/simonw/sqlite-utils/issues/371 | IC_kwDOCGYnMM48GJ3f | simonw 9599 | 2022-01-09T07:41:24Z | 2022-01-09T07:41:24Z | OWNER | Might be a case of modifying this line: https://github.com/simonw/sqlite-utils/blob/e0c476bc380744680c8b7675c24fb0e9f5ec6dcd/sqlite_utils/cli.py#L828 To: ```python docs = (fn(doc) or doc for doc in docs) ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support mutating row in `--convert` without returning it 1097128334 | |
| 1008246366 | https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246366 | https://api.github.com/repos/simonw/sqlite-utils/issues/371 | IC_kwDOCGYnMM48GJ5e | simonw 9599 | 2022-01-09T07:42:14Z | 2022-01-09T07:42:14Z | OWNER | Also need to update relevant docs for that example. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support mutating row in `--convert` without returning it 1097128334 | |
| 1008247370 | https://github.com/simonw/sqlite-utils/issues/372#issuecomment-1008247370 | https://api.github.com/repos/simonw/sqlite-utils/issues/372 | IC_kwDOCGYnMM48GKJK | simonw 9599 | 2022-01-09T07:51:18Z | 2022-01-09T07:51:18Z | OWNER | Pathlib says the stem of that would be `dogs.and.cats.jpg` - best stick with that for consistency. https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.suffix It calls the last bit `suffix` - maybe I should use that instead of `ext`. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Idea: `suffix` and `stem` file columns 1097129710 | |
| 1008252732 | https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008252732 | https://api.github.com/repos/simonw/sqlite-utils/issues/374 | IC_kwDOCGYnMM48GLc8 | simonw 9599 | 2022-01-09T08:25:30Z | 2022-01-09T08:25:30Z | OWNER | Need to change `if table:` to `if table or fmt:` in a few places. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `--fmt` should imply `-t` 1097135860 | |
| 1008275546 | https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008275546 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | IC_kwDOCGYnMM48GRBa | fgregg 536941 | 2022-01-09T11:01:15Z | 2022-01-09T13:37:51Z | CONTRIBUTOR | i don’t want to be such a partisan for analyze, but the query planner deciding *not* to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice. <s>the original poster in that stack overflow doesn’t say there’s a performance regression </s> | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | create-index should run analyze after creating index 1096558279 | |
| 1008279307 | https://github.com/simonw/datasette/pull/1574#issuecomment-1008279307 | https://api.github.com/repos/simonw/datasette/issues/1574 | IC_kwDOBm6k_c48GR8L | fs111 33631 | 2022-01-09T11:26:06Z | 2022-01-09T11:26:06Z | NONE | @fgregg my thinking was backwards compatibility. I don't know what people do to their builds, I just wanted a smaller image for my use case. @simonw any chance to take a look at this? If there is no interest, feel free to close the PR | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | introduce new option for datasette package to use a slim base image 1084193403 | |
| 1008338186 | https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008338186 | https://api.github.com/repos/simonw/sqlite-utils/issues/375 | IC_kwDOCGYnMM48GgUK | simonw 9599 | 2022-01-09T17:13:33Z | 2022-01-09T17:13:54Z | OWNER | cat blah.csv | sqlite-utils bulk blah.db - \ "insert into blah (:foo, :bar)" --csv | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `sqlite-utils bulk` command 1097251014 | |
| 1008341078 | https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008341078 | https://api.github.com/repos/simonw/sqlite-utils/issues/373 | IC_kwDOCGYnMM48GhBW | simonw 9599 | 2022-01-09T17:31:12Z | 2022-01-09T17:31:12Z | OWNER | Found an example of using `cog` in a rST file here: https://github.com/nedbat/coveragepy/blob/f3238eea7e403d13a217b30579b1a1c2cbff62e3/doc/dbschema.rst#L21 ``` .. [[[cog from coverage.sqldata import SCHEMA_VERSION print(".. code::") print() print(f" SCHEMA_VERSION = {SCHEMA_VERSION}") print() .. ]]] .. code:: SCHEMA_VERSION = 7 .. [[[end]]] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | List `--fmt` options in the docs 1097135732 | |
| 1008344525 | https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344525 | https://api.github.com/repos/simonw/sqlite-utils/issues/373 | IC_kwDOCGYnMM48Gh3N | simonw 9599 | 2022-01-09T17:52:22Z | 2022-01-09T17:52:22Z | OWNER | Updated docs: https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | List `--fmt` options in the docs 1097135732 | |
| 1008344980 | https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344980 | https://api.github.com/repos/simonw/sqlite-utils/issues/373 | IC_kwDOCGYnMM48Gh-U | simonw 9599 | 2022-01-09T17:54:53Z | 2022-01-09T17:54:53Z | OWNER | Updated TIL: https://til.simonwillison.net/python/cog-to-update-help-in-readme#user-content-cog-for-restructuredtext | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | List `--fmt` options in the docs 1097135732 | |
| 1008345267 | https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008345267 | https://api.github.com/repos/simonw/sqlite-utils/issues/374 | IC_kwDOCGYnMM48GiCz | simonw 9599 | 2022-01-09T17:56:37Z | 2022-01-09T17:56:37Z | OWNER | Better: ```python if fmt: table = True ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `--fmt` should imply `-t` 1097135860 | |
| 1008346338 | https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346338 | https://api.github.com/repos/simonw/sqlite-utils/issues/374 | IC_kwDOCGYnMM48GiTi | simonw 9599 | 2022-01-09T18:03:22Z | 2022-01-09T18:03:22Z | OWNER | Commands that support `--fmt` (via the `@output_options` decorator) are: - `tables` - `views` - `query` - `memory` - `search` - `rows` - `triggers` - `indexes` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `--fmt` should imply `-t` 1097135860 | |
| 1008346841 | https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346841 | https://api.github.com/repos/simonw/sqlite-utils/issues/374 | IC_kwDOCGYnMM48GibZ | simonw 9599 | 2022-01-09T18:06:50Z | 2022-01-09T18:06:50Z | OWNER | In addition to a unit test I manually tested all of the above, e.g. ``` % sqlite-utils indexes global-power-plants.db sqlite_master --fmt rst ======= ============ ======= ===== ====== ====== ====== ===== table index_name seqno cid name desc coll key ======= ============ ======= ===== ====== ====== ====== ===== ======= ============ ======= ===== ====== ====== ====== ===== ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | `--fmt` should imply `-t` 1097135860 | |
| 1008347768 | https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008347768 | https://api.github.com/repos/simonw/sqlite-utils/issues/371 | IC_kwDOCGYnMM48Gip4 | simonw 9599 | 2022-01-09T18:12:30Z | 2022-01-09T18:12:30Z | OWNER | Tried this test: ```python result = CliRunner().invoke( cli.cli, [ "insert", db_path, "rows", "-", "--convert", 'row["is_chicken"] = True', ], input='{"name": "Azi"}', ) ``` And got this error: > `E + where 1 = <Result SyntaxError('invalid syntax', ('<string>', 2, 30, ' return row["is_chicken"] = True\n'))>.exit_code` The code snippet compilation isn't currently compatible with this. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support mutating row in `--convert` without returning it 1097128334 | |
| 1008348032 | https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008348032 | https://api.github.com/repos/simonw/sqlite-utils/issues/371 | IC_kwDOCGYnMM48GiuA | simonw 9599 | 2022-01-09T18:14:02Z | 2022-01-09T18:14:02Z | OWNER | Here's the code in question: https://github.com/simonw/sqlite-utils/blob/b8c134059e89f0fa040b84fb7d0bda25b9a52759/sqlite_utils/utils.py#L288-L299 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support mutating row in `--convert` without returning it 1097128334 | |
| 1008354207 | https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008354207 | https://api.github.com/repos/simonw/sqlite-utils/issues/371 | IC_kwDOCGYnMM48GkOf | simonw 9599 | 2022-01-09T18:54:54Z | 2022-01-09T18:54:54Z | OWNER | This seems to work: ```python def _compile_code(code, imports, variable="value"): locals = {} globals = {"r": recipes, "recipes": recipes} # If user defined a convert() function, return that try: exec(code, globals, locals) return locals["convert"] except (AttributeError, SyntaxError, NameError, KeyError, TypeError): pass # Try compiling their code as a function instead body_variants = [code] # If single line and no 'return', try adding the return if "\n" not in code and not code.strip().startswith("return "): body_variants.insert(0, "return {}".format(code)) for variant in body_variants: new_code = ["def fn({}):".format(variable)] for line in variant.split("\n"): new_code.append(" {}".format(line)) try: code_o = compile("\n".join(new_code), "<string>", "exec") break except SyntaxError: # Try another variant, e.g. for 'return row["column"] = 1' continue for import_ in imports: globals[import_.split(".")[0]] = __import__(import_) exec(code_o, globals, locals) return locals["fn"] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support mutating row in `--convert` without returning it 1097128334 | |
| 1008364701 | https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008364701 | https://api.github.com/repos/simonw/sqlite-utils/issues/371 | IC_kwDOCGYnMM48Gmyd | simonw 9599 | 2022-01-09T20:04:35Z | 2022-01-09T20:04:35Z | OWNER | The previous code for highlighting errors in syntax (which was already a bit confused thanks to the added `return`, see https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684 - isn't compatible with this approach at all. I'm going to ditch it and just show a generic `Error: Could not compile code` message. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Support mutating row in `--convert` without returning it 1097128334 | |
| 1008367607 | https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008367607 | https://api.github.com/repos/simonw/sqlite-utils/issues/348 | IC_kwDOCGYnMM48Gnf3 | simonw 9599 | 2022-01-09T20:22:43Z | 2022-01-09T20:22:43Z | OWNER | I'm not going to implement `--page-size` unless someone specifically requests it - I don't like having features that I've never needed to use myself. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for creating an empty database 1067771698 | |
| 1008383293 | https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008383293 | https://api.github.com/repos/simonw/sqlite-utils/issues/348 | IC_kwDOCGYnMM48GrU9 | simonw 9599 | 2022-01-09T20:38:17Z | 2022-01-09T20:38:17Z | OWNER | Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#creating-an-empty-database | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Command for creating an empty database 1067771698 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] (
[html_url] TEXT,
[issue_url] TEXT,
[id] INTEGER PRIMARY KEY,
[node_id] TEXT,
[user] INTEGER REFERENCES [users]([id]),
[created_at] TEXT,
[updated_at] TEXT,
[author_association] TEXT,
[body] TEXT,
[reactions] TEXT,
[issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
ON [issue_comments] ([user]);