html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/datasette/pull/1243#issuecomment-785485597,https://api.github.com/repos/simonw/datasette/issues/1243,785485597,MDEyOklzc3VlQ29tbWVudDc4NTQ4NTU5Nw==,22429695,codecov[bot],2021-02-25T00:28:30Z,2021-02-25T00:28:30Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=h1) Report > Merging [#1243](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=desc) (887bfd2) into [main](https://codecov.io/gh/simonw/datasette/commit/726f781c50e88f557437f6490b8479c3d6fabfc2?el=desc) (726f781) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1243/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1243 +/- ## ======================================= Coverage 91.56% 91.56% ======================================= Files 34 34 Lines 4242 4242 ======================================= Hits 3884 3884 Misses 358 358 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=footer). Last update [726f781...32652d9](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815955014,fix small typo, https://github.com/simonw/sqlite-utils/issues/238#issuecomment-785972074,https://api.github.com/repos/simonw/sqlite-utils/issues/238,785972074,MDEyOklzc3VlQ29tbWVudDc4NTk3MjA3NA==,9599,simonw,2021-02-25T15:08:36Z,2021-02-25T15:08:36Z,OWNER,I bet the bug is in here: https://github.com/simonw/sqlite-utils/blob/806c21044ac8d31da35f4c90600e98115aade7c6/sqlite_utils/db.py#L593-L602,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816523763,.add_foreign_key() corrupts database if column contains a space, https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785978689,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785978689,MDEyOklzc3VlQ29tbWVudDc4NTk3ODY4OQ==,9599,simonw,2021-02-25T15:18:03Z,2021-02-25T15:18:03Z,OWNER,"The Python `.extract()` method currently starts like this: ```python def extract(self, columns, table=None, fk_column=None, rename=None): rename = rename or {} if isinstance(columns, str): columns = [columns] if not set(columns).issubset(self.columns_dict.keys()): raise InvalidColumns( ""Invalid columns {} for table with columns {}"".format( columns, list(self.columns_dict.keys()) ) ) ... ``` Note that it takes a list of columns (and treats a string as a single item list). That's because it can be called with a list of columns and it will use them to populate another table of unique tuples of those column values. So a new mechanism that can instead read JSON values from a single column needs to be compatible with that existing design.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects, https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785979192,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785979192,MDEyOklzc3VlQ29tbWVudDc4NTk3OTE5Mg==,9599,simonw,2021-02-25T15:18:46Z,2021-02-25T15:18:46Z,OWNER,"Likewise the `sqlite-utils extract` command takes one or more columns: ``` Usage: sqlite-utils extract [OPTIONS] PATH TABLE COLUMNS... Extract one or more columns into a separate table Options: --table TEXT Name of the other table to extract columns to --fk-column TEXT Name of the foreign key column to add to the table --rename ... Rename this column in extracted table ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects, https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785979769,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785979769,MDEyOklzc3VlQ29tbWVudDc4NTk3OTc2OQ==,9599,simonw,2021-02-25T15:19:37Z,2021-02-25T15:19:37Z,OWNER,For the Python version I'd like to be able to provide a transformation callback function - which can be `json.loads` but could also be anything else which accepts the value of the current column and returns a Python dictionary of columns and their values to use in the new table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects, https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785980083,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785980083,MDEyOklzc3VlQ29tbWVudDc4NTk4MDA4Mw==,9599,simonw,2021-02-25T15:20:02Z,2021-02-25T15:20:02Z,OWNER,It would be OK if the CLI version only allows you to specify a single column if you are using the `--json` option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects, https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785980813,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785980813,MDEyOklzc3VlQ29tbWVudDc4NTk4MDgxMw==,9599,simonw,2021-02-25T15:21:02Z,2021-02-25T15:23:47Z,OWNER,"Maybe the Python version takes an optional dictionary mapping column names to transformation functions? It could then merge all of those results together - and maybe throw an error if the same key is produced by more than one column. ```python db[""Reports""].extract([""Reported by""], transform={""Reported by"": json.loads}) ``` Or it could have an option for different strategies if keys collide: first wins, last wins, throw exception, add a prefix to the new column name. That feels a bit too complex for an edge-case though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects, https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785983070,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785983070,MDEyOklzc3VlQ29tbWVudDc4NTk4MzA3MA==,9599,simonw,2021-02-25T15:24:17Z,2021-02-25T15:24:17Z,OWNER,I'm going to go with last-wins - so if multiple transform functions return the same key the last one will over-write the others.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects, https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785983837,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785983837,MDEyOklzc3VlQ29tbWVudDc4NTk4MzgzNw==,9599,simonw,2021-02-25T15:25:21Z,2021-02-25T15:28:57Z,OWNER,"Problem with calling this argument `transform=` is that the term ""transform"" already means something else in this library. I could use `convert=` instead. ... but that doesn't instantly make me think of turning a value into multiple columns. How about `expand=`? I've not used that term anywhere yet. db[""Reports""].extract([""Reported by""], expand={""Reported by"": json.loads}) I think that works. You're expanding a single value into several columns of information.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects, https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785992158,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785992158,MDEyOklzc3VlQ29tbWVudDc4NTk5MjE1OA==,9599,simonw,2021-02-25T15:37:04Z,2021-02-25T15:37:04Z,OWNER,"Here's the current implementation of `.extract()`: https://github.com/simonw/sqlite-utils/blob/806c21044ac8d31da35f4c90600e98115aade7c6/sqlite_utils/db.py#L1049-L1074 Tricky detail here: I create the lookup table first, based on the types of the columns that are being extracted. I need to do this because extraction currently uses unique tuples of values, so the table has to be created in advance. But if I'm using these new expand functions to figure out what's going to be extracted, I don't know the names of the columns and their types in advance. I'm only going to find those out during the transformation. This may turn out to be incompatible with how `.extract()` works at the moment. I may need a new method, `.extract_expand()` perhaps? It could be simpler - work only against a single column for example. I can still use the existing `sqlite-utils extract` CLI command though, with a `--json` flag and a rule that you can't run it against multiple columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects, https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786001768,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786001768,MDEyOklzc3VlQ29tbWVudDc4NjAwMTc2OA==,9599,simonw,2021-02-25T15:50:28Z,2021-02-25T15:52:12Z,OWNER,"One option: `.rows_where()` could grow a `ensure_pk=True` option which checks to see if the table is a `rowid` table and, if it is, includes that in the `select`. Or... how about you can call `.rows_where(..., pks=True)` and it will yield `(pk, rowdict)` tuple pairs instead of just returning the sequence of dictionaries? I'm always a little bit nervous of methods that vary their return type based on their arguments. Maybe this would be a separate method instead? ```python for pk, row in table.pk_rows_where(...): # ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows, https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786005078,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786005078,MDEyOklzc3VlQ29tbWVudDc4NjAwNTA3OA==,9599,simonw,2021-02-25T15:54:59Z,2021-02-25T15:56:16Z,OWNER,"Is `pk_rows_where()` a good name? It sounds like it returns ""primary key rows"" which isn't a thing. It actually returns rows along with their primary key. Other options: - `table.rows_with_pk_where(...)` - should this return `(row, pk)` rather than `(pk, row)`? - `table.rows_where_pk(...)` - `table.pk_and_rows_where(...)` - `table.pk_with_rows_where(...)` - `table.pks_with_rows_where(...)` - because rows is pluralized, so pks should be pluralized too? - `table.pks_rows_where(...)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows, https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786006794,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786006794,MDEyOklzc3VlQ29tbWVudDc4NjAwNjc5NA==,9599,simonw,2021-02-25T15:57:17Z,2021-02-25T15:57:28Z,OWNER,"I quite like `pks_with_rows_where(...)` - but grammatically it suggests it will return the primary keys that exist where their rows match the criteria - ""pks with rows"" can be interpreted as ""pks for the rows that..."" as opposed to ""pks accompanied by rows""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows, https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786007209,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786007209,MDEyOklzc3VlQ29tbWVudDc4NjAwNzIwOQ==,9599,simonw,2021-02-25T15:57:50Z,2021-02-25T15:57:50Z,OWNER,`table.pks_and_rows_where(...)` is explicit and I think less ambiguous than the other options.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows, https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786016380,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786016380,MDEyOklzc3VlQ29tbWVudDc4NjAxNjM4MA==,9599,simonw,2021-02-25T16:10:01Z,2021-02-25T16:10:01Z,OWNER,"I prototyped this and I like it: ``` In [1]: import sqlite_utils In [2]: db = sqlite_utils.Database(""/Users/simon/Dropbox/Development/datasette/fixtures.db"") In [3]: list(db[""compound_primary_key""].pks_and_rows_where()) Out[3]: [(('a', 'b'), {'pk1': 'a', 'pk2': 'b', 'content': 'c'})] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows, https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786035142,https://api.github.com/repos/simonw/sqlite-utils/issues/239,786035142,MDEyOklzc3VlQ29tbWVudDc4NjAzNTE0Mg==,9599,simonw,2021-02-25T16:36:17Z,2021-02-25T16:36:17Z,OWNER,WIP in a pull request.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects, https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786036355,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786036355,MDEyOklzc3VlQ29tbWVudDc4NjAzNjM1NQ==,9599,simonw,2021-02-25T16:38:07Z,2021-02-25T16:38:07Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#listing-rows-with-their-primary-keys,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows, https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786037219,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786037219,MDEyOklzc3VlQ29tbWVudDc4NjAzNzIxOQ==,9599,simonw,2021-02-25T16:39:23Z,2021-02-25T16:39:23Z,OWNER,"Example from the docs: ```pycon >>> db = sqlite_utils.Database(memory=True) >>> db[""dogs""].insert({""name"": ""Cleo""}) >>> for pk, row in db[""dogs""].pks_and_rows_where(): ... print(pk, row) 1 {'rowid': 1, 'name': 'Cleo'} >>> db[""dogs_with_pk""].insert({""id"": 5, ""name"": ""Cleo""}, pk=""id"") >>> for pk, row in db[""dogs_with_pk""].pks_and_rows_where(): ... print(pk, row) 5 {'id': 5, 'name': 'Cleo'} >>> db[""dogs_with_compound_pk""].insert( ... {""species"": ""dog"", ""id"": 3, ""name"": ""Cleo""}, ... pk=(""species"", ""id"") ... ) >>> for pk, row in db[""dogs_with_compound_pk""].pks_and_rows_where(): ... print(pk, row) ('dog', 3) {'species': 'dog', 'id': 3, 'name': 'Cleo'} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows, https://github.com/simonw/sqlite-utils/issues/237#issuecomment-786049394,https://api.github.com/repos/simonw/sqlite-utils/issues/237,786049394,MDEyOklzc3VlQ29tbWVudDc4NjA0OTM5NA==,9599,simonw,2021-02-25T16:56:14Z,2021-02-25T16:56:14Z,OWNER,"Other methods (`db.create_view()` for example) have `ignore=True` to mean ""don't throw an error if this causes a problem"", so I'm good with adding that to `.drop_view()`. I don't like using it as the default partly because that would be a very minor breaking API change, but mainly because I don't want to hide mistakes people make - e.g. if you mistype the name of the table you are trying to drop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815554385,"db[""my_table""].drop(ignore=True) parameter, plus sqlite-utils drop-table --ignore and drop-view --ignore", https://github.com/simonw/sqlite-utils/issues/237#issuecomment-786049686,https://api.github.com/repos/simonw/sqlite-utils/issues/237,786049686,MDEyOklzc3VlQ29tbWVudDc4NjA0OTY4Ng==,9599,simonw,2021-02-25T16:56:42Z,2021-02-25T16:56:42Z,OWNER,"So: ```python db[""my_table""].drop(ignore=True) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815554385,"db[""my_table""].drop(ignore=True) parameter, plus sqlite-utils drop-table --ignore and drop-view --ignore", https://github.com/simonw/sqlite-utils/issues/237#issuecomment-786050562,https://api.github.com/repos/simonw/sqlite-utils/issues/237,786050562,MDEyOklzc3VlQ29tbWVudDc4NjA1MDU2Mg==,9599,simonw,2021-02-25T16:57:56Z,2021-02-25T16:57:56Z,OWNER,"`sqlite-utils create-view` currently has a `--ignore` option, so adding that to `sqlite-utils drop-view` and `sqlite-utils drop-table` makes sense as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815554385,"db[""my_table""].drop(ignore=True) parameter, plus sqlite-utils drop-table --ignore and drop-view --ignore",