html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1154373361,https://api.github.com/repos/simonw/sqlite-utils/issues/441,1154373361,IC_kwDOCGYnMM5Ezlbx,9599,simonw,2022-06-13T20:01:25Z,2022-06-13T20:01:25Z,OWNER,"Yeah, at the moment the best way to do this is with `search_sql()`, but you're right it really isn't very intuitive.
Here's how I would do this, using a CTE trick to combine the queries:
```python
search_sql = db[""articles""].search_sql(columns=[""title"", ""author""]))
sql = f""""""
with search_results as ({search_sql})
select * from search_results where owner = :owner
""""""
results = db.query(sql, {""query"": ""my search query"", ""owner"": ""my owner""})
```
I'm not sure if `sqlite-utils` should ever evolve to provide a better way of doing this kind of thing to be honest - if it did, it would turn into more of an ORM. Something like [PeeWee](http://docs.peewee-orm.com/en/latest/) may be a better option here.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1257724585,Combining `rows_where()` and `search()` to limit which rows are searched,
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154385916,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154385916,IC_kwDOCGYnMM5Ezof8,9599,simonw,2022-06-13T20:15:49Z,2022-06-13T20:15:49Z,OWNER,`rows_from_file()` isn't part of the documented API but maybe it should be!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors,
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154386795,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154386795,IC_kwDOCGYnMM5Ezotr,9599,simonw,2022-06-13T20:16:53Z,2022-06-13T20:16:53Z,OWNER,"Steps to demonstrate that `sqlite-utils insert` is not affected:
```bash
curl -o artsdatabanken.csv https://artsdatabanken.no/Fab2018/api/export/csv
sqlite-utils insert arts.db artsdatabanken artsdatabanken.csv --sniff --csv --encoding utf-16le
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors,
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154387591,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154387591,IC_kwDOCGYnMM5Ezo6H,9599,simonw,2022-06-13T20:17:51Z,2022-06-13T20:17:51Z,OWNER,I don't understand why that works but calling `insert_all()` does not.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors,
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154396400,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154396400,IC_kwDOCGYnMM5EzrDw,9599,simonw,2022-06-13T20:28:25Z,2022-06-13T20:28:25Z,OWNER,"Fixing that `key` thing (to ignore any key that is `None`) revealed a new bug:
```
File ~/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py:376, in hash_record(record, keys)
373 if keys is not None:
374 to_hash = {key: record[key] for key in keys}
375 return hashlib.sha1(
--> 376 json.dumps(to_hash, separators=("","", "":""), sort_keys=True, default=repr).encode(
377 ""utf8""
378 )
379 ).hexdigest()
File ~/.pyenv/versions/3.8.2/lib/python3.8/json/__init__.py:234, in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
232 if cls is None:
233 cls = JSONEncoder
--> 234 return cls(
235 skipkeys=skipkeys, ensure_ascii=ensure_ascii,
236 check_circular=check_circular, allow_nan=allow_nan, indent=indent,
237 separators=separators, default=default, sort_keys=sort_keys,
238 **kw).encode(obj)
File ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:199, in JSONEncoder.encode(self, o)
195 return encode_basestring(o)
196 # This doesn't pass the iterator directly to ''.join() because the
197 # exceptions aren't as detailed. The list call should be roughly
198 # equivalent to the PySequence_Fast that ''.join() would do.
--> 199 chunks = self.iterencode(o, _one_shot=True)
200 if not isinstance(chunks, (list, tuple)):
201 chunks = list(chunks)
File ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:257, in JSONEncoder.iterencode(self, o, _one_shot)
252 else:
253 _iterencode = _make_iterencode(
254 markers, self.default, _encoder, self.indent, floatstr,
255 self.key_separator, self.item_separator, self.sort_keys,
256 self.skipkeys, _one_shot)
--> 257 return _iterencode(o, 0)
TypeError: '<' not supported between instances of 'NoneType' and 'str'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors,
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154449442,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154449442,IC_kwDOCGYnMM5Ez4Ai,9599,simonw,2022-06-13T21:18:26Z,2022-06-13T21:20:12Z,OWNER,"Here are full steps to replicate the bug:
```python
from urllib.request import urlopen
import sqlite_utils
db = sqlite_utils.Database(memory=True)
with urlopen(""https://artsdatabanken.no/Fab2018/api/export/csv"") as fab:
reader, other = sqlite_utils.utils.rows_from_file(fab, encoding=""utf-16le"")
db[""fab2018""].insert_all(reader, pk=""Id"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors,
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154453319,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154453319,IC_kwDOCGYnMM5Ez49H,9599,simonw,2022-06-13T21:23:16Z,2022-06-13T21:23:16Z,OWNER,"Aha! I think I see what's happening here. Here's what `DictReader` does if one of the lines has too many items in it:
```pycon
>>> import csv, io
>>> list(csv.DictReader(io.StringIO(""id,name\n1,Cleo,nohead\n2,Barry"")))
[{'id': '1', 'name': 'Cleo', None: ['nohead']}, {'id': '2', 'name': 'Barry'}]
```
See how that row with too many items gets this:
`[{'id': '1', 'name': 'Cleo', None: ['nohead']}`
That's a `None` for the key and (weirdly) a list containing the single item for the value!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors,
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154454127,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154454127,IC_kwDOCGYnMM5Ez5Jv,9599,simonw,2022-06-13T21:24:18Z,2022-06-13T21:24:18Z,OWNER,"That weird behaviour is documented here: https://docs.python.org/3/library/csv.html#csv.DictReader
> If a row has more fields than fieldnames, the remaining data is put in a list and stored with the fieldname specified by *restkey* (which defaults to `None`). If a non-blank row has fewer fields than fieldnames, the missing values are filled-in with the value of *restval* (which defaults to `None`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors,
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154456183,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154456183,IC_kwDOCGYnMM5Ez5p3,9599,simonw,2022-06-13T21:26:55Z,2022-06-13T21:26:55Z,OWNER,"So I need to make a design decision here: what should `sqlite-utils` do with CSV files that have rows with more values than there are headings?
Some options:
- Ignore those extra fields entirely - silently drop that data. I'm not keen on this.
- Throw an error. The library does this already, but the error is incomprehensible - it could turn into a useful, human-readable error instead.
- Put the data in a JSON list in a column with a known name (`None` is not a valid column name, so not that). This could be something like `_restkey` or `_values_with_no_heading`. This feels like a better option, but I'd need to carefully pick a name for it - and come up with an answer for the question of what to do if the CSV file being important already uses that heading name for something else.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors,
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154457028,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154457028,IC_kwDOCGYnMM5Ez53E,9599,simonw,2022-06-13T21:28:03Z,2022-06-13T21:28:03Z,OWNER,"Whatever I decide, I can implement it in `rows_from_file()`, maybe as an optional parameter - then decide how to call it from the `sqlite-utils insert` CLI (perhaps with a new option there too).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors,
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154457893,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154457893,IC_kwDOCGYnMM5Ez6El,9599,simonw,2022-06-13T21:29:02Z,2022-06-13T21:29:02Z,OWNER,"Here's the current function signature for `rows_from_file()`:
https://github.com/simonw/sqlite-utils/blob/26e6d2622c57460a24ffdd0128bbaac051d51a5f/sqlite_utils/utils.py#L174-L179","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors,
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154474482,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154474482,IC_kwDOCGYnMM5Ez-Hy,9599,simonw,2022-06-13T21:50:59Z,2022-06-13T21:51:24Z,OWNER,"Decision: I'm going to default to raising an exception if a row has too many values in it.
You'll be able to pass `ignore_extras=True` to ignore those extra values, or pass `restkey=""the_rest""` to stick them in a list in the `restkey` column.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors,
https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154475454,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154475454,IC_kwDOCGYnMM5Ez-W-,9599,simonw,2022-06-13T21:52:03Z,2022-06-13T21:52:03Z,OWNER,The exception will be called `RowError`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors,