issue_comments
11 rows where issue = 1173023272
This data as json, CSV (advanced)
Suggested facets: user, author_association, created_at (date), updated_at (date)
| id ▼ | html_url | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1072833174 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072833174 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_8iKW | simonw 9599 | 2022-03-18T21:34:06Z | 2022-03-18T21:34:06Z | OWNER | Good call-out: right now the `parsedate()` and `parsedatetime()` functions both terminate with an exception if they hit something invalid: https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes It would be better if this was configurable by the user (and properly documented) - options could include "set null if date is invalid" and "leave the value as it is if invalid" in addition to throwing an error. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
| 1072834273 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072834273 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_8ibh | simonw 9599 | 2022-03-18T21:36:05Z | 2022-03-18T21:36:05Z | OWNER | Python's `str.encode()` method has a `errors=` parameter that does something along these lines: https://docs.python.org/3/library/stdtypes.html#str.encode > *errors* may be given to set a different error handling scheme. The default for *errors* is `'strict'`, meaning that encoding errors raise a [`UnicodeError`](https://docs.python.org/3/library/exceptions.html#UnicodeError "UnicodeError"). Other possible values are `'ignore'`, `'replace'`, `'xmlcharrefreplace'`, `'backslashreplace'` and any other name registered via [`codecs.register_error()`](https://docs.python.org/3/library/codecs.html#codecs.register_error "codecs.register_error"), Imitating this might be the way to go. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
| 1073448904 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073448904 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-4fI | simonw 9599 | 2022-03-21T03:28:12Z | 2022-03-21T03:30:37Z | OWNER | Generating a test database using a pattern from https://www.geekytidbits.com/date-range-table-sqlite/ ``` sqlite-utils create-database test-dates.db sqlite-utils create-table test-dates.db dates id integer date text --pk id sqlite-utils test-dates.db "WITH RECURSIVE cnt(x) AS ( SELECT 0 UNION ALL SELECT x+1 FROM cnt LIMIT (SELECT ((julianday('2016-04-01') - julianday('2016-03-15'))) + 1) ) insert into dates (date) select date(julianday('2016-03-15'), '+' || x || ' days') as date FROM cnt;" ``` After running that: ``` % sqlite-utils rows test-dates.db dates [{"id": 1, "date": "2016-03-15"}, {"id": 2, "date": "2016-03-16"}, {"id": 3, "date": "2016-03-17"}, {"id": 4, "date": "2016-03-18"}, {"id": 5, "date": "2016-03-19"}, {"id": 6, "date": "2016-03-20"}, {"id": 7, "date": "2016-03-21"}, {"id": 8, "date": "2016-03-22"}, {"id": 9, "date": "2016-03-23"}, {"id": 10, "date": "2016-03-24"}, {"id": 11, "date": "2016-03-25"}, {"id": 12, "date": "2016-03-26"}, {"id": 13, "date": "2016-03-27"}, {"id": 14, "date": "2016-03-28"}, {"id": 15, "date": "2016-03-29"}, {"id": 16, "date": "2016-03-30"}, {"id": 17, "date": "2016-03-31"}, {"id": 18, "date": "2016-04-01"}] ``` Then to make one of them invalid: sqlite-utils test-dates.db "update dates set date = '//' where id = 10" | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
| 1073450588 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073450588 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-45c | simonw 9599 | 2022-03-21T03:32:58Z | 2022-03-21T03:32:58Z | OWNER | Then I ran this to convert `2016-03-27` etc to `2016/03/27` so I could see which ones were later converted: sqlite-utils convert test-dates.db dates date 'value.replace("-", "/")' | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
| 1073451659 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073451659 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-5KL | simonw 9599 | 2022-03-21T03:35:01Z | 2022-03-21T03:35:01Z | OWNER | I confirmed that if it fails for any value ALL values are left alone, since it runs in a transaction. Here's the code that does that: https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L2523-L2526 | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
| 1073453230 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453230 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-5iu | simonw 9599 | 2022-03-21T03:40:37Z | 2022-03-21T03:40:37Z | OWNER | I think the options here should be: - On error, raise an exception and revert the transaction (the current default) - On error, leave the value as-is - On error, set the value to `None` These need to be indicated by parameters to the `r.parsedate()` function. Some design options: - `ignore=True` to ignore errors - but how does it know if it should leave the value or set it to `None`? This is similar to other `ignore=True` parameters elsewhere in the Python API. - `errors="ignore"`, `errors="set-null"` - I don't like magic string values very much, but this is similar to Python's `str.encode(errors=)` mechanism - `errors=r.IGNORE` - using constants, which at least avoids magic strings. The other one could be `errors=r.SET_NULL` - `error=lambda v: None` or `error=lambda v: v` - this is a bit confusing though, introducing another callback that gets to have a go at converting the error if the first callback failed? And what happens if that lambda itself raises an error? | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
| 1073453370 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453370 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-5k6 | simonw 9599 | 2022-03-21T03:41:06Z | 2022-03-21T03:41:06Z | OWNER | I'm going to try the `errors=r.IGNORE` option and see what that looks like once implemented. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
| 1073455905 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073455905 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-6Mh | simonw 9599 | 2022-03-21T03:44:47Z | 2022-03-21T03:45:00Z | OWNER | This is quite nice: ``` % sqlite-utils convert test-dates.db dates date "r.parsedate(value, errors=r.IGNORE)" [####################################] 100% % sqlite-utils rows test-dates.db dates [{"id": 1, "date": "2016-03-15"}, {"id": 2, "date": "2016-03-16"}, {"id": 3, "date": "2016-03-17"}, {"id": 4, "date": "2016-03-18"}, {"id": 5, "date": "2016-03-19"}, {"id": 6, "date": "2016-03-20"}, {"id": 7, "date": "2016-03-21"}, {"id": 8, "date": "2016-03-22"}, {"id": 9, "date": "2016-03-23"}, {"id": 10, "date": "//"}, {"id": 11, "date": "2016-03-25"}, {"id": 12, "date": "2016-03-26"}, {"id": 13, "date": "2016-03-27"}, {"id": 14, "date": "2016-03-28"}, {"id": 15, "date": "2016-03-29"}, {"id": 16, "date": "2016-03-30"}, {"id": 17, "date": "2016-03-31"}, {"id": 18, "date": "2016-04-01"}] % sqlite-utils convert test-dates.db dates date "r.parsedate(value, errors=r.SET_NULL)" [####################################] 100% % sqlite-utils rows test-dates.db dates [{"id": 1, "date": "2016-03-15"}, {"id": 2, "date": "2016-03-16"}, {"id": 3, "date": "2016-03-17"}, {"id": 4, "date": "2016-03-18"}, {"id": 5, "date": "2016-03-19"}, {"id": 6, "date": "2016-03-20"}, {"id": 7, "date": "2016-03-21"}, {"id": 8, "date": "2016-03-22"}, {"id": 9, "date": "2016-03-23"}, {"id": 10, "date": null}, {"id": 11, "date": "2016-03-25"}, {"id": 12, "date": "2016-03-26"}, {"id": 13, "date": "2016-03-27"}, {"id": 14, "date": "2016-03-28"}, {"id": 15, "date": "2016-03-29"}, {"id": 16, "date": "2016-03-30"}, {"id": 17, "date": "2016-03-31"}, {"id": 18, "date": "2016-04-01"}] ``` | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
| 1073456155 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456155 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-6Qb | simonw 9599 | 2022-03-21T03:45:37Z | 2022-03-21T03:45:37Z | OWNER | Prototype: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 8255b56..0a3693e 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -2583,7 +2583,11 @@ def _generate_convert_help(): """ ).strip() recipe_names = [ - n for n in dir(recipes) if not n.startswith("_") and n not in ("json", "parser") + n + for n in dir(recipes) + if not n.startswith("_") + and n not in ("json", "parser") + and callable(getattr(recipes, n)) ] for name in recipe_names: fn = getattr(recipes, name) diff --git a/sqlite_utils/recipes.py b/sqlite_utils/recipes.py index 6918661..569c30d 100644 --- a/sqlite_utils/recipes.py +++ b/sqlite_utils/recipes.py @@ -1,17 +1,38 @@ from dateutil import parser import json +IGNORE = object() +SET_NULL = object() -def parsedate(value, dayfirst=False, yearfirst=False): + +def parsedate(value, dayfirst=False, yearfirst=False, errors=None): "Parse a date and convert it to ISO date format: yyyy-mm-dd" - return ( - parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() - ) + try: + return ( + parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst) + .date() + .isoformat() + ) + except parser.ParserError: + if errors is IGNORE: + return value + elif errors is SET_NULL: + return None + else: + raise -def parsedatetime(value, dayfirst=False, yearfirst=False): +def parsedatetime(value, dayfirst=False, yearfirst=False, errors=None): "Parse a datetime and convert it to ISO datetime format: yyyy-mm-ddTHH:MM:SS" - return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + try: + return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + except parser.ParserError: + if errors is IGNORE: + return value + elif erro… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
| 1073456222 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456222 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM4_-6Re | simonw 9599 | 2022-03-21T03:45:52Z | 2022-03-21T03:45:52Z | OWNER | Needs tests and documentation. | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 | |
| 1116684581 | https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1116684581 | https://api.github.com/repos/simonw/sqlite-utils/issues/416 | IC_kwDOCGYnMM5Cj0El | mattkiefer 638427 | 2022-05-03T21:36:49Z | 2022-05-03T21:36:49Z | NONE | Thanks for addressing this @simonw! However, I just reinstalled sqlite-utils 3.26.1 and get an `ParserError: Unknown string format: None`: ``` sqlite-utils --version sqlite-utils, version 3.26.1 ``` ``` sqlite-utils convert idfpr.db license "Original Issue Date" "r.parsedate(value)" Traceback (most recent call last): File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/db.py", line 2514, in convert_value return fn(v) File "<string>", line 2, in fn File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/recipes.py", line 19, in parsedate parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst) File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 1374, in parse return DEFAULTPARSER.parse(timestr, **kwargs) File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 649, in parse raise ParserError("Unknown string format: %s", timestr) dateutil.parser._parser.ParserError: Unknown string format: None Traceback (most recent call last): File "/home/matt/.local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/usr/lib/python3/dist-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/cli.py", line 2707, in convert db[table].convert( File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/db.py", line 2530, in convert self.db.execute(sql, where_args or []) File "/home/matt/.local/lib/python3.9/site-packages/s… | {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} | Options for how `r.parsedate()` should handle invalid dates 1173023272 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] (
[html_url] TEXT,
[issue_url] TEXT,
[id] INTEGER PRIMARY KEY,
[node_id] TEXT,
[user] INTEGER REFERENCES [users]([id]),
[created_at] TEXT,
[updated_at] TEXT,
[author_association] TEXT,
[body] TEXT,
[reactions] TEXT,
[issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
ON [issue_comments] ([user]);