{"html_url": "https://github.com/simonw/sqlite-utils/pull/277#issuecomment-864092515", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/277", "id": 864092515, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDA5MjUxNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T14:47:57Z", "updated_at": "2021-06-18T14:47:57Z", "author_association": "OWNER", "body": "This is a neat improvement.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 923612361, "label": "add -h support closes #276"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/273#issuecomment-864099764", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/273", "id": 864099764, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDA5OTc2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T14:59:27Z", "updated_at": "2021-06-18T14:59:27Z", "author_association": "OWNER", "body": "I'm going to merge this as-is and work on the JSON/TSV support in a separate issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 922099793, "label": "sqlite-utils memory command for directly querying CSV/JSON data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864101267", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 864101267, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDEwMTI2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T15:01:41Z", "updated_at": "2021-06-18T15:01:41Z", "author_association": "OWNER", "body": "I'll split the remaining work out into separate issues.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864103005", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279", "id": 864103005, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDEwMzAwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T15:04:15Z", "updated_at": "2021-06-18T15:04:15Z", "author_association": "OWNER", "body": "To detect JSON, check to see if the stream starts with `[` or `{` - maybe do something more sophisticated than that.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924990677, "label": "sqlite-utils memory should handle TSV and JSON in addition to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864126781", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/278", "id": 864126781, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDEyNjc4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T15:43:19Z", "updated_at": "2021-06-18T15:43:19Z", "author_association": "OWNER", "body": "I don't think it's possible to do this without breaking backwards compatibility, unfortunately.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 923697888, "label": "Support db as first parameter before subcommand, or as environment variable"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864128489", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/278", "id": 864128489, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDEyODQ4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T15:46:24Z", "updated_at": "2021-06-18T15:46:24Z", "author_association": "OWNER", "body": "A workaround could be to define a bash or zsh alias of some sort.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 923697888, "label": "Support db as first parameter before subcommand, or as environment variable"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864129273", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279", "id": 864129273, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDEyOTI3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T15:47:47Z", "updated_at": "2021-06-18T15:47:47Z", "author_association": "OWNER", "body": "Detecting valid JSON is tricky - just because a stream starts with `[` or `{` doesn't mean the entire stream is valid JSON. You need to parse the entire stream to determine that for sure.\r\n\r\nOne way to solve this would be with a custom state machine. Another would be to use the `ijson` streaming parser - annoyingly it throws the same exception class for invalid JSON for different reasons, but the `e.args[0]` for that exception includes human-readable text about the error - if it's anything other than `parse error: premature EOF` then it probably means the JSON was invalid.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924990677, "label": "sqlite-utils memory should handle TSV and JSON in addition to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864206308", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279", "id": 864206308, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDIwNjMwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T18:25:04Z", "updated_at": "2021-06-18T18:25:04Z", "author_association": "OWNER", "body": "Or... since I'm not using a streaming JSON parser at the moment, if I think something is JSON I can load the entire thing into memory to validate it.\r\n\r\nI still need to detect newline-delimited JSON. For that I can consume the first line of the input to see if it's a valid JSON object, then maybe sniff the second line too?\r\n\r\nThis does mean that if the input is a single line of GIANT JSON it will all be consumed into memory at once, but that's going to happen anyway.\r\n\r\nSo I need a function which, given a file pointer, consumes from it, detects the type, then returns that type AND a file pointer to the beginning of the file again. I can use `io.BufferedReader` for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924990677, "label": "sqlite-utils memory should handle TSV and JSON in addition to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279", "id": 864207841, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDIwNzg0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T18:28:40Z", "updated_at": "2021-06-18T18:28:46Z", "author_association": "OWNER", "body": "```python\r\ndef detect_format(fp):\r\n    # ...\r\n    return \"csv\", fp, dialect\r\n    # or\r\n    return \"json\", fp, parsed_data\r\n    # or\r\n    return \"json-nl\", fp, docs\r\n```\r\nThe mixed return types here are ugly. In all of these cases what we really want is to return a generator of `{...}` objects. So maybe it returns that instead.\r\n```python\r\ndef filepointer_to_documents(fp):\r\n    # ...\r\n    yield from documents\r\n```\r\nI can refactor `sqlite-utils insert` to use this new code too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924990677, "label": "sqlite-utils memory should handle TSV and JSON in addition to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864208476", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279", "id": 864208476, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDIwODQ3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T18:30:08Z", "updated_at": "2021-06-18T23:30:19Z", "author_association": "OWNER", "body": "So maybe this is a function which can either be told the format or, if none is provided, it detects one for itself.\r\n```python\r\ndef rows_from_file(fp, format=None):\r\n    # ...\r\n    yield from rows\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924990677, "label": "sqlite-utils memory should handle TSV and JSON in addition to CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/281#issuecomment-864323438", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/281", "id": 864323438, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDMyMzQzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T23:55:06Z", "updated_at": "2021-06-18T23:55:06Z", "author_association": "OWNER", "body": "The `-:json` idea is flawed: Click thinks that's the syntax for an option called `:json`.\r\n\r\nI'm going to do `stdin:json` - which means you can't open a file called `stdin` - but you could use `cat stdin | sqlite-utils memory stdin:json ...`  instead which is an OK workaround.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924992318, "label": "Mechanism for explicitly stating CSV or JSON or TSV for sqlite-utils memory"}, "performed_via_github_app": null}